aituner/docs/opprof/oracle-gap-protocol.md

# Static-policy oracle-gap protocol

Status: **FROZEN WITH A-OG-1 THROUGH A-OG-4 AMENDMENTS**.

Date frozen: 2026-07-13 (Asia/Singapore). Existing Phase-3 measurements were
inspected only to choose the workload pair and rate brackets. They are
exploratory calibration data, not primary observations in this protocol.

### A-OG-4 — close a majority-shifted boundary (before confirmation scores)

After all four primary frontiers and 70 scores were complete, the controller
was interrupted before the first confirmation produced a score. The partial
`P01-C10-r26-rep1` attempt contained no result, sanity file, or score and is
archived separately. This amendment is therefore blind to confirmation
outcomes.

The original confirmation schedule still runs first. Afterward, each cell's
majority-vote frontier is recomputed. If either side of the *actual* final
boundary has fewer than three trials because the provisional boundary moved,
the controller runs only enough repetitions at that existing rate to reach
three, high-to-low, on a fresh server for that config. It then recomputes the
boundary and repeats for at most three closure rounds. No new rate anchor may
be added. Failure to obtain a monotone, bracketed boundary with three trials
per side within three rounds or the 6 H20-hour cap makes the experiment
inconclusive.

This fills a stopping-rule omission: it does not change any existing score,
majority rule, phase, config, rate grid, SLO, or oracle calculation.

### A-OG-3 — freeze a per-logical-trial transport retry rule (after 62 scores)

The first attempt at `P06-C10-r1.9-rep0` reproduced the same isolated local
transport signature seen in A-OG-2: one clean request raised
`ClientOSError` 3.56 ms after admission, with HTTP status 0, no first token,
and no output. The other 343 requests produced exactly 512 tokens, the server
stayed healthy, drain took 4.00 s, and every other client invariant passed.
The controller again rejected the attempt before creating a `score.json`.

For the remainder of this experiment, an attempt may be quarantined and the
identical logical trial retried once on a fresh server only if all of the
following hold: the sole failed client invariant is `clean_failures_zero`;
exactly one clean request failed; its error is `ClientOSError`, HTTP status is
0, it produced neither a first token nor output, and it completed within 10
ms of admission; every successful request has exact output; and the server
has no crash or error. The attempt is never scored. A second invalid attempt
for the same logical key, more than one failed request, or any other error
signature is a stop condition and makes the experiment inconclusive.

This rule is independent of config, rate, and observed performance, and
supersedes the one-key wording in A-OG-2 without changing how that retry was
executed. Existing scores remain immutable. No client, grid, policy,
threshold, order, SLO, or oracle calculation changes.

### A-OG-2 — retry one transport-invalid attempt (after 49 scored trials)

The first attempt at `P06-C01-r2.2-rep0` produced one local
`ClientOSError` 3.27 ms after admission, with HTTP status 0, no first token,
and no output. The other 399 requests completed successfully, every successful
request produced exactly 512 tokens, the server stayed healthy, and all other
client invariants passed. The controller rejected the attempt before creating
a `score.json`; no TTFT/TPOT result from this attempt was used to choose the
recovery rule.

This is a measurement-transport failure, not an observed server SLO outcome.
The entire attempt is content-hashed and moved under `invalid-attempts/`, then
the identical logical trial (phase, config, rate, repetition, derived manifest,
seeds, timeline, and SLO) is run once on a fresh C01 server. All 49 existing
scores remain immutable. A second client transport failure in the retry is a
stop condition and makes the experiment inconclusive; it must not be retried
again. No grid, policy, threshold, order, or oracle calculation changes.

### A-OG-1 — extend the P06 upper bracket (after trial 33)

The controller stopped as registered after C00/P06 remained SLO-feasible at
every original and upward-extension anchor through 2.3 requests/s. At that
point 33 trials and 1.519475 H20-hours were complete, all GPU memory had been
released, and no C00/P06 infeasible upper bound existed. No oracle inference
was performed.

This amendment changes only the P06 upward-extension list from
`2.1,2.2,2.3` to `2.1,2.2,2.3,2.4,2.5,2.6,2.8,3.0`. The controller stops at
the first bracket exactly as before. Existing trials are immutable and are
reused; config order, P01 rates, SLO, timelines, repetitions, placement,
metrics, and decision threshold do not change. The resumable controller may
accept the new code/protocol fingerprint only when all immutable runtime,
client, model, manifest, config, and base-grid fields match the pre-amendment
state, and it records both fingerprints under `A-OG-1`.

## Question and decision gate

The candidate motivation is:

> A single global static batching policy leaves at least 10% end-to-end
> SLO-goodput on the table when serving temporally heterogeneous phases; a
> phase-aware runtime policy can recover that gap without changing hardware,
> model, precision, or tensor-parallel topology.

This experiment tests a necessary condition in the existing TP1 policy space
`{C00,C10,C01,C11}`. The optimistic oracle knows the phase and switches with
zero delay, zero state-transfer cost, and no prediction error. If even this
oracle cannot beat the best one-config-for-all-phases policy by 10%, an online
controller over these MNS/MBT choices cannot do so either.

The primary gate uses a conservative capacity bracket:

- `L[p,c]`: highest offered rate accepted as SLO-feasible for phase `p` and
  config `c`;
- `U[p,c]`: lowest higher offered rate accepted as SLO-infeasible;
- oracle upper bound at phase-time weights `w`:
  `sum_p w[p] * max_c U[p,c]`;
- best-static lower bound:
  `max_c sum_p w[p] * L[p,c]`.

We scan every P01/P06 time mixture, including pure endpoints. The current
motivation is **REFUTED** if the maximum conservative ratio
`oracle_upper / static_lower - 1` is below 10%. It is **NOT ESTABLISHED** if the
bound crosses 10% but the observed point estimate does not. A positive result
requires a point-estimate gap of at least 10% and then a separately
pre-registered interleaved-trace validation; this frontier experiment alone
cannot establish a positive E2E contribution.

The conclusion is scoped to the measured MNS/MBT policy family and the chosen
strongest-conflict phase pair. It does not rule out new scheduling mechanisms,
KV-state policies, topology changes, or other workload phases.

## Fixed system boundary

| Item | Frozen value |
|---|---|
| Host | `dash0`, one run at a time on physical GPU0 |
| GPU | NVIDIA H20; no other GPU process anywhere on the host |
| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, BF16 |
| Runtime | `/tmp/wjh-opprof-phase2-dash0-20260711/.venv`, vLLM `0.24.1.dev3+g668cfb7e2` |
| vLLM source | `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0` |
| Topology | TP1, one server, no data/pipeline parallelism |
| Fixed mechanisms | chunked prefill on; prefix caching on |
| Client | Phase-5 timestamp/fixed-rate wrapper over the Phase-3 exact-token client |
| Seeds | workload `20260712`; trial token-domain seed derived only from phase/rate/repetition, never config |

SLO co-location results in Phase 6 showed pass-rate flips despite small
throughput deltas. Therefore unused H20s remain idle: parallel placement is not
authoritative for this experiment.

## Workloads and policies

The pair is chosen before new measurements because Phase 3 showed the strongest
opposing static preference:

- **P01:** input `U[128,512]`, output exactly 64 tokens, deterministic steady
  arrivals. C10 lost 24.27% saturation throughput relative to C00.
- **P06:** 50/50 input mixture `U[128,512]`/`U[4096,8192]`, output exactly 512
  tokens, deterministic bursts of eight. C10 gained 3.37% over C00.

Both reuse the immutable 32,768-row Phase-3 manifests. For every trial a
derived manifest preserves request order, lengths, outputs, and arrival class,
but applies a trial-specific token-seed offset. The same derived manifest is
used for all four configs. This prevents prefix-cache carry-over when a hot
server executes several anchors without changing the logical workload.

| Config | Effective MNS | Effective MBT | Extra flags |
|---|---:|---:|---|
| C00 | 1024 | 8192 | none |
| C10 | 64 | 8192 | `--max-num-seqs 64` |
| C01 | 1024 | 2048 | `--max-num-batched-tokens 2048` |
| C11 | 64 | 2048 | both flags |

Startup logs must confirm these values. A default drift is a stop condition.

## Load grid, order, and repetitions

Primary grids:

- P01: `{26,28,30,32,34,36}` requests/s; execution order
  `32,26,36,28,34,30`.
- P06: `{1.4,1.5,1.6,1.7,1.8,1.9,2.0}` requests/s; execution order
  `1.7,1.4,2.0,1.5,1.9,1.6,1.8`.

Every primary anchor runs once. For each phase/config, the highest primary
feasible anchor and its next higher primary anchor are then run two more times,
giving three trials at both sides of the boundary. If all primary anchors are
feasible, extend upward in the fixed order P01 `38,40,42` or P06
`2.1,2.2,2.3,2.4,2.5,2.6,2.8,3.0`.
If all are infeasible, extend downward in the fixed order P01 `24,22,20` or P06
`1.3,1.2,1.1`. Stop extending at the first bracket.

One primary server is launched per config in order `C11,C00,C01,C10`.
Confirmation servers are fresh and launch in reverse order
`C10,C01,C00,C11`; their boundary anchors run high-to-low. This balances
machine-time drift and makes confirmation independent of the primary server's
cache/compiler state.

Timelines:

- P01: 60 s warm-up + 60 s clean measurement; drain cap 120 s.
- P06: 60 s warm-up + 120 s clean measurement; drain cap 240 s.
- no Kineto profiling; exact greedy output with `ignore_eos`; maximum client
  concurrency 256.

A trial is SLO-feasible when at least 95% of requests admitted during the clean
interval eventually finish successfully and individually satisfy both:

- TTFT <= 2 s for input <= 4,096 tokens; <= 4 s for input <= 32,768; <= 6 s
  otherwise;
- TPOT <= 50 ms, computed as `(completion - first_token)/(output_tokens - 1)`.

SLO-goodput is the number of those passing clean-admission requests divided by
clean seconds. Client schedule lag must stay <=1 s and achieved clean offered
rate must be within 5% of target. Failure of either condition makes the anchor
infeasible; its admitted-only latency is not used to rescue it.

At a repeated boundary, feasibility is the majority of three trial verdicts.
All accepted anchor verdicts must be monotone in offered rate. A persistent
non-monotone result after the registered repeats is a red flag and stops the
oracle-gap inference.

## Validity and stopping rules

Before every server launch record host, GPU, driver, clocks, runtime package
versions, git/source hashes, manifest hashes, exact commands, and process
contamination. Stop on another GPU process, request/output mismatch, manifest
drift, server crash, non-finite latency, ratio outside `[0,1]`, negative
counter, or discontinuous/non-monotone accepted frontier.

The controller is detached and resumable. It kills only process groups it
created, checks zero GPU memory after every server, never overwrites a complete
trial, and writes state atomically. The hard budget is 6 H20-hours; expected
cost is 3.0--4.0 H20-hours and approximately the same wall time because runs
are serialized.

## Required report

The report includes every trial's target/achieved rate, clean cohort size,
pass count/rate, SLO-goodput, TTFT/TPOT percentiles, schedule lag, failure
reasons, accepted frontier brackets, per-phase oracle choices, best static
choice, equal-time gap, worst-mixture conservative gap, and GPU-hours.

The final statistics section ends with a data-sanity block containing `n`,
min/max, distinct-value counts, and checks for non-negative counters, ratios in
`[0,1]`, non-identical per-config results, exact output work, monotone
frontiers, and continuous rate brackets.