Add static-policy oracle gap experiment

docs/opprof/oracle-gap-protocol.md

@@ -0,0 +1,159 @@
+# Static-policy oracle-gap protocol
+
+Status: **FROZEN BEFORE NEW GPU EXECUTION**.
+
+Date frozen: 2026-07-13 (Asia/Singapore). Existing Phase-3 measurements were
+inspected only to choose the workload pair and rate brackets. They are
+exploratory calibration data, not primary observations in this protocol.
+
+## Question and decision gate
+
+The candidate motivation is:
+
+> A single global static batching policy leaves at least 10% end-to-end
+> SLO-goodput on the table when serving temporally heterogeneous phases; a
+> phase-aware runtime policy can recover that gap without changing hardware,
+> model, precision, or tensor-parallel topology.
+
+This experiment tests a necessary condition in the existing TP1 policy space
+`{C00,C10,C01,C11}`. The optimistic oracle knows the phase and switches with
+zero delay, zero state-transfer cost, and no prediction error. If even this
+oracle cannot beat the best one-config-for-all-phases policy by 10%, an online
+controller over these MNS/MBT choices cannot do so either.
+
+The primary gate uses a conservative capacity bracket:
+
+- `L[p,c]`: highest offered rate accepted as SLO-feasible for phase `p` and
+  config `c`;
+- `U[p,c]`: lowest higher offered rate accepted as SLO-infeasible;
+- oracle upper bound at phase-time weights `w`:
+  `sum_p w[p] * max_c U[p,c]`;
+- best-static lower bound:
+  `max_c sum_p w[p] * L[p,c]`.
+
+We scan every P01/P06 time mixture, including pure endpoints. The current
+motivation is **REFUTED** if the maximum conservative ratio
+`oracle_upper / static_lower - 1` is below 10%. It is **NOT ESTABLISHED** if the
+bound crosses 10% but the observed point estimate does not. A positive result
+requires a point-estimate gap of at least 10% and then a separately
+pre-registered interleaved-trace validation; this frontier experiment alone
+cannot establish a positive E2E contribution.
+
+The conclusion is scoped to the measured MNS/MBT policy family and the chosen
+strongest-conflict phase pair. It does not rule out new scheduling mechanisms,
+KV-state policies, topology changes, or other workload phases.
+
+## Fixed system boundary
+
+| Item | Frozen value |
+|---|---|
+| Host | `dash0`, one run at a time on physical GPU0 |
+| GPU | NVIDIA H20; no other GPU process anywhere on the host |
+| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, BF16 |
+| Runtime | `/tmp/wjh-opprof-phase2-dash0-20260711/.venv`, vLLM `0.24.1.dev3+g668cfb7e2` |
+| vLLM source | `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0` |
+| Topology | TP1, one server, no data/pipeline parallelism |
+| Fixed mechanisms | chunked prefill on; prefix caching on |
+| Client | Phase-5 timestamp/fixed-rate wrapper over the Phase-3 exact-token client |
+| Seeds | workload `20260712`; trial token-domain seed derived only from phase/rate/repetition, never config |
+
+SLO co-location results in Phase 6 showed pass-rate flips despite small
+throughput deltas. Therefore unused H20s remain idle: parallel placement is not
+authoritative for this experiment.
+
+## Workloads and policies
+
+The pair is chosen before new measurements because Phase 3 showed the strongest
+opposing static preference:
+
+- **P01:** input `U[128,512]`, output exactly 64 tokens, deterministic steady
+  arrivals. C10 lost 24.27% saturation throughput relative to C00.
+- **P06:** 50/50 input mixture `U[128,512]`/`U[4096,8192]`, output exactly 512
+  tokens, deterministic bursts of eight. C10 gained 3.37% over C00.
+
+Both reuse the immutable 32,768-row Phase-3 manifests. For every trial a
+derived manifest preserves request order, lengths, outputs, and arrival class,
+but applies a trial-specific token-seed offset. The same derived manifest is
+used for all four configs. This prevents prefix-cache carry-over when a hot
+server executes several anchors without changing the logical workload.
+
+| Config | Effective MNS | Effective MBT | Extra flags |
+|---|---:|---:|---|
+| C00 | 1024 | 8192 | none |
+| C10 | 64 | 8192 | `--max-num-seqs 64` |
+| C01 | 1024 | 2048 | `--max-num-batched-tokens 2048` |
+| C11 | 64 | 2048 | both flags |
+
+Startup logs must confirm these values. A default drift is a stop condition.
+
+## Load grid, order, and repetitions
+
+Primary grids:
+
+- P01: `{26,28,30,32,34,36}` requests/s; execution order
+  `32,26,36,28,34,30`.
+- P06: `{1.4,1.5,1.6,1.7,1.8,1.9,2.0}` requests/s; execution order
+  `1.7,1.4,2.0,1.5,1.9,1.6,1.8`.
+
+Every primary anchor runs once. For each phase/config, the highest primary
+feasible anchor and its next higher primary anchor are then run two more times,
+giving three trials at both sides of the boundary. If all primary anchors are
+feasible, extend upward in the fixed order P01 `38,40,42` or P06 `2.1,2.2,2.3`.
+If all are infeasible, extend downward in the fixed order P01 `24,22,20` or P06
+`1.3,1.2,1.1`. Stop extending at the first bracket.
+
+One primary server is launched per config in order `C11,C00,C01,C10`.
+Confirmation servers are fresh and launch in reverse order
+`C10,C01,C00,C11`; their boundary anchors run high-to-low. This balances
+machine-time drift and makes confirmation independent of the primary server's
+cache/compiler state.
+
+Timelines:
+
+- P01: 60 s warm-up + 60 s clean measurement; drain cap 120 s.
+- P06: 60 s warm-up + 120 s clean measurement; drain cap 240 s.
+- no Kineto profiling; exact greedy output with `ignore_eos`; maximum client
+  concurrency 256.
+
+A trial is SLO-feasible when at least 95% of requests admitted during the clean
+interval eventually finish successfully and individually satisfy both:
+
+- TTFT <= 2 s for input <= 4,096 tokens; <= 4 s for input <= 32,768; <= 6 s
+  otherwise;
+- TPOT <= 50 ms, computed as `(completion - first_token)/(output_tokens - 1)`.
+
+SLO-goodput is the number of those passing clean-admission requests divided by
+clean seconds. Client schedule lag must stay <=1 s and achieved clean offered
+rate must be within 5% of target. Failure of either condition makes the anchor
+infeasible; its admitted-only latency is not used to rescue it.
+
+At a repeated boundary, feasibility is the majority of three trial verdicts.
+All accepted anchor verdicts must be monotone in offered rate. A persistent
+non-monotone result after the registered repeats is a red flag and stops the
+oracle-gap inference.
+
+## Validity and stopping rules
+
+Before every server launch record host, GPU, driver, clocks, runtime package
+versions, git/source hashes, manifest hashes, exact commands, and process
+contamination. Stop on another GPU process, request/output mismatch, manifest
+drift, server crash, non-finite latency, ratio outside `[0,1]`, negative
+counter, or discontinuous/non-monotone accepted frontier.
+
+The controller is detached and resumable. It kills only process groups it
+created, checks zero GPU memory after every server, never overwrites a complete
+trial, and writes state atomically. The hard budget is 6 H20-hours; expected
+cost is 3.0--4.0 H20-hours and approximately the same wall time because runs
+are serialized.
+
+## Required report
+
+The report includes every trial's target/achieved rate, clean cohort size,
+pass count/rate, SLO-goodput, TTFT/TPOT percentiles, schedule lag, failure
+reasons, accepted frontier brackets, per-phase oracle choices, best static
+choice, equal-time gap, worst-mixture conservative gap, and GPU-hours.
+
+The final statistics section ends with a data-sanity block containing `n`,
+min/max, distinct-value counts, and checks for non-negative counters, ratios in
+`[0,1]`, non-identical per-config results, exact output work, monotone
+frontiers, and continuous rate brackets.