aituner/docs/opprof/oracle-gap-results.md

# Static-policy oracle-gap results

Status: **FINAL — REFUTED WITHIN THE FROZEN TP1 MNS/MBT POLICY SPACE**.

Date: 2026-07-13. The registered experiment completed 104 valid trials on
`dash0` and returned `REFUTED`. Even a phase-perfect oracle with zero detection,
switching, and state-transfer cost is bounded below the registered 10%
SLO-goodput contribution gate.

The machine result is
`runs/opprof-oracle-gap/metrics.json` (SHA-256
`250ba4c1657a8830795ee06392eea4e21c62d958fea11701ba60581ef0266543`).
All 104 trial-level measurements are in
`runs/opprof-oracle-gap/trials.csv` (SHA-256
`60cd901f18bbf107eb130f0095e867c3b6d47a78a42e83bcbe57fecf23cc5f9c`).
The final resumable controller state is
`runs/opprof-oracle-gap/controller-state.json` (SHA-256
`27ec5e9a3cd32a871d583ba8eb6d7d3fe2a338a8fd42369233ada57cd4da6436`).

## Decision

The tested motivation was:

> One static batching policy leaves at least 10% end-to-end SLO-goodput on the
> table across temporally heterogeneous phases, and a phase-aware runtime can
> recover it by switching MNS/MBT policies.

The registered 10,001-point mixture scan finds a worst conservative gap of
**8.333219%**, at P01 time weight `0.0244`. Equal phase time gives
**7.260726%**. An independent implementation that evaluates every exact
static-policy crossover gives a slightly more conservative exact maximum of
**8.333333% = 1/12**, at P01 weight `1/41`. This leaves 1.666667 percentage
points below the 10% gate.

The negative result is stronger than a failed online prototype. The oracle
already knows the current phase and pays no switching cost. A realizable
controller over the same four policies cannot exceed this oracle bound.

This is a scoped refutation, not a universal claim about adaptive serving. It
rules out the current contribution based on phase-aware selection among the
four MNS/MBT configurations for the frozen P01/P06 pair, Qwen3-30B-A3B, TP1,
H20, vLLM 0.24, and the registered SLO. It does **not** rule out a new scheduler,
KV/cache policy, routing-aware mechanism, topology change, or a different
workload family.

## Fixed setup

| Item | Value |
|---|---|
| Placement | `dash0`, serialized on physical GPU0; GPUs 1-7 idle |
| GPU | NVIDIA H20, driver 580.95.05 |
| Model | Qwen3-30B-A3B, BF16 |
| Runtime | vLLM `0.24.1.dev3+g668cfb7e2`, source `4b253fd8619764b6971a7f2e3a3aa7545f6ace05` |
| Topology | TP1; one server and one client |
| Fixed mechanisms | chunked prefill on; prefix caching on |
| P01 | input `U[128,512]`, exactly 64 output tokens, deterministic steady arrivals |
| P06 | 50/50 input `U[128,512]` / `U[4096,8192]`, exactly 512 output tokens, deterministic bursts of eight |
| P01 timeline | 60 s warm-up + 60 s clean measurement |
| P06 timeline | 60 s warm-up + 120 s clean measurement |
| SLO feasibility | at least 95% of clean-admitted requests pass both TTFT and TPOT |
| TTFT SLO | <=2 s for input <=4096; <=4 s for input <=32768; otherwise <=6 s |
| TPOT SLO | <=50 ms/token |

The four policies were:

| Config | MNS | MBT |
|---|---:|---:|
| C00 | 1024 | 8192 |
| C10 | 64 | 8192 |
| C01 | 1024 | 2048 |
| C11 | 64 | 2048 |

## Final SLO frontiers

`L` is the highest majority-feasible offered rate. `U` is the next higher
majority-infeasible rate. Both sides below have at least three trials, and all
accepted rate sequences are monotone.

| Config | P01 `L` | P01 `U` | P06 `L` | P06 `U` |
|---|---:|---:|---:|---:|
| C00 | 28.0 | 30.0 | 2.3 | 2.4 |
| C10 | 24.0 | 26.0 | 2.4 | 2.5 |
| C01 | 28.0 | 30.0 | 2.2 | 2.3 |
| C11 | 24.0 | 26.0 | 2.2 | 2.3 |

The optimistic oracle upper bound chooses C00/C01 at `U=30` for P01 and C10
at `U=2.5` for P06. The static lower bound chooses C10 below P01 weight `1/41`
and C00 above it; they tie at the exact worst point.

| Mixture | Oracle upper | Best-static lower | Conservative gap |
|---|---:|---:|---:|
| Pure P06 | 2.500000 | 2.400000 | 4.166667% |
| Exact worst, P01 weight `1/41` | 3.170732 | 2.926829 | **8.333333%** |
| Equal phase time | 16.250000 | 15.150000 | **7.260726%** |
| Pure P01 | 30.000000 | 28.000000 | 7.142857% |

The specialization exists but is too small. Reducing MNS from 1024 to 64
raises the conservative P06 lower bound from 2.3 to 2.4 req/s, while lowering
P01 from 28 to 24 req/s. Even after using infeasible `U` values for the oracle
and feasible `L` values for the static baseline, the best possible phase-aware
selection cannot reach 10%.

## Robustness finding

The strongest system finding is not config specialization but a repeat-level
mode flip at P01/26 rps. Both C10 and C11 show the identical verdict sequence
`rep0=0% pass`, `rep1=100% pass`, `rep2=0% pass`. For C10, TTFT p50 is
3864.62, 252.71, and 3994.88 ms; for C11 it is 6533.62, 593.44, and
5812.71 ms. TPOT remains below the 50 ms SLO in these trials.

These trials are measurement-valid: exact outputs, offered-rate tolerance,
schedule lag, timestamps, and client invariants all pass. The majority rule
therefore classifies 26 rps as infeasible for both configs. The result does not
change the oracle-gap decision because neither config supplies the P01 oracle
maximum or the relevant best-static P01 lower bound.

Token-domain seed and server execution history change together across
repetitions, so this experiment cannot attribute the flip to MoE routing,
cache/compiler state, or another source. It does motivate a narrower factorial
study that crosses token seed with fresh/reused server state and randomizes
order, while recording routed-expert and per-step scheduler telemetry. That is
a mechanism question; it should not be presented as evidence for a phase-aware
MNS/MBT controller.

## Execution and audit history

The 104 scored trials comprise 52 base-grid primaries, 18 registered upward or
downward extensions, 32 boundary confirmations, and two boundary-closure
trials. The final closure moved C00/P06 from the provisional `[2.4,2.5)` to
`[2.3,2.4)` and repeated 2.3 rps to 3/3 feasible.

Four frozen amendments are recorded in the protocol:

- A-OG-1 extended only the P06 upward anchors after C00 remained feasible
  through the original 2.3-rps limit.
- A-OG-2 quarantined one P06/C01/2.2 local `ClientOSError` attempt and retried
  the identical logical trial on a fresh server.
- A-OG-3 generalized the same pre-score transport rule after one
  P06/C10/1.9 attempt showed the identical signature.
- A-OG-4, frozen before any confirmation score, added closure for a
  majority-shifted boundary without adding new anchors.

The two transport-invalid attempts and one pre-A-OG-4 interrupted attempt
created no score and do not enter any metric. Their retained tree hashes are,
respectively,
`434863ba90513cbc54534ffbc1a13c980b3ef7d567190a0aa3f97b55650acbb2`,
`a57b1ac5f090680bb70c16b5d709eb2b8ac47dce57c7a03b8077cd9b6d80d831`,
and `bc3feb53514601e641ef1db204c74cffe3d282b24ad797e5b161468f5d15de5c`.

Ten deliberately overloaded primary anchors exceed both the 1 s schedule-lag
gate and the 5% offered-rate tolerance. They are correctly classified as
infeasible and are not used as final boundary points. All 48 final boundary
trials pass both client-side gates; their maximum schedule lag is 573.59 ms.

The final experiment fingerprint uses repository commit `16177b0`, analyzer
SHA-256
`d86ecb1f077472906cbb729bd2c9d4b3a82ac6dfdc90838a17ae300d0767110d`,
controller SHA-256
`f7c5c2f74f2002f1e4b097e608d165a3a2e9374fbf07935a4d6d6a7c5d45a83a`,
and protocol SHA-256
`173f969a4428643cf6c4b950413aa82bef25cafd163e20d7944370a5d87af435`.
The archived launch log SHA-256 is
`54e2f4804a5efee072b670043c4092cf41f1a27664faa05ea71bfc1412c3e9db`.

## GPU accounting and cleanup

The campaign used **4.8877033845 H20-hours**, below the 6.0-hour cap, across
5 h 41 min 51 s wall time including amendment, audit, restart, and server
startup intervals. At completion, all eight H20s report 0 MiB, 0% utilization,
and zero compute processes.

## Sanity block

There are no data-sanity red flags. The P01/26 all-or-none mode flip is a
scientific robustness finding, not an invalid trial signature; it repeats in
two configs and all measurement invariants pass.

| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
|---|---:|---:|---:|---:|---|
| Score-row indicator | 104 | 1 | 1 | 1 expected | 104 score files; no overwrite |
| Target rate (req/s) | 104 | 1.4 | 36.0 | 19 | Non-negative; fixed grid/extensions only |
| Clean cohort per trial | 104 | 168 | 2160 | 30 | Non-empty and non-negative |
| Pass rate | 104 | 0.0 | 1.0 | 57 | All ratios in `[0,1]` |
| SLO-goodput (req/s) | 104 | 0.0 | 27.816667 | 59 | Non-negative; per-cell results not all identical |
| Boundary pass rate | 48 | 0.0 | 1.0 | 33 | Both final sides have three trials |
| Boundary max schedule lag (ms) | 48 | 2.145861 | 573.593768 | 48 | All below the 1000 ms gate |
| Exact-output indicator | 85,402 | 1 | 1 | 1 expected | Every clean request produced the requested token count |
| Frontier cells | 8 | 8 | 8 | 1 expected | 8/8 bracketed and monotone |
| Registered weight scan | 10,001 | 0.0 | 1.0 | 10,001 | Continuous 0.0001 increments, endpoints included |
| Campaign H20-hours | 1 | 4.887703 | 4.887703 | 1 | Non-negative and below 6.0 |
| Final GPU memory/utilization | 8 | 0 MiB / 0% | 0 MiB / 0% | 1 expected | Zero compute processes |

Checked invariants: fixed model, runtime, manifests, SLOs, config values, seeds,
and serialized placement; exact output work; nondecreasing timestamps;
non-negative counters and latencies; pass ratios in range; offered-rate and
schedule gates; majority-of-three final boundaries; monotone accepted
frontiers; complete 10,001-point scan; independent exact-crossover
recomputation; transport-attempt quarantine; GPU hard-cap compliance; and
complete GPU cleanup.