Add SimFid+OpProf campaign overview

Entry-point summary of both closed campaigns: verdicts, corrected
readings (P5 supersedes P3's P10-vs-P04 magnitude), methodological
findings (co-location SLO validity, arrival uniformization, compile-
factor env poisoning), artifact map, and GPU accounting.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/opprof-simfid-overview-20260713.md

@@ -0,0 +1,131 @@
+# SimFid + OpProf Campaign Overview
+
+Date: 2026-07-13. Status: both campaigns **CLOSED**. This is the entry-point
+summary; every claim below links to a frozen protocol and a results document.
+Decision history lives in `docs/opprof_campaign_state.md` (this repo) and
+`replayserve/docs/simfid_campaign_state.md`.
+
+## Why these campaigns exist
+
+The paper's motivation requires three evidence-backed claims:
+
+1. **Simulators cannot replace real replay** for config tuning → SimFid.
+2. **The leverage point is engine-knob configuration, not operator
+   implementation**, at the measured regimes → OpProf P3–P5.
+3. **Tuning is genuinely hard**: the surface is workload-conditioned
+   (sign-flips defeat defaults), real evaluation is expensive (measured GPU
+   cost), and the surface churns across engine versions → OpProf P4/P6.
+
+All three now rest on pre-registered protocols with frozen decision rules,
+Holm-corrected contrasts, and no imputation over censored data.
+
+## Campaign 1 — SimFid (replayserve repo)
+
+**Question.** Can a Frontier-class simulator replace real replay for AITuner
+config tuning? **Verdict: NOT ADEQUATE** under the pre-declared decision rule.
+
+- S2-E (3-config TP family, same model/HW): ranking perfect after throughput
+  calibration, but latency is uncalibrated (TTFT p95 sim/real 0.30–0.38, TPOT
+  0.63–0.79) → false-feasible counterexample across a 50 ms TPOT SLO.
+- S2-R-b (12-cell TP×MNS surface, exact C1 workload replayed in Frontier,
+  184 runs, 0 failures): the decision-bearing frozen-calibrated
+  throughput-proxy reading picks TP1/MNS64 → **30.46% real top-1 regret**;
+  trap detection 3/6; LOAO 0/92. All three adequacy components FAIL.
+- Post-hoc diagnostic (NOT decision-bearing): an SLO-gated reading achieves
+  0–0.76% regret, tau-b 0.967, trap 6/6 — anchor-level SLO errors partially
+  cancel at per-cell peaks. This motivates hybrid sim-prune + real-final
+  designs but must always carry the post-hoc label.
+
+Sources: `replayserve/docs/simfid_s2e_report.md`,
+`replayserve/docs/simfid_s2rb_results.md`.
+
+## Campaign 2 — OpProf (this repo, `docs/opprof/`)
+
+**Question.** Do operators face materially different patterns online than in
+offline rectangular benchmarks ("workload-conditioned operator profiling"),
+and if so, where is the recoverable gain? Patched vLLM 0.24.0
+(`patches/vllm-0.24.0-opprof/`), Qwen3-30B-A3B BF16, dash0 H20.
+
+| Phase | Deliverable | Verdict / headline |
+|---|---|---|
+| P0–P2 | Dual-layer instrumentation (always-on Layer-1 telemetry + sampled Layer-2 Kineto) | Overhead **−0.04%** (CI [−0.17, +0.05]) after the compile-factor fix; Layer-2 perturbs 51.3% → sampled-only by design |
+| P3 | 10-pattern × config matrix (`phase3-{protocol,results}.md`) | **H1b PASS** (5/6 evaluable contrasts, Holm p≈0): irregular patterns carry R64 raggedness +23.0 to +44.8 pp over rectangular controls with 8.3–44.7% useful-token efficiency loss. **H1a INCONCLUSIVE** (Layer-2 window representativeness) |
+| P4 | Ranked optimization plan (`phase4-optimization-plan.md`) | #1 prefix-affine routing: **+82.14%** saturation req/s (P08 vs matched P07, 62.3% fewer prefill tokens). #4 MNS is pattern-conditioned: MNS64 gives +3.4/+3.7% on P06/P10 but **−24.27% on P01** — the sign flips by workload |
+| P5 | Mechanism decomposition (`phase5-{protocol,results}.md`) | All four intuitive mechanisms ≈0 at rho=0.60: raggedness 3.8% n.s., capture-size fix +1.6% n.s. (padding 13.74%→2.40% with no E2E gain), prefix ~0. **Real finding:** P3's P10-vs-P04 gap was largely an arrival-uniformization artifact — replaying recorded (bursty) arrival recovers +12.9%; burstiness helps batch formation at low rate |
+| P6 | Cross-version churn, paired 12-cell surface, vLLM 0.20→0.24 (`phase6-{protocol,results}.md`) | Old #2 config TP2/MNS64 **−29.41%** (solo-confirmed, bounded both sides); TP1 plateau **+13.5%** at MNS8; old argmax TP2/MNS32 held (−0.76%). Formal ARGMAX/RANKING/TRAP **INCONCLUSIVE** — 4 cells right-censored/non-monotonic, no imputation |
+
+**P6 mechanism note.** TP2/MNS64's old anchor now fails with decode-batch
+means 11.5–18.1 and 4.6–9.5% of steps executing outside cudagraph coverage;
+all 37 solo primaries had zero preemptions.
+
+**Combined churn claim (paper-usable).** Across one ~2-month engine upgrade
+the surface below the argmax reorganized (rank-2 config −29%, plateau +13%)
+even though the argmax survived → warm-start/transfer from stale tuning
+surfaces is unreliable; every engine upgrade is a retuning trigger.
+
+## Cross-cutting methodological findings (reusable beyond this paper)
+
+1. **Co-location validity is metric-dependent.** 21 exact same-request pairs
+   (co-located vs solo, P6): throughput and operator shares move <3%, but SLO
+   pass rates flip by up to **+92.86 pp** (0.071→1.000) at frontier anchors —
+   feasible/infeasible verdicts invert. Deltas are cell- and anchor-dependent
+   (0 to +92.9 pp), so no fixed correction exists. Rule adopted: SLO-frontier
+   measurements must be solo; mean-type metrics may be co-located behind the
+   pre-registered A-P3-1 validity gate (which rejected 8-way, passed 4-way).
+2. **Uniformizing arrival distorts efficiency** (−12.9% at low rate) in the
+   opposite direction from intuition — offline benchmarks that regularize
+   arrival misestimate real efficiency.
+3. **Env vars hashed into vLLM compile factors** silently cause cold
+   torch.compile caches and ~4% slower artifacts (root cause of our phantom
+   overhead; fixed with a one-line ignore-list entry). Upstream-report
+   candidate.
+4. **Long-context real traces break short-load-calibrated harness
+   assumptions** (drain deadlines, warm-up stabilization); P10/TP2 never
+   stabilizes (36.8% drift).
+
+## Corrections and honest limits — do NOT quote these stale readings
+
+- P3's "P10 is 14.3% worse than P04" is **superseded by P5**: largely a
+  materialization artifact of uniformized arrival. Quote pattern-vs-control
+  raggedness/padding effects (H1b) and the P5-corrected arrival result
+  instead. The remaining P10-vs-P03 gap (~36%) is workload physics, not
+  recoverable waste at this regime.
+- H1a (operator bottleneck-ranking inversions) is **not refuted** — it is
+  inconclusive at the measured regimes; saturation-regime decomposition
+  remains open.
+- P6 formal verdicts are INCONCLUSIVE by right-censoring, not by data
+  invalidity; the −29.41%/+13.5%/−0.76% numbers are bounded and quotable.
+- Only solo-tier SLO numbers are quotable; co-located W1–W3 artifacts are
+  preserved but superseded.
+- The <3% co-location bound is an empirical gate verified at moderate load on
+  specific patterns, not a theorem; re-run the gate before reusing 4-way
+  placement in new regimes.
+- SimFid's SLO-gated 0–0.76% regret reading is post-hoc, not decision-bearing.
+
+## Artifact map
+
+| Path | Content |
+|---|---|
+| `docs/opprof_campaign_state.md` | Full OpProf decision ledger (echoes, amendments, acceptances) |
+| `docs/opprof/phase{0,2}-*.md`, `docs/opprof/patch-design.md` | Recon, patch design, smoke + overhead evidence |
+| `docs/opprof/phase{3,5,6}-protocol.md` | Frozen pre-registered protocols incl. amendments |
+| `docs/opprof/phase{3,5,6}-results.md`, `phase4-optimization-plan.md` | Results; phase6 metrics pinned by SHA-256 `290ba7fc…` |
+| `patches/vllm-0.24.0-opprof/` | 7-patch series + apply.sh + tests (base `ee0da84a`) |
+| `runs/opprof-phase{3,5,6}/` | Decision-bearing evidence (metrics/manifests/validation); raw Layer-1 JSONL streams are git-ignored (507 MB, kept on disk and dash0) |
+| `docs/simulator-fidelity-frontier-20260711.md` | Standalone review: does the data show simulator mis-ranking rigorously? |
+| `replayserve/docs/simfid_*` | SimFid protocols, results, ledger |
+
+## GPU accounting
+
+OpProf total ≈ **22.2 H20-hours** (P0–P5 ≈ 16.5, P6 5.64 of a 6.0 cap).
+SimFid accounting lives in the replayserve ledger. No prompt or generated
+text appears in any committed artifact; the prompt-bearing trace copy stays
+in git-ignored `trace_windows/`.
+
+## Open items (not committed to)
+
+- Bound the 3 right-censored TP4 cells + TP2/MNS16 (~1.6 H20-h, exceeds the
+  P6 cap) to convert ARGMAX/TRAP into formal verdicts.
+- Saturation-regime mechanism decomposition (P5 analogue at high rho).
+- Upstream reports: compile-factor env poisoning; co-location SLO validity.
+- Synthesis of both campaigns into the paper's motivation section.