# SimFid + OpProf Campaign Overview

Date: 2026-07-13. Status: both campaigns **CLOSED**. This is the entry-point
summary; every claim below links to a frozen protocol and a results document.
Decision history lives in `docs/opprof_campaign_state.md` (this repo) and
`replayserve/docs/simfid_campaign_state.md`.

## Why these campaigns exist

The paper's motivation requires three evidence-backed claims:

1. **Simulators cannot replace real replay** for config tuning → SimFid.
2. **The leverage point is engine-knob configuration, not operator
   implementation**, at the measured regimes → OpProf P3–P5.
3. **Tuning is genuinely hard**: the surface is workload-conditioned
   (sign-flips defeat defaults), real evaluation is expensive (measured GPU
   cost), and the surface churns across engine versions → OpProf P4/P6.

All three now rest on pre-registered protocols with frozen decision rules,
Holm-corrected contrasts, and no imputation over censored data.

## Campaign 1 — SimFid (replayserve repo)

**Question.** Can a Frontier-class simulator replace real replay for AITuner
config tuning? **Verdict: NOT ADEQUATE** under the pre-declared decision rule.

- S2-E (3-config TP family, same model/HW): ranking perfect after throughput
  calibration, but latency is uncalibrated (TTFT p95 sim/real 0.30–0.38, TPOT
  0.63–0.79) → false-feasible counterexample across a 50 ms TPOT SLO.
- S2-R-b (12-cell TP×MNS surface, exact C1 workload replayed in Frontier,
  184 runs, 0 failures): the decision-bearing frozen-calibrated
  throughput-proxy reading picks TP1/MNS64 → **30.46% real top-1 regret**;
  trap detection 3/6; LOAO 0/92. All three adequacy components FAIL.
- Post-hoc diagnostic (NOT decision-bearing): an SLO-gated reading achieves
  0–0.76% regret, tau-b 0.967, trap 6/6 — anchor-level SLO errors partially
  cancel at per-cell peaks. This motivates hybrid sim-prune + real-final
  designs but must always carry the post-hoc label.

Sources: `replayserve/docs/simfid_s2e_report.md`,
`replayserve/docs/simfid_s2rb_results.md`.

## Campaign 2 — OpProf (this repo, `docs/opprof/`)

**Question.** Do operators face materially different patterns online than in
offline rectangular benchmarks ("workload-conditioned operator profiling"),
and if so, where is the recoverable gain? Patched vLLM 0.24.0
(`patches/vllm-0.24.0-opprof/`), Qwen3-30B-A3B BF16, dash0 H20.

| Phase | Deliverable | Verdict / headline |
|---|---|---|
| P0–P2 | Dual-layer instrumentation (always-on Layer-1 telemetry + sampled Layer-2 Kineto) | Overhead **−0.04%** (CI [−0.17, +0.05]) after the compile-factor fix; Layer-2 perturbs 51.3% → sampled-only by design |
| P3 | 10-pattern × config matrix (`phase3-{protocol,results}.md`) | **H1b PASS** (5/6 evaluable contrasts, Holm p≈0): irregular patterns carry R64 raggedness +23.0 to +44.8 pp over rectangular controls with 8.3–44.7% useful-token efficiency loss. **H1a INCONCLUSIVE** (Layer-2 window representativeness) |
| P4 | Ranked optimization plan (`phase4-optimization-plan.md`) | #1 prefix-affine routing: **+82.14%** saturation req/s (P08 vs matched P07, 62.3% fewer prefill tokens). #4 MNS is pattern-conditioned: MNS64 gives +3.4/+3.7% on P06/P10 but **−24.27% on P01** — the sign flips by workload |
| P5 | Mechanism decomposition (`phase5-{protocol,results}.md`) | All four intuitive mechanisms ≈0 at rho=0.60: raggedness 3.8% n.s., capture-size fix +1.6% n.s. (padding 13.74%→2.40% with no E2E gain), prefix ~0. **Real finding:** P3's P10-vs-P04 gap was largely an arrival-uniformization artifact — replaying recorded (bursty) arrival recovers +12.9%; burstiness helps batch formation at low rate |
| P6 | Cross-version churn, paired 12-cell surface, vLLM 0.20→0.24 (`phase6-{protocol,results}.md`) | Old #2 config TP2/MNS64 **−29.41%** (solo-confirmed, bounded both sides); TP1 plateau **+13.5%** at MNS8; old argmax TP2/MNS32 held (−0.76%). Formal ARGMAX/RANKING/TRAP **INCONCLUSIVE** — 4 cells right-censored/non-monotonic, no imputation |

**P6 mechanism note.** TP2/MNS64's old anchor now fails with decode-batch
means 11.5–18.1 and 4.6–9.5% of steps executing outside cudagraph coverage;
all 37 solo primaries had zero preemptions.

**Combined churn claim (paper-usable).** Across one ~2-month engine upgrade
the surface below the argmax reorganized (rank-2 config −29%, plateau +13%)
even though the argmax survived → warm-start/transfer from stale tuning
surfaces is unreliable; every engine upgrade is a retuning trigger.

## Cross-cutting methodological findings (reusable beyond this paper)

1. **Co-location validity is metric-dependent.** 21 exact same-request pairs
   (co-located vs solo, P6): throughput and operator shares move <3%, but SLO
   pass rates flip by up to **+92.86 pp** (0.071→1.000) at frontier anchors —
   feasible/infeasible verdicts invert. Deltas are cell- and anchor-dependent
   (0 to +92.9 pp), so no fixed correction exists. Rule adopted: SLO-frontier
   measurements must be solo; mean-type metrics may be co-located behind the
   pre-registered A-P3-1 validity gate (which rejected 8-way, passed 4-way).
2. **Uniformizing arrival distorts efficiency** (−12.9% at low rate) in the
   opposite direction from intuition — offline benchmarks that regularize
   arrival misestimate real efficiency.
3. **Env vars hashed into vLLM compile factors** silently cause cold
   torch.compile caches and ~4% slower artifacts (root cause of our phantom
   overhead; fixed with a one-line ignore-list entry). Upstream-report
   candidate.
4. **Long-context real traces break short-load-calibrated harness
   assumptions** (drain deadlines, warm-up stabilization); P10/TP2 never
   stabilizes (36.8% drift).

## Corrections and honest limits — do NOT quote these stale readings

- P3's "P10 is 14.3% worse than P04" is **superseded by P5**: largely a
  materialization artifact of uniformized arrival. Quote pattern-vs-control
  raggedness/padding effects (H1b) and the P5-corrected arrival result
  instead. The remaining P10-vs-P03 gap (~36%) is workload physics, not
  recoverable waste at this regime.
- H1a (operator bottleneck-ranking inversions) is **not refuted** — it is
  inconclusive at the measured regimes; saturation-regime decomposition
  remains open.
- P6 formal verdicts are INCONCLUSIVE by right-censoring, not by data
  invalidity; the −29.41%/+13.5%/−0.76% numbers are bounded and quotable.
- Only solo-tier SLO numbers are quotable; co-located W1–W3 artifacts are
  preserved but superseded.
- The <3% co-location bound is an empirical gate verified at moderate load on
  specific patterns, not a theorem; re-run the gate before reusing 4-way
  placement in new regimes.
- SimFid's SLO-gated 0–0.76% regret reading is post-hoc, not decision-bearing.

## Artifact map

| Path | Content |
|---|---|
| `docs/opprof_campaign_state.md` | Full OpProf decision ledger (echoes, amendments, acceptances) |
| `docs/opprof/phase{0,2}-*.md`, `docs/opprof/patch-design.md` | Recon, patch design, smoke + overhead evidence |
| `docs/opprof/phase{3,5,6}-protocol.md` | Frozen pre-registered protocols incl. amendments |
| `docs/opprof/phase{3,5,6}-results.md`, `phase4-optimization-plan.md` | Results; phase6 metrics pinned by SHA-256 `290ba7fc…` |
| `patches/vllm-0.24.0-opprof/` | 7-patch series + apply.sh + tests (base `ee0da84a`) |
| `runs/opprof-phase{3,5,6}/` | Decision-bearing evidence (metrics/manifests/validation); raw Layer-1 JSONL streams are git-ignored (507 MB, kept on disk and dash0) |
| `docs/simulator-fidelity-frontier-20260711.md` | Standalone review: does the data show simulator mis-ranking rigorously? |
| `replayserve/docs/simfid_*` | SimFid protocols, results, ledger |

## GPU accounting

OpProf total ≈ **22.2 H20-hours** (P0–P5 ≈ 16.5, P6 5.64 of a 6.0 cap).
SimFid accounting lives in the replayserve ledger. No prompt or generated
text appears in any committed artifact; the prompt-bearing trace copy stays
in git-ignored `trace_windows/`.

## Open items (not committed to)

- Bound the 3 right-censored TP4 cells + TP2/MNS16 (~1.6 H20-h, exceeds the
  P6 cap) to convert ARGMAX/TRAP into formal verdicts.
- Saturation-regime mechanism decomposition (P5 analogue at high rho).
- Upstream reports: compile-factor env poisoning; co-location SLO validity.
- Synthesis of both campaigns into the paper's motivation section.