aituner/docs/opprof-simfid-overview-20260713.md

SimFid + OpProf Campaign Overview

Date: 2026-07-13. Status: both campaigns CLOSED. This is the entry-point summary; every claim below links to a frozen protocol and a results document. Decision history lives in docs/opprof_campaign_state.md (this repo) and replayserve/docs/simfid_campaign_state.md.

Why these campaigns exist

The paper's motivation requires three evidence-backed claims:

1. Simulators cannot replace real replay for config tuning → SimFid.
2. The leverage point is engine-knob configuration, not operator implementation, at the measured regimes → OpProf P3–P5.
3. Tuning is genuinely hard: the surface is workload-conditioned (sign-flips defeat defaults), real evaluation is expensive (measured GPU cost), and the surface churns across engine versions → OpProf P4/P6.

All three now rest on pre-registered protocols with frozen decision rules, Holm-corrected contrasts, and no imputation over censored data.

Campaign 1 — SimFid (replayserve repo)

Question. Can a Frontier-class simulator replace real replay for AITuner config tuning? Verdict: NOT ADEQUATE under the pre-declared decision rule.

• S2-E (3-config TP family, same model/HW): ranking perfect after throughput calibration, but latency is uncalibrated (TTFT p95 sim/real 0.30–0.38, TPOT 0.63–0.79) → false-feasible counterexample across a 50 ms TPOT SLO.
• S2-R-b (12-cell TP×MNS surface, exact C1 workload replayed in Frontier, 184 runs, 0 failures): the decision-bearing frozen-calibrated throughput-proxy reading picks TP1/MNS64 → 30.46% real top-1 regret; trap detection 3/6; LOAO 0/92. All three adequacy components FAIL.
• Post-hoc diagnostic (NOT decision-bearing): an SLO-gated reading achieves 0–0.76% regret, tau-b 0.967, trap 6/6 — anchor-level SLO errors partially cancel at per-cell peaks. This motivates hybrid sim-prune + real-final designs but must always carry the post-hoc label.

Sources: replayserve/docs/simfid_s2e_report.md, replayserve/docs/simfid_s2rb_results.md.

Campaign 2 — OpProf (this repo, docs/opprof/)

Question. Do operators face materially different patterns online than in offline rectangular benchmarks ("workload-conditioned operator profiling"), and if so, where is the recoverable gain? Patched vLLM 0.24.0 (patches/vllm-0.24.0-opprof/), Qwen3-30B-A3B BF16, dash0 H20.

Phase	Deliverable	Verdict / headline
P0–P2	Dual-layer instrumentation (always-on Layer-1 telemetry + sampled Layer-2 Kineto)	Overhead −0.04% (CI [−0.17, +0.05]) after the compile-factor fix; Layer-2 perturbs 51.3% → sampled-only by design
P3	10-pattern × config matrix (phase3-{protocol,results}.md)	H1b PASS (5/6 evaluable contrasts, Holm p≈0): irregular patterns carry R64 raggedness +23.0 to +44.8 pp over rectangular controls with 8.3–44.7% useful-token efficiency loss. H1a INCONCLUSIVE (Layer-2 window representativeness)
P4	Ranked optimization plan (phase4-optimization-plan.md)	#1 prefix-affine routing: +82.14% saturation req/s (P08 vs matched P07, 62.3% fewer prefill tokens). #4 MNS is pattern-conditioned: MNS64 gives +3.4/+3.7% on P06/P10 but −24.27% on P01 — the sign flips by workload
P5	Mechanism decomposition (phase5-{protocol,results}.md)	All four intuitive mechanisms ≈0 at rho=0.60: raggedness 3.8% n.s., capture-size fix +1.6% n.s. (padding 13.74%→2.40% with no E2E gain), prefix ~0. Real finding: P3's P10-vs-P04 gap was largely an arrival-uniformization artifact — replaying recorded (bursty) arrival recovers +12.9%; burstiness helps batch formation at low rate
P6	Cross-version churn, paired 12-cell surface, vLLM 0.20→0.24 (phase6-{protocol,results}.md)	Old #2 config TP2/MNS64 −29.41% (solo-confirmed, bounded both sides); TP1 plateau +13.5% at MNS8; old argmax TP2/MNS32 held (−0.76%). Formal ARGMAX/RANKING/TRAP INCONCLUSIVE — 4 cells right-censored/non-monotonic, no imputation

P6 mechanism note. TP2/MNS64's old anchor now fails with decode-batch means 11.5–18.1 and 4.6–9.5% of steps executing outside cudagraph coverage; all 37 solo primaries had zero preemptions.

Combined churn claim (paper-usable). Across one ~2-month engine upgrade the surface below the argmax reorganized (rank-2 config −29%, plateau +13%) even though the argmax survived → warm-start/transfer from stale tuning surfaces is unreliable; every engine upgrade is a retuning trigger.

Cross-cutting methodological findings (reusable beyond this paper)

1. Co-location validity is metric-dependent. 21 exact same-request pairs (co-located vs solo, P6): throughput and operator shares move <3%, but SLO pass rates flip by up to +92.86 pp (0.071→1.000) at frontier anchors — feasible/infeasible verdicts invert. Deltas are cell- and anchor-dependent (0 to +92.9 pp), so no fixed correction exists. Rule adopted: SLO-frontier measurements must be solo; mean-type metrics may be co-located behind the pre-registered A-P3-1 validity gate (which rejected 8-way, passed 4-way).
2. Uniformizing arrival distorts efficiency (−12.9% at low rate) in the opposite direction from intuition — offline benchmarks that regularize arrival misestimate real efficiency.
3. Env vars hashed into vLLM compile factors silently cause cold torch.compile caches and ~4% slower artifacts (root cause of our phantom overhead; fixed with a one-line ignore-list entry). Upstream-report candidate.
4. Long-context real traces break short-load-calibrated harness assumptions (drain deadlines, warm-up stabilization); P10/TP2 never stabilizes (36.8% drift).

Corrections and honest limits — do NOT quote these stale readings

• P3's "P10 is 14.3% worse than P04" is superseded by P5: largely a materialization artifact of uniformized arrival. Quote pattern-vs-control raggedness/padding effects (H1b) and the P5-corrected arrival result instead. The remaining P10-vs-P03 gap (~36%) is workload physics, not recoverable waste at this regime.
• H1a (operator bottleneck-ranking inversions) is not refuted — it is inconclusive at the measured regimes; saturation-regime decomposition remains open.
• P6 formal verdicts are INCONCLUSIVE by right-censoring, not by data invalidity; the −29.41%/+13.5%/−0.76% numbers are bounded and quotable.
• Only solo-tier SLO numbers are quotable; co-located W1–W3 artifacts are preserved but superseded.
• The <3% co-location bound is an empirical gate verified at moderate load on specific patterns, not a theorem; re-run the gate before reusing 4-way placement in new regimes.
• SimFid's SLO-gated 0–0.76% regret reading is post-hoc, not decision-bearing.

Artifact map

Path	Content
docs/opprof_campaign_state.md	Full OpProf decision ledger (echoes, amendments, acceptances)
docs/opprof/phase{0,2}-*.md, docs/opprof/patch-design.md	Recon, patch design, smoke + overhead evidence
docs/opprof/phase{3,5,6}-protocol.md	Frozen pre-registered protocols incl. amendments
docs/opprof/phase{3,5,6}-results.md, phase4-optimization-plan.md	Results; phase6 metrics pinned by SHA-256 290ba7fc…
patches/vllm-0.24.0-opprof/	7-patch series + apply.sh + tests (base ee0da84a)
runs/opprof-phase{3,5,6}/	Decision-bearing evidence (metrics/manifests/validation); raw Layer-1 JSONL streams are git-ignored (507 MB, kept on disk and dash0)
docs/simulator-fidelity-frontier-20260711.md	Standalone review: does the data show simulator mis-ranking rigorously?
replayserve/docs/simfid_*	SimFid protocols, results, ledger

GPU accounting

OpProf total ≈ 22.2 H20-hours (P0–P5 ≈ 16.5, P6 5.64 of a 6.0 cap). SimFid accounting lives in the replayserve ledger. No prompt or generated text appears in any committed artifact; the prompt-bearing trace copy stays in git-ignored trace_windows/.

Open items (not committed to)

• Bound the 3 right-censored TP4 cells + TP2/MNS16 (~1.6 H20-h, exceeds the P6 cap) to convert ARGMAX/TRAP into formal verdicts.
• Saturation-regime mechanism decomposition (P5 analogue at high rho).
• Upstream reports: compile-factor env poisoning; co-location SLO validity.
• Synthesis of both campaigns into the paper's motivation section.