Gahow Wang
6db7308558
Entry-point summary of both closed campaigns: verdicts, corrected readings (P5 supersedes P3's P10-vs-P04 magnitude), methodological findings (co-location SLO validity, arrival uniformization, compile- factor env poisoning), artifact map, and GPU accounting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
132 lines
8.2 KiB
Markdown
132 lines
8.2 KiB
Markdown
# SimFid + OpProf Campaign Overview
|
||
|
||
Date: 2026-07-13. Status: both campaigns **CLOSED**. This is the entry-point
|
||
summary; every claim below links to a frozen protocol and a results document.
|
||
Decision history lives in `docs/opprof_campaign_state.md` (this repo) and
|
||
`replayserve/docs/simfid_campaign_state.md`.
|
||
|
||
## Why these campaigns exist
|
||
|
||
The paper's motivation requires three evidence-backed claims:
|
||
|
||
1. **Simulators cannot replace real replay** for config tuning → SimFid.
|
||
2. **The leverage point is engine-knob configuration, not operator
|
||
implementation**, at the measured regimes → OpProf P3–P5.
|
||
3. **Tuning is genuinely hard**: the surface is workload-conditioned
|
||
(sign-flips defeat defaults), real evaluation is expensive (measured GPU
|
||
cost), and the surface churns across engine versions → OpProf P4/P6.
|
||
|
||
All three now rest on pre-registered protocols with frozen decision rules,
|
||
Holm-corrected contrasts, and no imputation over censored data.
|
||
|
||
## Campaign 1 — SimFid (replayserve repo)
|
||
|
||
**Question.** Can a Frontier-class simulator replace real replay for AITuner
|
||
config tuning? **Verdict: NOT ADEQUATE** under the pre-declared decision rule.
|
||
|
||
- S2-E (3-config TP family, same model/HW): ranking perfect after throughput
|
||
calibration, but latency is uncalibrated (TTFT p95 sim/real 0.30–0.38, TPOT
|
||
0.63–0.79) → false-feasible counterexample across a 50 ms TPOT SLO.
|
||
- S2-R-b (12-cell TP×MNS surface, exact C1 workload replayed in Frontier,
|
||
184 runs, 0 failures): the decision-bearing frozen-calibrated
|
||
throughput-proxy reading picks TP1/MNS64 → **30.46% real top-1 regret**;
|
||
trap detection 3/6; LOAO 0/92. All three adequacy components FAIL.
|
||
- Post-hoc diagnostic (NOT decision-bearing): an SLO-gated reading achieves
|
||
0–0.76% regret, tau-b 0.967, trap 6/6 — anchor-level SLO errors partially
|
||
cancel at per-cell peaks. This motivates hybrid sim-prune + real-final
|
||
designs but must always carry the post-hoc label.
|
||
|
||
Sources: `replayserve/docs/simfid_s2e_report.md`,
|
||
`replayserve/docs/simfid_s2rb_results.md`.
|
||
|
||
## Campaign 2 — OpProf (this repo, `docs/opprof/`)
|
||
|
||
**Question.** Do operators face materially different patterns online than in
|
||
offline rectangular benchmarks ("workload-conditioned operator profiling"),
|
||
and if so, where is the recoverable gain? Patched vLLM 0.24.0
|
||
(`patches/vllm-0.24.0-opprof/`), Qwen3-30B-A3B BF16, dash0 H20.
|
||
|
||
| Phase | Deliverable | Verdict / headline |
|
||
|---|---|---|
|
||
| P0–P2 | Dual-layer instrumentation (always-on Layer-1 telemetry + sampled Layer-2 Kineto) | Overhead **−0.04%** (CI [−0.17, +0.05]) after the compile-factor fix; Layer-2 perturbs 51.3% → sampled-only by design |
|
||
| P3 | 10-pattern × config matrix (`phase3-{protocol,results}.md`) | **H1b PASS** (5/6 evaluable contrasts, Holm p≈0): irregular patterns carry R64 raggedness +23.0 to +44.8 pp over rectangular controls with 8.3–44.7% useful-token efficiency loss. **H1a INCONCLUSIVE** (Layer-2 window representativeness) |
|
||
| P4 | Ranked optimization plan (`phase4-optimization-plan.md`) | #1 prefix-affine routing: **+82.14%** saturation req/s (P08 vs matched P07, 62.3% fewer prefill tokens). #4 MNS is pattern-conditioned: MNS64 gives +3.4/+3.7% on P06/P10 but **−24.27% on P01** — the sign flips by workload |
|
||
| P5 | Mechanism decomposition (`phase5-{protocol,results}.md`) | All four intuitive mechanisms ≈0 at rho=0.60: raggedness 3.8% n.s., capture-size fix +1.6% n.s. (padding 13.74%→2.40% with no E2E gain), prefix ~0. **Real finding:** P3's P10-vs-P04 gap was largely an arrival-uniformization artifact — replaying recorded (bursty) arrival recovers +12.9%; burstiness helps batch formation at low rate |
|
||
| P6 | Cross-version churn, paired 12-cell surface, vLLM 0.20→0.24 (`phase6-{protocol,results}.md`) | Old #2 config TP2/MNS64 **−29.41%** (solo-confirmed, bounded both sides); TP1 plateau **+13.5%** at MNS8; old argmax TP2/MNS32 held (−0.76%). Formal ARGMAX/RANKING/TRAP **INCONCLUSIVE** — 4 cells right-censored/non-monotonic, no imputation |
|
||
|
||
**P6 mechanism note.** TP2/MNS64's old anchor now fails with decode-batch
|
||
means 11.5–18.1 and 4.6–9.5% of steps executing outside cudagraph coverage;
|
||
all 37 solo primaries had zero preemptions.
|
||
|
||
**Combined churn claim (paper-usable).** Across one ~2-month engine upgrade
|
||
the surface below the argmax reorganized (rank-2 config −29%, plateau +13%)
|
||
even though the argmax survived → warm-start/transfer from stale tuning
|
||
surfaces is unreliable; every engine upgrade is a retuning trigger.
|
||
|
||
## Cross-cutting methodological findings (reusable beyond this paper)
|
||
|
||
1. **Co-location validity is metric-dependent.** 21 exact same-request pairs
|
||
(co-located vs solo, P6): throughput and operator shares move <3%, but SLO
|
||
pass rates flip by up to **+92.86 pp** (0.071→1.000) at frontier anchors —
|
||
feasible/infeasible verdicts invert. Deltas are cell- and anchor-dependent
|
||
(0 to +92.9 pp), so no fixed correction exists. Rule adopted: SLO-frontier
|
||
measurements must be solo; mean-type metrics may be co-located behind the
|
||
pre-registered A-P3-1 validity gate (which rejected 8-way, passed 4-way).
|
||
2. **Uniformizing arrival distorts efficiency** (−12.9% at low rate) in the
|
||
opposite direction from intuition — offline benchmarks that regularize
|
||
arrival misestimate real efficiency.
|
||
3. **Env vars hashed into vLLM compile factors** silently cause cold
|
||
torch.compile caches and ~4% slower artifacts (root cause of our phantom
|
||
overhead; fixed with a one-line ignore-list entry). Upstream-report
|
||
candidate.
|
||
4. **Long-context real traces break short-load-calibrated harness
|
||
assumptions** (drain deadlines, warm-up stabilization); P10/TP2 never
|
||
stabilizes (36.8% drift).
|
||
|
||
## Corrections and honest limits — do NOT quote these stale readings
|
||
|
||
- P3's "P10 is 14.3% worse than P04" is **superseded by P5**: largely a
|
||
materialization artifact of uniformized arrival. Quote pattern-vs-control
|
||
raggedness/padding effects (H1b) and the P5-corrected arrival result
|
||
instead. The remaining P10-vs-P03 gap (~36%) is workload physics, not
|
||
recoverable waste at this regime.
|
||
- H1a (operator bottleneck-ranking inversions) is **not refuted** — it is
|
||
inconclusive at the measured regimes; saturation-regime decomposition
|
||
remains open.
|
||
- P6 formal verdicts are INCONCLUSIVE by right-censoring, not by data
|
||
invalidity; the −29.41%/+13.5%/−0.76% numbers are bounded and quotable.
|
||
- Only solo-tier SLO numbers are quotable; co-located W1–W3 artifacts are
|
||
preserved but superseded.
|
||
- The <3% co-location bound is an empirical gate verified at moderate load on
|
||
specific patterns, not a theorem; re-run the gate before reusing 4-way
|
||
placement in new regimes.
|
||
- SimFid's SLO-gated 0–0.76% regret reading is post-hoc, not decision-bearing.
|
||
|
||
## Artifact map
|
||
|
||
| Path | Content |
|
||
|---|---|
|
||
| `docs/opprof_campaign_state.md` | Full OpProf decision ledger (echoes, amendments, acceptances) |
|
||
| `docs/opprof/phase{0,2}-*.md`, `docs/opprof/patch-design.md` | Recon, patch design, smoke + overhead evidence |
|
||
| `docs/opprof/phase{3,5,6}-protocol.md` | Frozen pre-registered protocols incl. amendments |
|
||
| `docs/opprof/phase{3,5,6}-results.md`, `phase4-optimization-plan.md` | Results; phase6 metrics pinned by SHA-256 `290ba7fc…` |
|
||
| `patches/vllm-0.24.0-opprof/` | 7-patch series + apply.sh + tests (base `ee0da84a`) |
|
||
| `runs/opprof-phase{3,5,6}/` | Decision-bearing evidence (metrics/manifests/validation); raw Layer-1 JSONL streams are git-ignored (507 MB, kept on disk and dash0) |
|
||
| `docs/simulator-fidelity-frontier-20260711.md` | Standalone review: does the data show simulator mis-ranking rigorously? |
|
||
| `replayserve/docs/simfid_*` | SimFid protocols, results, ledger |
|
||
|
||
## GPU accounting
|
||
|
||
OpProf total ≈ **22.2 H20-hours** (P0–P5 ≈ 16.5, P6 5.64 of a 6.0 cap).
|
||
SimFid accounting lives in the replayserve ledger. No prompt or generated
|
||
text appears in any committed artifact; the prompt-bearing trace copy stays
|
||
in git-ignored `trace_windows/`.
|
||
|
||
## Open items (not committed to)
|
||
|
||
- Bound the 3 right-censored TP4 cells + TP2/MNS16 (~1.6 H20-h, exceeds the
|
||
P6 cap) to convert ARGMAX/TRAP into formal verdicts.
|
||
- Saturation-regime mechanism decomposition (P5 analogue at high rho).
|
||
- Upstream reports: compile-factor env poisoning; co-location SLO validity.
|
||
- Synthesis of both campaigns into the paper's motivation section.
|