Add fidelity-aware verification pilot

docs/fidelity-aware-harness-headroom-20260714.md

@@ -0,0 +1,95 @@
+# Fidelity-aware harness headroom audit
+
+Status: **PROMISING PREMISE, NO CONTRIBUTION CLAIM**.
+
+The audit answers whether engine instrumentation has enough incremental signal
+to justify a prospective experiment.  It does not establish generalization.
+
+## Simulator shortlist lower bound
+
+On the frozen 12-cell SimFid task, the strongest calibrated SLO simulator
+reading places TP2/MNS32 and TP2/MNS64 in the same first tie bucket.  Real-final
+evaluation of that two-cell bucket selects TP2/MNS32 and has zero real regret.
+A method requiring a real calibration probe plus final verification cannot beat
+two real cell evaluations on this task.  Therefore “better initial selection”
+is not a viable claim here; the remaining headroom is shorter real verification
+inside the same shortlist.
+
+## Five-second prefix result
+
+The retrospective Phase-6 dataset contains 37 primary anchors across 12 cells.
+Stable labels use the frozen same-placement 2-of-3 adjudication: 28 feasible and
+9 infeasible.  Three TP4 primary measurements disagree with their repeated
+labels, so single-run feasibility is not treated as ground truth.
+
+Using leave-one-cell-out folds, identical L2 logistic models, and a 5-second
+prefix:
+
+| Metric | Outcome-only | Instrumentation-aware | Delta |
+|---|---:|---:|---:|
+| Accuracy | 78.38% | 89.19% | +10.81 pp |
+| Balanced accuracy | 70.63% | 81.55% | +10.92 pp |
+| Brier score | 0.1297 | 0.0901 | -0.0396 |
+| Correct only in this model | 0 | 4 | +4 |
+| McNemar exact two-sided p | — | 0.125 | not significant |
+
+At the frozen conservative threshold 0.95, both policies make zero false
+accepts and zero false rejects on this retrospective set.  Outcome-only safely
+cuts 36.35% of measured primary-trial cost; instrumentation-aware safely cuts
+61.10%, an additional 24.75 percentage points.  Regularization sensitivity for
+accuracy delta is `[0.00, +10.81]` percentage points, so the sign is
+non-negative but the magnitude is not stable.
+
+Longer prefixes do not strengthen the case monotonically.  At 10 seconds,
+headline accuracy is 91.89% outcome-only versus 89.19% instrumentation-aware;
+at 15 seconds it is 88.89% versus 91.67%; at 20 seconds it is 86.11% versus
+91.67%, but both 0.95 policies make one false reject.  Five seconds is therefore
+a training-selected operating point, not a test result.
+
+## Interpretation
+
+There is enough headroom to run a held-out pilot, but not enough evidence to
+claim the harness contribution:
+
+- the 5-second cost gap is operationally large;
+- only four paired classifications differ, so significance is absent;
+- all examples share one workload/SLO/engine task;
+- completion timestamps are reconstructed from arrival + TTFT + TPOT rather
+  than recorded directly;
+- three adjudication disagreements are concentrated in transient TP4 runs;
+- outcome-only already recovers the simulator shortlist oracle with very few
+  real cells.
+
+The next experiment must therefore freeze the 5-second model and threshold,
+record exact monotonic completions, use a held-out trace, and label each anchor
+with three full repetitions.  The registered protocol is
+`docs/fidelity-aware-harness-protocol-20260714.md`.
+
+## Artifacts
+
+- `runs/fidelity-headroom/analyze_existing.py`
+- `runs/fidelity-headroom/metrics.json`
+- `runs/fidelity-headroom/analyze_prefixes.py`
+- `runs/fidelity-headroom/prefix-metrics.json`
+- `runs/fidelity-headroom/test_analysis.py`
+- `runs/fidelity-headroom/test_prefix_analysis.py`
+
+## Sanity block
+
+| Family | n | Min | Max | Distinct | Invariant/result |
+|---|---:|---:|---:|---:|---|
+| Real SimFid cell scores | 12 | 1.2833 | 3.2833 | 7 | Non-negative; not identical |
+| Prefix examples at 5 s | 37 | 5 s | 5 s | 1 expected | All 12 cells represented |
+| Adjudicated labels | 37 | 0 | 1 | 2 | 28 positive / 9 negative |
+| Primary/adjudicated disagreement | 37 | 0 | 1 | 2 | 3 TP4 disagreements retained |
+| Full primary elapsed time | 37 | 14.566 s | 62.064 s | 37 | Every 5 s prefix is in range |
+| Outcome probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
+| Instrumentation probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
+| Layer-1 streams | 12 | 14,174 records | 58,725 records | 12 | Contiguous, zero drops |
+
+Checked invariants: same folds/model family and cutoff; no full verdict in a
+feature; prefix-only Layer-1 slicing; non-negative costs/counters; bounded
+ratios/probabilities; both labels present; per-config results not identical;
+tie expansion before top-k; no imputation of non-monotonic frontiers.  The main
+limitation is reconstructed request completion time, explicitly marked on all
+37 five-second examples.

docs/fidelity-aware-harness-protocol-20260714.md

@@ -0,0 +1,193 @@
+# Fidelity-aware real-verification harness protocol
+
+Status: **PRE-REGISTERED STAGED EVALUATION; CONTRIBUTION NOT YET ESTABLISHED**.
+
+Date frozen: 2026-07-14 (Asia/Singapore).
+
+## Research question and contribution bar
+
+The harness has an independent systems contribution only if engine-internal
+instrumentation improves a tuning decision beyond what is already achievable
+with a simulator shortlist and external benchmark outcomes.  The intended
+claim is therefore deliberately stronger than “telemetry explains a run”:
+
+> Given the same simulator ranking, the same candidate order, and the same
+> short real-GPU probe, a learned instrumentation-aware verifier reaches a
+> configuration with at most 5% real SLO-goodput regret using materially fewer
+> H20-hours than both (a) simulator top-k followed by full real evaluation and
+> (b) an outcome-only verifier given exactly the same probe.
+
+The paper-facing gate is:
+
+- at least 20% lower real-verification H20-hours than outcome-only calibration;
+- at least 30% lower real-verification H20-hours than simulator top-k plus full
+  real final evaluation;
+- paired 95% task-bootstrap confidence interval for the outcome-only cost
+  reduction strictly above zero;
+- selected-configuration SLO-goodput regret at most 5% on every headline task;
+- no false-safe early accept in the pilot and at most 1% in the expanded suite;
+- profiling, warm-up, confirmation, instrumentation, and failed-run costs are
+  included rather than amortized away.  An amortized profile-cost view may be
+  reported only as a secondary result.
+
+If these conditions fail, instrumentation remains a debugging facility.  It is
+not an independent tuning-harness contribution.
+
+## What is learned, and what is not a rule
+
+The decision target is a stable, repeated real verdict, not a hand-authored
+diagnosis such as “queue length above N means reject.”  Each anchor receives
+three full real repetitions and a frozen 2-of-3 feasibility label.  A nested
+pair of regularized models predicts that label from a fixed prefix:
+
+- **Outcome-only input X:** configuration, offered rate, admitted/completed
+  progress, observed TTFT/TPOT margins, failures, and known workload lengths.
+- **Instrumentation input Z:** the same X plus generic engine state: running and
+  waiting queues, decode-batch shape, KV usage, graph mode and padding, prefill
+  share, preemptions, and model-step rate.
+
+Both models use the same L2 logistic family, train split, standardization,
+regularization, cutoff, and probability threshold.  The only experimental
+difference is Z.  The initial family is intentionally simple: a positive result
+then demonstrates value in the engine signal rather than capacity in a larger
+learner.  A sequence model is admissible only as a later, paired ablation.
+
+The frozen first policy uses a 5-second prefix, L2 regularization 1.0, and a
+two-sided abstaining threshold of 0.95: accept at `p(feasible)>=0.95`, reject at
+`p(feasible)<=0.05`, otherwise continue the exact same trial to completion.
+Threshold and cutoff were selected on the historical training task and are
+therefore not evidence; all claims come from subsequent held-out tasks.
+
+## Fair baselines
+
+| Method | Simulator | 5-second real prefix | External outcomes | Engine state | Full real continuation |
+|---|---:|---:|---:|---:|---:|
+| Real-only oracle | no | no | full | optional diagnostic | every candidate/anchor |
+| Sim top-k + real final | yes | included in full run | full | no decision use | every shortlisted candidate/anchor |
+| Outcome-only calibration | yes | yes | yes | no | only on abstention |
+| Instrumentation-aware | yes | yes | yes | yes | only on abstention |
+
+Tie buckets are expanded before top-k.  `k` is selected on training tasks and
+is fixed on held-out tasks; an oracle per-task k is forbidden.  Outcome-only
+receives all information available outside the engine, including config and
+workload features.  Instrumentation cannot use any record submitted after the
+cutoff.  The full label, confirmation votes, simulator error, and later
+requests are never model features.
+
+## Staged experiment
+
+### R0: historical premise and headroom audit
+
+The frozen SimFid surface has 12 cells.  The strongest calibrated SLO simulator
+reading has a top tie bucket `{TP2/MNS32, TP2/MNS64}`; full real evaluation of
+those two cells already finds the oracle with zero regret.  Consequently this
+single task cannot demonstrate a selection-count advantage: any method needing
+one real calibration probe and one real final verification has a lower bound of
+two real cells.
+
+The viable estimand is instead the duration and number of full real frontier
+evaluations inside a fixed shortlist.  Historical Phase-6 prefixes are analyzed
+only as training/premise data.  Their request completion times are reconstructed
+from arrival, TTFT, TPOT, and token count, so they cannot support a final claim.
+
+### P1: exact-timestamp prospective pilot
+
+- Engine/model/hardware: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B, one solo
+  server/client on dash0, NVIDIA H20, `TP in {1,2,4}`.
+- Held-out workload: `chat_w20260312_1000`, 60-second replay after the frozen
+  0.1 time scale, raw input `[0,8192]`, exactly 128 output tokens.
+- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, 95% request pass rate.
+- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
+- Per cell: one attainable low offered rate near 0.85x the historical v0.24
+  frontier and one high rate near 1.25x.  The exact threshold and selected
+  request hashes are frozen by a CPU preflight before launch.
+- Each cell uses a fresh server, the accepted long-request warm-up, one
+  unmeasured full-window burn-in, then three repetitions per rate.  Rate order
+  alternates and reverses across cells to prevent a fixed warm-state/order
+  confound.
+- The first repetition supplies the exact prefix.  All three repetitions supply
+  the 2-of-3 label.  Every request records a monotonic completion timestamp;
+  Layer-1 records are cut at the same monotonic boundary.
+- Placement is serialized.  Co-location is forbidden because Phase 6 observed
+  up to 92.86 percentage-point pass-rate shifts under co-location.
+- Hard cap: 3.5 H20-hours, including startup, warm-up, burn-in, all repetitions,
+  failures, and cleanup.  Projected cap violation stops before the next cell.
+
+P1 opens P2 only if all data invariants pass and instrumentation-aware has zero
+false accept/reject, is no worse than outcome-only, and either makes at least
+three additional correct early decisions or improves total valid trial-cost
+reduction by at least 15 absolute percentage points.  The pilot is a gate, not
+paper evidence.
+
+### P2: held-out task replication
+
+If P1 passes, freeze the model and run at least six independent task groups:
+three trace windows spanning distinct date/slot combinations and two SLO
+regimes.  No task used for threshold/model selection enters the headline test.
+The candidate surface is the full 12-cell `TP={1,2,4} x MNS={8,16,32,64}`
+surface.  Splits are by complete task, never by anchor or request.  A task-level
+paired bootstrap (10,000 repetitions, fixed seed) estimates cost and regret
+intervals.  Non-monotonic or split 2-of-3 anchors remain explicit; no frontier
+is imputed.
+
+### P3: end-to-end shortlist and search replay
+
+For each P2 task, run the same frozen simulator and tie-expanded top-k policy.
+Replay the real binary/frontier search under all three verification policies:
+full real, outcome-only, and instrumentation-aware.  The policy consumes only
+prefixes that would have been available at that decision point.  Report:
+
+- selected cell and real SLO-goodput regret;
+- number of real cells, anchors, and confirmations;
+- measured H20-hours and wall time;
+- false accept, false reject, and abstention counts;
+- profile, startup/warm-up, probe, full-continuation, confirmation, logging, and
+  failure cost breakdowns.
+
+### P4: simulator-rank-error attribution
+
+This phase distinguishes an outdated implementation/profile from a structural
+simulator limitation.  For each held-out task compare:
+
+1. the original simulator/profile;
+2. a version-matched re-profiled simulator;
+3. a trajectory-conditioned run supplied with the realized arrival and request
+   length sequence;
+4. outcome-only residual calibration;
+5. instrumentation-aware residual calibration.
+
+The engine trace is extended only as needed with a worker-level step UID and
+CUDA-event duration, because current async submit-to-complete spans overlap and
+are not GPU step time.  Residuals are decomposed into operator-profile error,
+scheduler/state error, and run-to-run noise.  If re-profiling alone restores the
+ranking, the old 30% loss was an implementation/profile defect.  If exact
+profiles and realized trajectories still mis-rank cells, and the residual is
+systematically explained by queue/KV/graph/batch state unavailable to the
+simulator, that is evidence of a structural state-abstraction gap.  Correlation
+alone is not called causal.
+
+## Failure modes that reject the route
+
+- Outcome-only matches or beats instrumentation-aware under the same cutoff.
+- Instrumentation gains average accuracy but introduces false-safe decisions.
+- Gains disappear under task-level rather than request/anchor-level splitting.
+- Savings come only from excluding startup, warm-up, profiling, confirmations,
+  or failed trials.
+- A different cutoff/threshold must be selected after seeing each test task.
+- The simulator top-k baseline already reaches the target with equal or lower
+  total H20-hours.
+- Exact instrumentation overhead exceeds 1% throughput or materially changes
+  p95/p99 latency.
+- Results depend on TP4 transient/non-monotonic trials and do not replicate on
+  held-out tasks.
+
+## Data sanity contract
+
+Every analysis ends with n, min/max, distinct count, label balance, and these
+invariants: non-negative counters/costs; probabilities and ratios in `[0,1]`;
+per-config results not all identical; timestamps monotonic; every prefix record
+at or before its cutoff; selected request ID/arrival/length hashes stable across
+repetitions; exact 128-token completion or counted failure; no dropped Layer-1
+records; 2-of-3 labels reproducible; no co-resident GPU process; total H20-hours
+below the hard cap; final GPUs idle.  A red flag is reported first and blocks
+the contribution claim.