Fidelity-aware harness headroom audit

Status: PROMISING PREMISE, NO CONTRIBUTION CLAIM.

The audit answers whether engine instrumentation has enough incremental signal to justify a prospective experiment. It does not establish generalization.

Simulator shortlist lower bound

On the frozen 12-cell SimFid task, the strongest calibrated SLO simulator reading places TP2/MNS32 and TP2/MNS64 in the same first tie bucket. Real-final evaluation of that two-cell bucket selects TP2/MNS32 and has zero real regret. A method requiring a real calibration probe plus final verification cannot beat two real cell evaluations on this task. Therefore “better initial selection” is not a viable claim here; the remaining headroom is shorter real verification inside the same shortlist.

Five-second prefix result

The retrospective Phase-6 dataset contains 37 primary anchors across 12 cells. Stable labels use the frozen same-placement 2-of-3 adjudication: 28 feasible and 9 infeasible. Three TP4 primary measurements disagree with their repeated labels, so single-run feasibility is not treated as ground truth.

Using leave-one-cell-out folds, identical L2 logistic models, and a 5-second prefix:

Metric	Outcome-only	Instrumentation-aware	Delta
Accuracy	78.38%	89.19%	+10.81 pp
Balanced accuracy	70.63%	81.55%	+10.92 pp
Brier score	0.1297	0.0901	-0.0396
Correct only in this model	0	4	+4
McNemar exact two-sided p	—	0.125	not significant

At the frozen conservative threshold 0.95, both policies make zero false accepts and zero false rejects on this retrospective set. Outcome-only safely cuts 36.35% of measured primary-trial cost; instrumentation-aware safely cuts 61.10%, an additional 24.75 percentage points. Regularization sensitivity for accuracy delta is [0.00, +10.81] percentage points, so the sign is non-negative but the magnitude is not stable.

Longer prefixes do not strengthen the case monotonically. At 10 seconds, headline accuracy is 91.89% outcome-only versus 89.19% instrumentation-aware; at 15 seconds it is 88.89% versus 91.67%; at 20 seconds it is 86.11% versus 91.67%, but both 0.95 policies make one false reject. Five seconds is therefore a training-selected operating point, not a test result.

Interpretation

There is enough headroom to run a held-out pilot, but not enough evidence to claim the harness contribution:

the 5-second cost gap is operationally large;
only four paired classifications differ, so significance is absent;
all examples share one workload/SLO/engine task;
completion timestamps are reconstructed from arrival + TTFT + TPOT rather than recorded directly;
three adjudication disagreements are concentrated in transient TP4 runs;
outcome-only already recovers the simulator shortlist oracle with very few real cells.

The next experiment must therefore freeze the 5-second model and threshold, record exact monotonic completions, use a held-out trace, and label each anchor with three full repetitions. The registered protocol is docs/fidelity-aware-harness-protocol-20260714.md.

Artifacts

runs/fidelity-headroom/analyze_existing.py
runs/fidelity-headroom/metrics.json
runs/fidelity-headroom/analyze_prefixes.py
runs/fidelity-headroom/prefix-metrics.json
runs/fidelity-headroom/test_analysis.py
runs/fidelity-headroom/test_prefix_analysis.py

Sanity block

Family	n	Min	Max	Distinct	Invariant/result
Real SimFid cell scores	12	1.2833	3.2833	7	Non-negative; not identical
Prefix examples at 5 s	37	5 s	5 s	1 expected	All 12 cells represented
Adjudicated labels	37	0	1	2	28 positive / 9 negative
Primary/adjudicated disagreement	37	0	1	2	3 TP4 disagreements retained
Full primary elapsed time	37	14.566 s	62.064 s	37	Every 5 s prefix is in range
Outcome probability	37	in `[0,1]`	in `[0,1]`	>1	Checked before metrics
Instrumentation probability	37	in `[0,1]`	in `[0,1]`	>1	Checked before metrics
Layer-1 streams	12	14,174 records	58,725 records	12	Contiguous, zero drops

Checked invariants: same folds/model family and cutoff; no full verdict in a feature; prefix-only Layer-1 slicing; non-negative costs/counters; bounded ratios/probabilities; both labels present; per-config results not identical; tie expansion before top-k; no imputation of non-monotonic frontiers. The main limitation is reconstructed request completion time, explicitly marked on all 37 five-second examples.

4.5 KiB Raw Blame History