Fidelity-aware harness headroom audit

Status: HISTORICAL PREMISE DID NOT PASS PROSPECTIVE P1; NO CONTRIBUTION CLAIM.

The audit answers whether engine instrumentation has enough incremental signal to justify a prospective experiment. It does not establish generalization.

Post-run update: the exact-timestamp held-out P1 completed and failed the registered gate. Under the stronger simulator-aware k=2 end-to-end replay, telemetry preserved zero regret but saved only 1.426% online H20-hours versus sim top-k + real final. The current route is closed; see docs/fidelity-aware-harness-p1-report-20260714.md.

Simulator shortlist lower bound

On the frozen 12-cell SimFid task, the strongest calibrated SLO simulator reading places TP2/MNS32 and TP2/MNS64 in the same first tie bucket. Real-final evaluation of that two-cell bucket selects TP2/MNS32 and has zero real regret. A method requiring a real calibration probe plus final verification cannot beat two real cell evaluations on this task. Therefore “better initial selection” is not a viable claim here; the remaining headroom is shorter real verification inside the same shortlist.

Five-second prefix result

The retrospective Phase-6 dataset contains 37 primary anchors across 12 cells. Stable labels use the frozen same-placement 2-of-3 adjudication: 28 feasible and 9 infeasible. Three TP4 primary measurements disagree with their repeated labels, so single-run feasibility is not treated as ground truth.

Using leave-one-cell-out folds, identical L2 logistic models, and a 5-second prefix:

Metric	Outcome-only	Instrumentation-aware	Delta
Accuracy	78.38%	89.19%	+10.81 pp
Balanced accuracy	70.63%	81.55%	+10.92 pp
Brier score	0.1297	0.0901	-0.0396
Correct only in this model	0	4	+4
McNemar exact two-sided p	—	0.125	not significant

At the frozen conservative threshold 0.95, both policies make zero false accepts and zero false rejects on this retrospective set. Outcome-only safely cuts 36.35% of measured primary-trial cost; instrumentation-aware safely cuts 61.10%, an additional 24.75 percentage points. Regularization sensitivity for accuracy delta is [0.00, +10.81] percentage points, so the sign is non-negative but the magnitude is not stable.

Longer prefixes do not strengthen the case monotonically. At 10 seconds, headline accuracy is 91.89% outcome-only versus 89.19% instrumentation-aware; at 15 seconds it is 88.89% versus 91.67%; at 20 seconds it is 86.11% versus 91.67%, but both 0.95 policies make one false reject. Five seconds is therefore a training-selected operating point, not a test result.

Strong simulator-aware calibration baseline

The original nested comparison used the same simulator shortlist but did not put Frontier's per-anchor prediction in either model. A stronger retrospective audit now gives both models frozen-calibrated simulated throughput, simulated SLO pass rate, and simulated feasibility. Under the same leave-one-cell-out folds, 5-second cutoff, L2 logistic family, regularization 1.0, and threshold 0.95:

Metric	Sim + outcome	Sim + outcome + instrumentation	Delta
Accuracy	81.08%	89.19%	+8.11 pp
Balanced accuracy	72.42%	81.55%	+9.13 pp
Brier score	0.1058	0.0957	-0.0101
Safe early decisions	20/37	25/37	+5
Valid full-trial cost reduction	50.89%	68.98%	+18.09 pp
Residual verification H20-hours	0.5240	0.3310	-36.84%

Both 0.95 policies have zero false accept and zero false reject on this retrospective task. Only three 0.5-threshold classifications differ in favor of instrumentation and none in favor of the strong baseline; McNemar's exact two-sided p-value is 0.25. The cell-bootstrap accuracy-delta interval is [0.00,+18.18] percentage points. The result is not robust to regularization: at 0.1 the strong baseline is more accurate and the instrumentation policy makes two unsafe decisions; at 10.0 the strong baseline is also more accurate. Thus the stronger comparison still has enough point-estimate headroom for a held-out test, but it materially weakens the evidence and makes a prospective task-level result mandatory.

Interpretation

There is enough headroom to run a held-out pilot, but not enough evidence to claim the harness contribution:

the 5-second cost gap is operationally large;
only four paired classifications differ, so significance is absent;
all examples share one workload/SLO/engine task;
completion timestamps are reconstructed from arrival + TTFT + TPOT rather than recorded directly;
three adjudication disagreements are concentrated in transient TP4 runs;
outcome-only already recovers the simulator shortlist oracle with very few real cells.

The next experiment must therefore freeze the 5-second model and threshold, record exact monotonic completions, use a held-out trace, and label each anchor with three full repetitions. The registered protocol is docs/fidelity-aware-harness-protocol-20260714.md.

Artifacts

runs/fidelity-headroom/analyze_existing.py
runs/fidelity-headroom/metrics.json
runs/fidelity-headroom/analyze_prefixes.py
runs/fidelity-headroom/prefix-metrics.json
runs/fidelity-headroom/test_analysis.py
runs/fidelity-headroom/test_prefix_analysis.py
runs/fidelity-headroom/analyze_strong_baseline.py
runs/fidelity-headroom/strong-baseline-metrics.json
runs/fidelity-headroom/test_strong_baseline.py

Sanity block

Family	n	Min	Max	Distinct	Invariant/result
Real SimFid cell scores	12	1.2833	3.2833	7	Non-negative; not identical
Prefix examples at 5 s	37	5 s	5 s	1 expected	All 12 cells represented
Adjudicated labels	37	0	1	2	28 positive / 9 negative
Primary/adjudicated disagreement	37	0	1	2	3 TP4 disagreements retained
Full primary elapsed time	37	14.566 s	62.064 s	37	Every 5 s prefix is in range
Outcome probability	37	in `[0,1]`	in `[0,1]`	>1	Checked before metrics
Instrumentation probability	37	in `[0,1]`	in `[0,1]`	>1	Checked before metrics
Layer-1 streams	12	14,174 records	58,725 records	12	Contiguous, zero drops
Matched frozen simulator anchors	37	pass rate 0.0688	pass rate 1.0	12 pass-rate values	Every prefix matched exactly once
Frozen simulator anchor corpus	92	positive throughput	positive throughput	>1	No duplicate cell/anchor run

Checked invariants: same folds/model family and cutoff; no full verdict in a feature; prefix-only Layer-1 slicing; non-negative costs/counters; bounded ratios/probabilities; both labels present; per-config results not identical; tie expansion before top-k; no imputation of non-monotonic frontiers. The main limitation is reconstructed request completion time, explicitly marked on all 37 five-second examples.

6.8 KiB Raw Blame History