136 lines
6.8 KiB
Markdown
136 lines
6.8 KiB
Markdown
# Fidelity-aware harness headroom audit
|
|
|
|
Status: **HISTORICAL PREMISE DID NOT PASS PROSPECTIVE P1; NO CONTRIBUTION CLAIM**.
|
|
|
|
The audit answers whether engine instrumentation has enough incremental signal
|
|
to justify a prospective experiment. It does not establish generalization.
|
|
|
|
Post-run update: the exact-timestamp held-out P1 completed and failed the
|
|
registered gate. Under the stronger simulator-aware `k=2` end-to-end replay,
|
|
telemetry preserved zero regret but saved only 1.426% online H20-hours versus
|
|
sim top-k + real final. The current route is closed; see
|
|
`docs/fidelity-aware-harness-p1-report-20260714.md`.
|
|
|
|
## Simulator shortlist lower bound
|
|
|
|
On the frozen 12-cell SimFid task, the strongest calibrated SLO simulator
|
|
reading places TP2/MNS32 and TP2/MNS64 in the same first tie bucket. Real-final
|
|
evaluation of that two-cell bucket selects TP2/MNS32 and has zero real regret.
|
|
A method requiring a real calibration probe plus final verification cannot beat
|
|
two real cell evaluations on this task. Therefore “better initial selection”
|
|
is not a viable claim here; the remaining headroom is shorter real verification
|
|
inside the same shortlist.
|
|
|
|
## Five-second prefix result
|
|
|
|
The retrospective Phase-6 dataset contains 37 primary anchors across 12 cells.
|
|
Stable labels use the frozen same-placement 2-of-3 adjudication: 28 feasible and
|
|
9 infeasible. Three TP4 primary measurements disagree with their repeated
|
|
labels, so single-run feasibility is not treated as ground truth.
|
|
|
|
Using leave-one-cell-out folds, identical L2 logistic models, and a 5-second
|
|
prefix:
|
|
|
|
| Metric | Outcome-only | Instrumentation-aware | Delta |
|
|
|---|---:|---:|---:|
|
|
| Accuracy | 78.38% | 89.19% | +10.81 pp |
|
|
| Balanced accuracy | 70.63% | 81.55% | +10.92 pp |
|
|
| Brier score | 0.1297 | 0.0901 | -0.0396 |
|
|
| Correct only in this model | 0 | 4 | +4 |
|
|
| McNemar exact two-sided p | — | 0.125 | not significant |
|
|
|
|
At the frozen conservative threshold 0.95, both policies make zero false
|
|
accepts and zero false rejects on this retrospective set. Outcome-only safely
|
|
cuts 36.35% of measured primary-trial cost; instrumentation-aware safely cuts
|
|
61.10%, an additional 24.75 percentage points. Regularization sensitivity for
|
|
accuracy delta is `[0.00, +10.81]` percentage points, so the sign is
|
|
non-negative but the magnitude is not stable.
|
|
|
|
Longer prefixes do not strengthen the case monotonically. At 10 seconds,
|
|
headline accuracy is 91.89% outcome-only versus 89.19% instrumentation-aware;
|
|
at 15 seconds it is 88.89% versus 91.67%; at 20 seconds it is 86.11% versus
|
|
91.67%, but both 0.95 policies make one false reject. Five seconds is therefore
|
|
a training-selected operating point, not a test result.
|
|
|
|
## Strong simulator-aware calibration baseline
|
|
|
|
The original nested comparison used the same simulator shortlist but did not
|
|
put Frontier's per-anchor prediction in either model. A stronger retrospective
|
|
audit now gives both models frozen-calibrated simulated throughput, simulated
|
|
SLO pass rate, and simulated feasibility. Under the same leave-one-cell-out
|
|
folds, 5-second cutoff, L2 logistic family, regularization 1.0, and threshold
|
|
0.95:
|
|
|
|
| Metric | Sim + outcome | Sim + outcome + instrumentation | Delta |
|
|
|---|---:|---:|---:|
|
|
| Accuracy | 81.08% | 89.19% | +8.11 pp |
|
|
| Balanced accuracy | 72.42% | 81.55% | +9.13 pp |
|
|
| Brier score | 0.1058 | 0.0957 | -0.0101 |
|
|
| Safe early decisions | 20/37 | 25/37 | +5 |
|
|
| Valid full-trial cost reduction | 50.89% | 68.98% | +18.09 pp |
|
|
| Residual verification H20-hours | 0.5240 | 0.3310 | -36.84% |
|
|
|
|
Both 0.95 policies have zero false accept and zero false reject on this
|
|
retrospective task. Only three 0.5-threshold classifications differ in favor
|
|
of instrumentation and none in favor of the strong baseline; McNemar's exact
|
|
two-sided p-value is 0.25. The cell-bootstrap accuracy-delta interval is
|
|
`[0.00,+18.18]` percentage points. The result is not robust to regularization:
|
|
at 0.1 the strong baseline is more accurate and the instrumentation policy
|
|
makes two unsafe decisions; at 10.0 the strong baseline is also more accurate.
|
|
Thus the stronger comparison still has enough point-estimate headroom for a
|
|
held-out test, but it materially weakens the evidence and makes a prospective
|
|
task-level result mandatory.
|
|
|
|
## Interpretation
|
|
|
|
There is enough headroom to run a held-out pilot, but not enough evidence to
|
|
claim the harness contribution:
|
|
|
|
- the 5-second cost gap is operationally large;
|
|
- only four paired classifications differ, so significance is absent;
|
|
- all examples share one workload/SLO/engine task;
|
|
- completion timestamps are reconstructed from arrival + TTFT + TPOT rather
|
|
than recorded directly;
|
|
- three adjudication disagreements are concentrated in transient TP4 runs;
|
|
- outcome-only already recovers the simulator shortlist oracle with very few
|
|
real cells.
|
|
|
|
The next experiment must therefore freeze the 5-second model and threshold,
|
|
record exact monotonic completions, use a held-out trace, and label each anchor
|
|
with three full repetitions. The registered protocol is
|
|
`docs/fidelity-aware-harness-protocol-20260714.md`.
|
|
|
|
## Artifacts
|
|
|
|
- `runs/fidelity-headroom/analyze_existing.py`
|
|
- `runs/fidelity-headroom/metrics.json`
|
|
- `runs/fidelity-headroom/analyze_prefixes.py`
|
|
- `runs/fidelity-headroom/prefix-metrics.json`
|
|
- `runs/fidelity-headroom/test_analysis.py`
|
|
- `runs/fidelity-headroom/test_prefix_analysis.py`
|
|
- `runs/fidelity-headroom/analyze_strong_baseline.py`
|
|
- `runs/fidelity-headroom/strong-baseline-metrics.json`
|
|
- `runs/fidelity-headroom/test_strong_baseline.py`
|
|
|
|
## Sanity block
|
|
|
|
| Family | n | Min | Max | Distinct | Invariant/result |
|
|
|---|---:|---:|---:|---:|---|
|
|
| Real SimFid cell scores | 12 | 1.2833 | 3.2833 | 7 | Non-negative; not identical |
|
|
| Prefix examples at 5 s | 37 | 5 s | 5 s | 1 expected | All 12 cells represented |
|
|
| Adjudicated labels | 37 | 0 | 1 | 2 | 28 positive / 9 negative |
|
|
| Primary/adjudicated disagreement | 37 | 0 | 1 | 2 | 3 TP4 disagreements retained |
|
|
| Full primary elapsed time | 37 | 14.566 s | 62.064 s | 37 | Every 5 s prefix is in range |
|
|
| Outcome probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
|
|
| Instrumentation probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
|
|
| Layer-1 streams | 12 | 14,174 records | 58,725 records | 12 | Contiguous, zero drops |
|
|
| Matched frozen simulator anchors | 37 | pass rate 0.0688 | pass rate 1.0 | 12 pass-rate values | Every prefix matched exactly once |
|
|
| Frozen simulator anchor corpus | 92 | positive throughput | positive throughput | >1 | No duplicate cell/anchor run |
|
|
|
|
Checked invariants: same folds/model family and cutoff; no full verdict in a
|
|
feature; prefix-only Layer-1 slicing; non-negative costs/counters; bounded
|
|
ratios/probabilities; both labels present; per-config results not identical;
|
|
tie expansion before top-k; no imputation of non-monotonic frontiers. The main
|
|
limitation is reconstructed request completion time, explicitly marked on all
|
|
37 five-second examples.
|