gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Strengthen fidelity calibration baseline

Browse Source
This commit is contained in:
Gahow Wang
2026-07-14 13:08:45 +08:00
parent 93daf291f6
commit 23142aa359
5 changed files with 881 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
docs/fidelity-aware-harness-headroom-20260714.md
View File

@@ -46,6 +46,35 @@ at 15 seconds it is 88.89% versus 91.67%; at 20 seconds it is 86.11% versus
91.67%, but both 0.95 policies make one false reject. Five seconds is therefore
a training-selected operating point, not a test result.
## Strong simulator-aware calibration baseline
The original nested comparison used the same simulator shortlist but did not
put Frontier's per-anchor prediction in either model. A stronger retrospective
audit now gives both models frozen-calibrated simulated throughput, simulated
SLO pass rate, and simulated feasibility. Under the same leave-one-cell-out
folds, 5-second cutoff, L2 logistic family, regularization 1.0, and threshold
0.95:
| Metric | Sim + outcome | Sim + outcome + instrumentation | Delta |
|---|---:|---:|---:|
| Accuracy | 81.08% | 89.19% | +8.11 pp |
| Balanced accuracy | 72.42% | 81.55% | +9.13 pp |
| Brier score | 0.1058 | 0.0957 | -0.0101 |
| Safe early decisions | 20/37 | 25/37 | +5 |
| Valid full-trial cost reduction | 50.89% | 68.98% | +18.09 pp |
| Residual verification H20-hours | 0.5240 | 0.3310 | -36.84% |
Both 0.95 policies have zero false accept and zero false reject on this
retrospective task. Only three 0.5-threshold classifications differ in favor
of instrumentation and none in favor of the strong baseline; McNemar's exact
two-sided p-value is 0.25. The cell-bootstrap accuracy-delta interval is
`[0.00,+18.18]` percentage points. The result is not robust to regularization:
at 0.1 the strong baseline is more accurate and the instrumentation policy
makes two unsafe decisions; at 10.0 the strong baseline is also more accurate.
Thus the stronger comparison still has enough point-estimate headroom for a
held-out test, but it materially weakens the evidence and makes a prospective
task-level result mandatory.
## Interpretation
There is enough headroom to run a held-out pilot, but not enough evidence to
@@ -73,6 +102,9 @@ with three full repetitions. The registered protocol is
- `runs/fidelity-headroom/prefix-metrics.json`
- `runs/fidelity-headroom/test_analysis.py`
- `runs/fidelity-headroom/test_prefix_analysis.py`
- `runs/fidelity-headroom/analyze_strong_baseline.py`
- `runs/fidelity-headroom/strong-baseline-metrics.json`
- `runs/fidelity-headroom/test_strong_baseline.py`
## Sanity block
@@ -86,6 +118,8 @@ with three full repetitions. The registered protocol is
| Outcome probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
| Instrumentation probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
| Layer-1 streams | 12 | 14,174 records | 58,725 records | 12 | Contiguous, zero drops |
| Matched frozen simulator anchors | 37 | pass rate 0.0688 | pass rate 1.0 | 12 pass-rate values | Every prefix matched exactly once |
| Frozen simulator anchor corpus | 92 | positive throughput | positive throughput | >1 | No duplicate cell/anchor run |
Checked invariants: same folds/model family and cutoff; no full verdict in a
feature; prefix-only Layer-1 slicing; non-negative costs/counters; bounded