gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Strengthen fidelity calibration baseline

Browse Source
This commit is contained in:
Gahow Wang
2026-07-14 13:08:45 +08:00
parent 93daf291f6
commit 23142aa359
5 changed files with 881 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
docs/fidelity-aware-harness-headroom-20260714.md
View File

@@ -46,6 +46,35 @@ at 15 seconds it is 88.89% versus 91.67%; at 20 seconds it is 86.11% versus
91.67%, but both 0.95 policies make one false reject. Five seconds is therefore
a training-selected operating point, not a test result.
## Strong simulator-aware calibration baseline
The original nested comparison used the same simulator shortlist but did not
put Frontier's per-anchor prediction in either model. A stronger retrospective
audit now gives both models frozen-calibrated simulated throughput, simulated
SLO pass rate, and simulated feasibility. Under the same leave-one-cell-out
folds, 5-second cutoff, L2 logistic family, regularization 1.0, and threshold
0.95:
| Metric | Sim + outcome | Sim + outcome + instrumentation | Delta |
|---|---:|---:|---:|
| Accuracy | 81.08% | 89.19% | +8.11 pp |
| Balanced accuracy | 72.42% | 81.55% | +9.13 pp |
| Brier score | 0.1058 | 0.0957 | -0.0101 |
| Safe early decisions | 20/37 | 25/37 | +5 |
| Valid full-trial cost reduction | 50.89% | 68.98% | +18.09 pp |
| Residual verification H20-hours | 0.5240 | 0.3310 | -36.84% |
Both 0.95 policies have zero false accept and zero false reject on this
retrospective task. Only three 0.5-threshold classifications differ in favor
of instrumentation and none in favor of the strong baseline; McNemar's exact
two-sided p-value is 0.25. The cell-bootstrap accuracy-delta interval is
`[0.00,+18.18]` percentage points. The result is not robust to regularization:
at 0.1 the strong baseline is more accurate and the instrumentation policy
makes two unsafe decisions; at 10.0 the strong baseline is also more accurate.
Thus the stronger comparison still has enough point-estimate headroom for a
held-out test, but it materially weakens the evidence and makes a prospective
task-level result mandatory.
## Interpretation
There is enough headroom to run a held-out pilot, but not enough evidence to
@@ -73,6 +102,9 @@ with three full repetitions. The registered protocol is
- `runs/fidelity-headroom/prefix-metrics.json`
- `runs/fidelity-headroom/test_analysis.py`
- `runs/fidelity-headroom/test_prefix_analysis.py`
- `runs/fidelity-headroom/analyze_strong_baseline.py`
- `runs/fidelity-headroom/strong-baseline-metrics.json`
- `runs/fidelity-headroom/test_strong_baseline.py`
## Sanity block
@@ -86,6 +118,8 @@ with three full repetitions. The registered protocol is
| Outcome probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
| Instrumentation probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
| Layer-1 streams | 12 | 14,174 records | 58,725 records | 12 | Contiguous, zero drops |
| Matched frozen simulator anchors | 37 | pass rate 0.0688 | pass rate 1.0 | 12 pass-rate values | Every prefix matched exactly once |
| Frozen simulator anchor corpus | 92 | positive throughput | positive throughput | >1 | No duplicate cell/anchor run |
Checked invariants: same folds/model family and cutoff; no full verdict in a
feature; prefix-only Layer-1 slicing; non-negative costs/counters; bounded

41
docs/fidelity-aware-harness-protocol-20260714.md
View File

@@ -52,6 +52,35 @@ difference is Z. The initial family is intentionally simple: a positive result
then demonstrates value in the engine signal rather than capacity in a larger
learner. A sequence model is admissible only as a later, paired ablation.
### Amendment A1: strengthen the calibration baseline before P2
Frozen 2026-07-14 13:08 Asia/Singapore, after P1 launch but before P1
completion or analysis. A baseline audit found that the first frozen P1
models use the simulator only to define candidate order; their feature vectors
do not contain the simulator's per-anchor prediction. This is insufficient
for the stronger term **outcome-only calibration**. P1 therefore remains a
prospective test of the originally frozen cross-workload predictor, but cannot
by itself open a contribution claim.
For P2/P3, both nested models must additionally receive the identical frozen
simulator outputs available at that decision: predicted completed throughput
per GPU, predicted SLO pass rate, and predicted feasibility. The comparison
is consequently `sim + config + workload + real outcome prefix` versus that
exact vector plus real engine state. Simulator features, regularization,
cutoff, and thresholds are frozen before any P2 task. If telemetry does not
improve this stronger baseline, the harness has no independent contribution.
The same audit also separates algorithm cost from benchmark-oracle cost.
Headline method cost includes every action the method would execute online:
simulator profiling/calibration, model onboarding, server startup, warm-up,
real prefix, continuation after abstention, method-requested confirmation,
logging overhead, failures, and cleanup. Exhaustive real-oracle runs and the
extra repetitions used only to construct 2-of-3 evaluation labels are common
benchmark annotation cost; they are reported separately and charged to no
method. A second, deliberately conservative table adds that common cost to
all methods. This prevents both hiding real method cost and making the
percentage gate mathematically depend on offline ground-truth annotation.
The frozen first policy uses a 5-second prefix, L2 regularization 1.0, and a
two-sided abstaining threshold of 0.95: accept at `p(feasible)>=0.95`, reject at
`p(feasible)<=0.05`, otherwise continue the exact same trial to completion.
@@ -64,15 +93,15 @@ therefore not evidence; all claims come from subsequent held-out tasks.
|---|---:|---:|---:|---:|---:|
| Real-only oracle | no | no | full | optional diagnostic | every candidate/anchor |
| Sim top-k + real final | yes | included in full run | full | no decision use | every shortlisted candidate/anchor |
| Outcome-only calibration | yes | yes | yes | no | only on abstention |
| Instrumentation-aware | yes | yes | yes | yes | only on abstention |
| Outcome-only calibration | yes, including its prediction features | yes | yes | no | only on abstention |
| Instrumentation-aware | same prediction features | yes | yes | yes | only on abstention |
Tie buckets are expanded before top-k. `k` is selected on training tasks and
is fixed on held-out tasks; an oracle per-task k is forbidden. Outcome-only
receives all information available outside the engine, including config and
workload features. Instrumentation cannot use any record submitted after the
cutoff. The full label, confirmation votes, simulator error, and later
requests are never model features.
receives all information available outside the engine, including config,
workload, and frozen simulator-prediction features. Instrumentation cannot use
any record submitted after the cutoff. The full label, confirmation votes,
realized simulator error, and later requests are never model features.
## Staged experiment