diff --git a/docs/fidelity-aware-harness-headroom-20260714.md b/docs/fidelity-aware-harness-headroom-20260714.md index 8f11346..6de9328 100644 --- a/docs/fidelity-aware-harness-headroom-20260714.md +++ b/docs/fidelity-aware-harness-headroom-20260714.md @@ -1,10 +1,16 @@ # Fidelity-aware harness headroom audit -Status: **PROMISING PREMISE, NO CONTRIBUTION CLAIM**. +Status: **HISTORICAL PREMISE DID NOT PASS PROSPECTIVE P1; NO CONTRIBUTION CLAIM**. The audit answers whether engine instrumentation has enough incremental signal to justify a prospective experiment. It does not establish generalization. +Post-run update: the exact-timestamp held-out P1 completed and failed the +registered gate. Under the stronger simulator-aware `k=2` end-to-end replay, +telemetry preserved zero regret but saved only 1.426% online H20-hours versus +sim top-k + real final. The current route is closed; see +`docs/fidelity-aware-harness-p1-report-20260714.md`. + ## Simulator shortlist lower bound On the frozen 12-cell SimFid task, the strongest calibrated SLO simulator diff --git a/docs/fidelity-aware-harness-p1-report-20260714.md b/docs/fidelity-aware-harness-p1-report-20260714.md new file mode 100644 index 0000000..b996dca --- /dev/null +++ b/docs/fidelity-aware-harness-p1-report-20260714.md @@ -0,0 +1,242 @@ +# Fidelity-aware harness P1 result + +Status: **REGISTERED ROUTE REJECTED; DO NOT OPEN P2/P3 FOR THE CURRENT METHOD**. + +Date: 2026-07-14 (Asia/Singapore). + +## Outcome + +The registered five-second instrumentation-aware verifier did not pass P1. +The stronger simulator-aware comparison also failed the independent +contribution bar. On the frozen `k=2` end-to-end replay: + +- `sim top-k + real final` selected the real oracle with zero regret; +- instrumentation-aware also selected the oracle, but reduced online H20-hours + by only **1.426%** (1.329% when the prior failed attempt is added to both); +- the required reduction was 30% versus full real final and 20% versus a safe + outcome-only calibrator; +- the outcome-only calibrator was not safe: it rejected the true best cell, so + its apparent cost saving is not a deployable comparison. + +This rejects the claim that the **current joint logistic verifier**, trained on +one historical workload, gives the harness an independent tuning contribution. +It does not prove that engine telemetry contains no useful signal. Telemetry +improved held-out classification and removed unsafe decisions, but did not turn +that signal into meaningful end-to-end tuning-cost reduction. + +## Frozen setup + +- Host: `dash0`, 8 NVIDIA H20 GPUs; cells were serialized and used TP1, TP2, + or TP4 without co-resident serving jobs. +- Engine/model: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16. +- Workload: held-out `chat_w20260312_1000`, seven disjoint repeat bands, + 60-second replay after 0.1 time scaling, input `[0,8192]`, exactly 128 output + tokens. +- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, request pass rate at least 0.95. +- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64. +- Per cell: burn-in, three low-rate repeats, and three high-rate repeats. The + first repeat supplied the five-second prefix; 2-of-3 supplied its label. +- Models: the registered pair used config/workload/outcome versus the same + vector plus Layer-1 engine telemetry. The strengthened pair additionally + gave both models identical frozen Frontier throughput, SLO pass-rate, and + feasibility predictions. +- Policy: accept at `p>=0.95`, reject at `p<=0.05`, otherwise continue the same + trial. Model, cutoff, threshold, role order, request hashes, and cap were + frozen before their applicable evaluation. + +The first launch failed its warm-up input-count validation before a measured +anchor. It cost 0.020552 H20-hours. The corrected primary attempt cost +1.722112 H20-hours, so aggregate campaign cost was **1.742664 H20-hours**, below +the 3.5 cap. The fix changed only warm-up validation; formal request counts and +hash checks were unchanged. + +## P1 labels are not an artificial easy split + +The 12 adjudicated anchor labels contain 7 feasible and 5 infeasible examples. +They are not simply “low feasible, high infeasible”: + +- TP2/MNS64 high was feasible in all three repeats; +- TP4/MNS64 low and high were feasible in all six repeats; +- TP4/MNS16 low and high were infeasible in all six repeats. + +That last pair creates a large real MNS interaction under an otherwise matched +TP4 configuration. Frontier correctly predicted TP4/MNS64 high as feasible, +but incorrectly predicted TP4/MNS16 low as feasible. It also incorrectly +predicted TP1/MNS64 high as feasible. Overall simulator-only feasibility was +10/12 correct: 83.33% accuracy, with two false-feasible predictions and no +false-infeasible prediction. + +The two false-feasible cases expose the intended latent-state problem. At five +seconds, all 26 completed TP4/MNS16-low requests and all 9 completed +TP1/MNS64-high requests still passed their SLO, although both full anchors were +infeasible. External outcomes had not yet exposed the future failure; queue, +running-batch, and scheduler state existed before the tail outcome. This is +mechanistic evidence that instrumentation can be useful, not evidence that the +current learned policy uses it well enough. + +## Registered and strengthened prefix results + +At the frozen 0.95 policy threshold: + +| Comparison | Accuracy | Balanced acc. | Early decisions | False accept | False reject | Valid primary-trial saving | +|---|---:|---:|---:|---:|---:|---:| +| Registered outcome-only | 41.67% | 50.00% | 6/12 | 0 | 2 | invalid | +| Registered + telemetry | 66.67% | 71.43% | 4/12 | 0 | 0 | 11.44% | +| Strong sim + outcome | 66.67% | 68.57% | 5/12 | 0 | 1 | invalid | +| Strong sim + outcome + telemetry | 83.33% | 85.71% | 4/12 | 0 | 0 | 11.44% | + +For the strong pair, telemetry was correct on two examples where the baseline +was wrong and lost none; McNemar's exact two-sided p-value is 0.5 at `n=12`. +This is a safety/classification improvement, not a cost contribution. The +registered instrumentation policy made two fewer early decisions than its +baseline, so it failed the registered `+3 decisions or +15 percentage points` +incremental gate. + +The result is not robust to the frozen regularization sensitivity: + +| L2 lambda | Sim+outcome acc. | +telemetry acc. | Base policy errors | Telemetry policy errors | Base saving | Telemetry saving | +|---:|---:|---:|---:|---:|---:|---:| +| 0.1 | 41.67% | 75.00% | 4 | 2 | invalid | invalid | +| 1.0 | 66.67% | 83.33% | 1 | 0 | invalid | 11.44% | +| 10.0 | 83.33% | 83.33% | 0 | 0 | 0.00% | 5.98% | + +Consequently the positive classification delta is neither statistically nor +hyperparameter robust. + +## End-to-end shortlist result + +Frontier's simulator-feasible ranking on the tested P1 surface was: + +| Rank | Cell / anchor | Sim throughput/GPU | Real feasible | Real offered goodput/GPU | +|---:|---|---:|---:|---:| +| 1 | TP4/MNS64 high | 3.0718 | yes | 3.1250 | +| 2 | TP1/MNS64 high | 2.8823 | no | 2.9833 | +| 3 | TP2/MNS64 high | 2.8096 | yes | 2.8750 | +| 4 | TP4/MNS16 low | 2.0866 | no | 2.1250 | +| 5 | TP1/MNS8 low | 1.9806 | yes | 2.0333 | +| 6 | TP2/MNS8 low | 1.8637 | yes | 1.9083 | + +The real oracle was TP4/MNS64 high at 3.125 req/s/GPU. Cost includes an +inferred per-cell startup/warm-up/burn-in/cleanup component and the selected +anchor; benchmark-only 2-of-3 annotation intervals are removed. Gaps around +annotation runs remain in the shared setup term, making this a conservative +method-cost estimate. + +| Frozen policy (`k=2`) | Online H20-hours | + prior failure | Real regret | Safety | Saving vs full | +|---|---:|---:|---:|---|---:| +| Sim top-2 + real final | 0.281383 | 0.301935 | 0.00% | valid | — | +| Sim + outcome prefix | 0.214664 | 0.235216 | no selected cell | 1 false reject | invalid | +| Sim + outcome + telemetry | 0.277370 | 0.297922 | 0.00% | valid | **1.426%** | + +For `k=1`, the simulator already picked the oracle and telemetry abstained, so +its saving was exactly 0%. For `k=3`, telemetry remained correct but saved +1.007%. The result therefore does not depend on an unfavorable choice of `k`. + +Across all six cells, a generous replay that removes benchmark-only repeat +intervals estimates 0.944244 H20-hours for the full online workflow. The +instrumentation policy saved 0.043467 H20-hours: 11.44% of primary trial time, +but only **4.603%** of end-to-end online cost. Even a post-hoc oracle symmetric +threshold cannot make the current telemetry model reach the contribution bar; +its best zero-error envelope saves at most 5.69% of inferred online cost. A +strong outcome-only model at another post-hoc regularization/threshold can save +16.13%, which further prevents attributing a unique advantage to telemetry. +These oracle-threshold numbers are diagnostics only and are not test evidence. + +## Why the learned verifier did not generalize + +The training corpus has only 37 anchors from one workload/SLO task. P1 shows +large covariate shift: + +- sim+outcome: 12/192 feature values exceed 3 training standard deviations and + 4 exceed 5; maximum absolute z-score is 10.36; +- sim+outcome+telemetry: 19/396 exceed 3 and 9 exceed 5; +- the largest shifts include admitted input-length mean (10.36), waiting state + (7.77), running maximum (6.38), and decode-batch maximum (6.08). + +Coefficient attribution shows that the input-length feature dominates several +wrong feasible-anchor logits. Because all training examples share one task, +the joint classifier can learn incidental within-task correlation and override +a correct simulator prior on TP2/MNS64-high and TP4/MNS64-high. This is a +supported diagnosis of model/data insufficiency; it is not a causal proof that +one feature alone caused the P1 failure. + +More importantly, retuning lambda, threshold, features, or cutoff on P1 and +then calling P1 a held-out result would violate calibration/evaluation +separation. P1 may now be used only as development data. + +## Decision and the only defensible reopening condition + +Do not run registered P2/P3 with the current model. It failed the predeclared +gate on the favorable primary-trial denominator and is even farther from the +bar under end-to-end cost. Spending six-task headline GPU budget on the same +method would be metric shopping, not replication. + +A new route may be opened only as a new hypothesis: + +1. Replace the joint classifier with a **simulator-residual verifier**. The + simulator prediction remains an explicit prior; nested outcome-only and + telemetry models learn when that prior is wrong, rather than freely + relearning feasibility and overriding it under workload shift. +2. Train on multiple complete workload/SLO tasks. SLO thresholds and target + pass rate must be explicit inputs; splits are by complete task. +3. Calibrate abstention with task-level risk control. No threshold is selected + on a headline task, and “never early decide” is included as the safe + outcome-only baseline. +4. Treat Phase 6 and P1 as development only, freeze the residual architecture, + features, cutoff, threshold, simulator reading, and `k`, then use entirely + new trace windows for a new gate. + +This reopening is justified only if development data show both (a) the +simulator's errors are predictable from pre-outcome engine state and (b) a +simulator-preserving residual model does not corrupt correct simulator +predictions. It is a new project decision, not a continuation automatically +authorized by P1. + +## Benchmark audit + +| Audit item | Verdict | Severity | Evidence / disposition | +|---|---|---|---| +| Calibration set separate from P1 | PASS | — | Phase 6/0311 trained; P1/0312 tested | +| Strong simulator-aware baseline | PASS | — | Identical Frontier features in both nested models | +| Sim top-k + real-final E2E baseline | PASS | — | Frozen `k=2`, tie expansion, measured setup/continuation cost | +| Multiple independent headline tasks | NEEDS EVIDENCE | Blocking for a positive claim | P1 gate failed; P2 correctly not opened | +| Statistical significance | NEEDS EVIDENCE | Blocking for a positive claim | n=12 anchors from one task; McNemar p=0.5 | +| Hyperparameter robustness | FAIL | Blocking | Lambda sensitivity changes safety and relative result | +| Full resource accounting | PASS for P1 | — | Failures, startup/warm-up/burn-in, continuation and annotation separated | +| Avoid post-test retuning | PASS only if route stops | Blocking if violated | P1 is now development-only | +| Selective winning-workload reporting | PASS | — | Negative P1 and TP/MNS losing cases retained | + +Overall recommendation: **Block the current independent harness contribution +claim.** + +## Artifacts + +- Registered protocol: `docs/fidelity-aware-harness-protocol-20260714.md` +- Historical headroom: `docs/fidelity-aware-harness-headroom-20260714.md` +- Registered P1 analysis: `runs/fidelity-headroom/analyze_pilot.py` +- Strong P1 analysis: `runs/fidelity-headroom/analyze_strong_pilot.py` +- E2E shortlist replay: `runs/fidelity-headroom/analyze_pilot_e2e.py` +- External immutable result root: + `/home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714` + +## Data sanity block + +| Data | n | Min | Max | Distinct | Invariant | +|---|---:|---:|---:|---:|---| +| P1 labels | 12 | 0 | 1 | 2 | 7 feasible / 5 infeasible | +| Primary elapsed seconds | 12 | 19.448 | 61.435 | 12 | Every five-second prefix is in range | +| Prefix Layer-1 records | 12 | 332 | 557 | 12 | Contiguous; zero drops | +| Exact timestamped outcomes | 12 anchors | 54 | 750 | 11 | Monotonic completion timestamps | +| Simulator pass rate | 12 | 0.1548 | 1.0 | 7 | Ratios in `[0,1]` | +| Strong nested probabilities | 24 | 0.000208 | 0.809422 | 24 | Ratios in `[0,1]` | +| E2E cost components | 36 | 0.001389 | 0.169653 H20-h | 21 | Non-negative | +| GPU attempts | 2 | 0.020552 | 1.722112 H20-h | 2 | Aggregate 1.742664 < 3.5 | +| Copied raw files | 191 | — | 153,093,348 bytes total | — | Remote/local aggregate SHA identical | + +Checked invariants: six cells and twelve anchors; exact request count and +request-ID/arrival/length hashes; all cell validation flags true; both labels +present; probabilities bounded; costs and counters non-negative; simulator +results not all identical; committed simulator rerun 12/12 numerically +identical to the exploratory run; no prompt text in public simulator fixtures; +no co-resident serving process; final eight GPUs at 0 MiB and 0% utilization. +No red flag remains. diff --git a/docs/fidelity-aware-harness-protocol-20260714.md b/docs/fidelity-aware-harness-protocol-20260714.md index 8bf05f0..b6176f5 100644 --- a/docs/fidelity-aware-harness-protocol-20260714.md +++ b/docs/fidelity-aware-harness-protocol-20260714.md @@ -1,9 +1,18 @@ # Fidelity-aware real-verification harness protocol -Status: **PRE-REGISTERED STAGED EVALUATION; CONTRIBUTION NOT YET ESTABLISHED**. +Status: **P1 FAILED; P2/P3 CLOSED FOR THE REGISTERED METHOD; CONTRIBUTION NOT ESTABLISHED**. Date frozen: 2026-07-14 (Asia/Singapore). +Post-run disposition (2026-07-14): P1 completed with valid data but failed its +registered incremental gate. The strengthened simulator-aware comparison and +end-to-end `k=2` replay also failed: instrumentation was safe and retained zero +regret, but reduced online H20-hours by only 1.426% versus sim top-k + real +final, against the 30% bar. Outcome-only was unsafe. P2/P3 are therefore not +opened for this model. Full results and the permitted reopening condition are +in `docs/fidelity-aware-harness-p1-report-20260714.md`; the protocol below is +retained unchanged as the pre-run record. + ## Research question and contribution bar The harness has an independent systems contribution only if engine-internal