Report failed fidelity pilot gate

docs/fidelity-aware-harness-headroom-20260714.md

@@ -1,10 +1,16 @@
 # Fidelity-aware harness headroom audit
 
-Status: **PROMISING PREMISE, NO CONTRIBUTION CLAIM**.
+Status: **HISTORICAL PREMISE DID NOT PASS PROSPECTIVE P1; NO CONTRIBUTION CLAIM**.
 
 The audit answers whether engine instrumentation has enough incremental signal
 to justify a prospective experiment.  It does not establish generalization.
 
+Post-run update: the exact-timestamp held-out P1 completed and failed the
+registered gate.  Under the stronger simulator-aware `k=2` end-to-end replay,
+telemetry preserved zero regret but saved only 1.426% online H20-hours versus
+sim top-k + real final.  The current route is closed; see
+`docs/fidelity-aware-harness-p1-report-20260714.md`.
+
 ## Simulator shortlist lower bound
 
 On the frozen 12-cell SimFid task, the strongest calibrated SLO simulator

docs/fidelity-aware-harness-p1-report-20260714.md

@@ -0,0 +1,242 @@
+# Fidelity-aware harness P1 result
+
+Status: **REGISTERED ROUTE REJECTED; DO NOT OPEN P2/P3 FOR THE CURRENT METHOD**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Outcome
+
+The registered five-second instrumentation-aware verifier did not pass P1.
+The stronger simulator-aware comparison also failed the independent
+contribution bar.  On the frozen `k=2` end-to-end replay:
+
+- `sim top-k + real final` selected the real oracle with zero regret;
+- instrumentation-aware also selected the oracle, but reduced online H20-hours
+  by only **1.426%** (1.329% when the prior failed attempt is added to both);
+- the required reduction was 30% versus full real final and 20% versus a safe
+  outcome-only calibrator;
+- the outcome-only calibrator was not safe: it rejected the true best cell, so
+  its apparent cost saving is not a deployable comparison.
+
+This rejects the claim that the **current joint logistic verifier**, trained on
+one historical workload, gives the harness an independent tuning contribution.
+It does not prove that engine telemetry contains no useful signal.  Telemetry
+improved held-out classification and removed unsafe decisions, but did not turn
+that signal into meaningful end-to-end tuning-cost reduction.
+
+## Frozen setup
+
+- Host: `dash0`, 8 NVIDIA H20 GPUs; cells were serialized and used TP1, TP2,
+  or TP4 without co-resident serving jobs.
+- Engine/model: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16.
+- Workload: held-out `chat_w20260312_1000`, seven disjoint repeat bands,
+  60-second replay after 0.1 time scaling, input `[0,8192]`, exactly 128 output
+  tokens.
+- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, request pass rate at least 0.95.
+- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
+- Per cell: burn-in, three low-rate repeats, and three high-rate repeats.  The
+  first repeat supplied the five-second prefix; 2-of-3 supplied its label.
+- Models: the registered pair used config/workload/outcome versus the same
+  vector plus Layer-1 engine telemetry.  The strengthened pair additionally
+  gave both models identical frozen Frontier throughput, SLO pass-rate, and
+  feasibility predictions.
+- Policy: accept at `p>=0.95`, reject at `p<=0.05`, otherwise continue the same
+  trial.  Model, cutoff, threshold, role order, request hashes, and cap were
+  frozen before their applicable evaluation.
+
+The first launch failed its warm-up input-count validation before a measured
+anchor.  It cost 0.020552 H20-hours.  The corrected primary attempt cost
+1.722112 H20-hours, so aggregate campaign cost was **1.742664 H20-hours**, below
+the 3.5 cap.  The fix changed only warm-up validation; formal request counts and
+hash checks were unchanged.
+
+## P1 labels are not an artificial easy split
+
+The 12 adjudicated anchor labels contain 7 feasible and 5 infeasible examples.
+They are not simply “low feasible, high infeasible”:
+
+- TP2/MNS64 high was feasible in all three repeats;
+- TP4/MNS64 low and high were feasible in all six repeats;
+- TP4/MNS16 low and high were infeasible in all six repeats.
+
+That last pair creates a large real MNS interaction under an otherwise matched
+TP4 configuration.  Frontier correctly predicted TP4/MNS64 high as feasible,
+but incorrectly predicted TP4/MNS16 low as feasible.  It also incorrectly
+predicted TP1/MNS64 high as feasible.  Overall simulator-only feasibility was
+10/12 correct: 83.33% accuracy, with two false-feasible predictions and no
+false-infeasible prediction.
+
+The two false-feasible cases expose the intended latent-state problem.  At five
+seconds, all 26 completed TP4/MNS16-low requests and all 9 completed
+TP1/MNS64-high requests still passed their SLO, although both full anchors were
+infeasible.  External outcomes had not yet exposed the future failure; queue,
+running-batch, and scheduler state existed before the tail outcome.  This is
+mechanistic evidence that instrumentation can be useful, not evidence that the
+current learned policy uses it well enough.
+
+## Registered and strengthened prefix results
+
+At the frozen 0.95 policy threshold:
+
+| Comparison | Accuracy | Balanced acc. | Early decisions | False accept | False reject | Valid primary-trial saving |
+|---|---:|---:|---:|---:|---:|---:|
+| Registered outcome-only | 41.67% | 50.00% | 6/12 | 0 | 2 | invalid |
+| Registered + telemetry | 66.67% | 71.43% | 4/12 | 0 | 0 | 11.44% |
+| Strong sim + outcome | 66.67% | 68.57% | 5/12 | 0 | 1 | invalid |
+| Strong sim + outcome + telemetry | 83.33% | 85.71% | 4/12 | 0 | 0 | 11.44% |
+
+For the strong pair, telemetry was correct on two examples where the baseline
+was wrong and lost none; McNemar's exact two-sided p-value is 0.5 at `n=12`.
+This is a safety/classification improvement, not a cost contribution.  The
+registered instrumentation policy made two fewer early decisions than its
+baseline, so it failed the registered `+3 decisions or +15 percentage points`
+incremental gate.
+
+The result is not robust to the frozen regularization sensitivity:
+
+| L2 lambda | Sim+outcome acc. | +telemetry acc. | Base policy errors | Telemetry policy errors | Base saving | Telemetry saving |
+|---:|---:|---:|---:|---:|---:|---:|
+| 0.1 | 41.67% | 75.00% | 4 | 2 | invalid | invalid |
+| 1.0 | 66.67% | 83.33% | 1 | 0 | invalid | 11.44% |
+| 10.0 | 83.33% | 83.33% | 0 | 0 | 0.00% | 5.98% |
+
+Consequently the positive classification delta is neither statistically nor
+hyperparameter robust.
+
+## End-to-end shortlist result
+
+Frontier's simulator-feasible ranking on the tested P1 surface was:
+
+| Rank | Cell / anchor | Sim throughput/GPU | Real feasible | Real offered goodput/GPU |
+|---:|---|---:|---:|---:|
+| 1 | TP4/MNS64 high | 3.0718 | yes | 3.1250 |
+| 2 | TP1/MNS64 high | 2.8823 | no | 2.9833 |
+| 3 | TP2/MNS64 high | 2.8096 | yes | 2.8750 |
+| 4 | TP4/MNS16 low | 2.0866 | no | 2.1250 |
+| 5 | TP1/MNS8 low | 1.9806 | yes | 2.0333 |
+| 6 | TP2/MNS8 low | 1.8637 | yes | 1.9083 |
+
+The real oracle was TP4/MNS64 high at 3.125 req/s/GPU.  Cost includes an
+inferred per-cell startup/warm-up/burn-in/cleanup component and the selected
+anchor; benchmark-only 2-of-3 annotation intervals are removed.  Gaps around
+annotation runs remain in the shared setup term, making this a conservative
+method-cost estimate.
+
+| Frozen policy (`k=2`) | Online H20-hours | + prior failure | Real regret | Safety | Saving vs full |
+|---|---:|---:|---:|---|---:|
+| Sim top-2 + real final | 0.281383 | 0.301935 | 0.00% | valid | — |
+| Sim + outcome prefix | 0.214664 | 0.235216 | no selected cell | 1 false reject | invalid |
+| Sim + outcome + telemetry | 0.277370 | 0.297922 | 0.00% | valid | **1.426%** |
+
+For `k=1`, the simulator already picked the oracle and telemetry abstained, so
+its saving was exactly 0%.  For `k=3`, telemetry remained correct but saved
+1.007%.  The result therefore does not depend on an unfavorable choice of `k`.
+
+Across all six cells, a generous replay that removes benchmark-only repeat
+intervals estimates 0.944244 H20-hours for the full online workflow.  The
+instrumentation policy saved 0.043467 H20-hours: 11.44% of primary trial time,
+but only **4.603%** of end-to-end online cost.  Even a post-hoc oracle symmetric
+threshold cannot make the current telemetry model reach the contribution bar;
+its best zero-error envelope saves at most 5.69% of inferred online cost.  A
+strong outcome-only model at another post-hoc regularization/threshold can save
+16.13%, which further prevents attributing a unique advantage to telemetry.
+These oracle-threshold numbers are diagnostics only and are not test evidence.
+
+## Why the learned verifier did not generalize
+
+The training corpus has only 37 anchors from one workload/SLO task.  P1 shows
+large covariate shift:
+
+- sim+outcome: 12/192 feature values exceed 3 training standard deviations and
+  4 exceed 5; maximum absolute z-score is 10.36;
+- sim+outcome+telemetry: 19/396 exceed 3 and 9 exceed 5;
+- the largest shifts include admitted input-length mean (10.36), waiting state
+  (7.77), running maximum (6.38), and decode-batch maximum (6.08).
+
+Coefficient attribution shows that the input-length feature dominates several
+wrong feasible-anchor logits.  Because all training examples share one task,
+the joint classifier can learn incidental within-task correlation and override
+a correct simulator prior on TP2/MNS64-high and TP4/MNS64-high.  This is a
+supported diagnosis of model/data insufficiency; it is not a causal proof that
+one feature alone caused the P1 failure.
+
+More importantly, retuning lambda, threshold, features, or cutoff on P1 and
+then calling P1 a held-out result would violate calibration/evaluation
+separation.  P1 may now be used only as development data.
+
+## Decision and the only defensible reopening condition
+
+Do not run registered P2/P3 with the current model.  It failed the predeclared
+gate on the favorable primary-trial denominator and is even farther from the
+bar under end-to-end cost.  Spending six-task headline GPU budget on the same
+method would be metric shopping, not replication.
+
+A new route may be opened only as a new hypothesis:
+
+1. Replace the joint classifier with a **simulator-residual verifier**.  The
+   simulator prediction remains an explicit prior; nested outcome-only and
+   telemetry models learn when that prior is wrong, rather than freely
+   relearning feasibility and overriding it under workload shift.
+2. Train on multiple complete workload/SLO tasks.  SLO thresholds and target
+   pass rate must be explicit inputs; splits are by complete task.
+3. Calibrate abstention with task-level risk control.  No threshold is selected
+   on a headline task, and “never early decide” is included as the safe
+   outcome-only baseline.
+4. Treat Phase 6 and P1 as development only, freeze the residual architecture,
+   features, cutoff, threshold, simulator reading, and `k`, then use entirely
+   new trace windows for a new gate.
+
+This reopening is justified only if development data show both (a) the
+simulator's errors are predictable from pre-outcome engine state and (b) a
+simulator-preserving residual model does not corrupt correct simulator
+predictions.  It is a new project decision, not a continuation automatically
+authorized by P1.
+
+## Benchmark audit
+
+| Audit item | Verdict | Severity | Evidence / disposition |
+|---|---|---|---|
+| Calibration set separate from P1 | PASS | — | Phase 6/0311 trained; P1/0312 tested |
+| Strong simulator-aware baseline | PASS | — | Identical Frontier features in both nested models |
+| Sim top-k + real-final E2E baseline | PASS | — | Frozen `k=2`, tie expansion, measured setup/continuation cost |
+| Multiple independent headline tasks | NEEDS EVIDENCE | Blocking for a positive claim | P1 gate failed; P2 correctly not opened |
+| Statistical significance | NEEDS EVIDENCE | Blocking for a positive claim | n=12 anchors from one task; McNemar p=0.5 |
+| Hyperparameter robustness | FAIL | Blocking | Lambda sensitivity changes safety and relative result |
+| Full resource accounting | PASS for P1 | — | Failures, startup/warm-up/burn-in, continuation and annotation separated |
+| Avoid post-test retuning | PASS only if route stops | Blocking if violated | P1 is now development-only |
+| Selective winning-workload reporting | PASS | — | Negative P1 and TP/MNS losing cases retained |
+
+Overall recommendation: **Block the current independent harness contribution
+claim.**
+
+## Artifacts
+
+- Registered protocol: `docs/fidelity-aware-harness-protocol-20260714.md`
+- Historical headroom: `docs/fidelity-aware-harness-headroom-20260714.md`
+- Registered P1 analysis: `runs/fidelity-headroom/analyze_pilot.py`
+- Strong P1 analysis: `runs/fidelity-headroom/analyze_strong_pilot.py`
+- E2E shortlist replay: `runs/fidelity-headroom/analyze_pilot_e2e.py`
+- External immutable result root:
+  `/home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714`
+
+## Data sanity block
+
+| Data | n | Min | Max | Distinct | Invariant |
+|---|---:|---:|---:|---:|---|
+| P1 labels | 12 | 0 | 1 | 2 | 7 feasible / 5 infeasible |
+| Primary elapsed seconds | 12 | 19.448 | 61.435 | 12 | Every five-second prefix is in range |
+| Prefix Layer-1 records | 12 | 332 | 557 | 12 | Contiguous; zero drops |
+| Exact timestamped outcomes | 12 anchors | 54 | 750 | 11 | Monotonic completion timestamps |
+| Simulator pass rate | 12 | 0.1548 | 1.0 | 7 | Ratios in `[0,1]` |
+| Strong nested probabilities | 24 | 0.000208 | 0.809422 | 24 | Ratios in `[0,1]` |
+| E2E cost components | 36 | 0.001389 | 0.169653 H20-h | 21 | Non-negative |
+| GPU attempts | 2 | 0.020552 | 1.722112 H20-h | 2 | Aggregate 1.742664 < 3.5 |
+| Copied raw files | 191 | — | 153,093,348 bytes total | — | Remote/local aggregate SHA identical |
+
+Checked invariants: six cells and twelve anchors; exact request count and
+request-ID/arrival/length hashes; all cell validation flags true; both labels
+present; probabilities bounded; costs and counters non-negative; simulator
+results not all identical; committed simulator rerun 12/12 numerically
+identical to the exploratory run; no prompt text in public simulator fixtures;
+no co-resident serving process; final eight GPUs at 0 MiB and 0% utilization.
+No red flag remains.

docs/fidelity-aware-harness-protocol-20260714.md

@@ -1,9 +1,18 @@
 # Fidelity-aware real-verification harness protocol
 
-Status: **PRE-REGISTERED STAGED EVALUATION; CONTRIBUTION NOT YET ESTABLISHED**.
+Status: **P1 FAILED; P2/P3 CLOSED FOR THE REGISTERED METHOD; CONTRIBUTION NOT ESTABLISHED**.
 
 Date frozen: 2026-07-14 (Asia/Singapore).
 
+Post-run disposition (2026-07-14): P1 completed with valid data but failed its
+registered incremental gate.  The strengthened simulator-aware comparison and
+end-to-end `k=2` replay also failed: instrumentation was safe and retained zero
+regret, but reduced online H20-hours by only 1.426% versus sim top-k + real
+final, against the 30% bar.  Outcome-only was unsafe.  P2/P3 are therefore not
+opened for this model.  Full results and the permitted reopening condition are
+in `docs/fidelity-aware-harness-p1-report-20260714.md`; the protocol below is
+retained unchanged as the pre-run record.
+
 ## Research question and contribution bar
 
 The harness has an independent systems contribution only if engine-internal