aituner/docs/fidelity-aware-harness-p1-report-20260714.md

Fidelity-aware harness P1 result

Status: REGISTERED ROUTE REJECTED; DO NOT OPEN P2/P3 FOR THE CURRENT METHOD.

Date: 2026-07-14 (Asia/Singapore).

Outcome

The registered five-second instrumentation-aware verifier did not pass P1. The stronger simulator-aware comparison also failed the independent contribution bar. On the frozen k=2 end-to-end replay:

• sim top-k + real final selected the real oracle with zero regret;
• instrumentation-aware also selected the oracle, but reduced online H20-hours by only 1.426% (1.329% when the prior failed attempt is added to both);
• the required reduction was 30% versus full real final and 20% versus a safe outcome-only calibrator;
• the outcome-only calibrator was not safe: it rejected the true best cell, so its apparent cost saving is not a deployable comparison.

This rejects the claim that the current joint logistic verifier, trained on one historical workload, gives the harness an independent tuning contribution. It does not prove that engine telemetry contains no useful signal. Telemetry improved held-out classification and removed unsafe decisions, but did not turn that signal into meaningful end-to-end tuning-cost reduction.

Frozen setup

• Host: dash0, 8 NVIDIA H20 GPUs; cells were serialized and used TP1, TP2, or TP4 without co-resident serving jobs.
• Engine/model: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16.
• Workload: held-out chat_w20260312_1000, seven disjoint repeat bands, 60-second replay after 0.1 time scaling, input [0,8192], exactly 128 output tokens.
• SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, request pass rate at least 0.95.
• Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
• Per cell: burn-in, three low-rate repeats, and three high-rate repeats. The first repeat supplied the five-second prefix; 2-of-3 supplied its label.
• Models: the registered pair used config/workload/outcome versus the same vector plus Layer-1 engine telemetry. The strengthened pair additionally gave both models identical frozen Frontier throughput, SLO pass-rate, and feasibility predictions.
• Policy: accept at p>=0.95, reject at p<=0.05, otherwise continue the same trial. Model, cutoff, threshold, role order, request hashes, and cap were frozen before their applicable evaluation.

The first launch failed its warm-up input-count validation before a measured anchor. It cost 0.020552 H20-hours. The corrected primary attempt cost 1.722112 H20-hours, so aggregate campaign cost was 1.742664 H20-hours, below the 3.5 cap. The fix changed only warm-up validation; formal request counts and hash checks were unchanged.

P1 labels are not an artificial easy split

The 12 adjudicated anchor labels contain 7 feasible and 5 infeasible examples. They are not simply “low feasible, high infeasible”:

• TP2/MNS64 high was feasible in all three repeats;
• TP4/MNS64 low and high were feasible in all six repeats;
• TP4/MNS16 low and high were infeasible in all six repeats.

That last pair creates a large real MNS interaction under an otherwise matched TP4 configuration. Frontier correctly predicted TP4/MNS64 high as feasible, but incorrectly predicted TP4/MNS16 low as feasible. It also incorrectly predicted TP1/MNS64 high as feasible. Overall simulator-only feasibility was 10/12 correct: 83.33% accuracy, with two false-feasible predictions and no false-infeasible prediction.

The two false-feasible cases expose the intended latent-state problem. At five seconds, all 26 completed TP4/MNS16-low requests and all 9 completed TP1/MNS64-high requests still passed their SLO, although both full anchors were infeasible. External outcomes had not yet exposed the future failure; queue, running-batch, and scheduler state existed before the tail outcome. This is mechanistic evidence that instrumentation can be useful, not evidence that the current learned policy uses it well enough.

Registered and strengthened prefix results

At the frozen 0.95 policy threshold:

Comparison	Accuracy	Balanced acc.	Early decisions	False accept	False reject	Valid primary-trial saving
Registered outcome-only	41.67%	50.00%	6/12	0	2	invalid
Registered + telemetry	66.67%	71.43%	4/12	0	0	11.44%
Strong sim + outcome	66.67%	68.57%	5/12	0	1	invalid
Strong sim + outcome + telemetry	83.33%	85.71%	4/12	0	0	11.44%

For the strong pair, telemetry was correct on two examples where the baseline was wrong and lost none; McNemar's exact two-sided p-value is 0.5 at n=12. This is a safety/classification improvement, not a cost contribution. The registered instrumentation policy made two fewer early decisions than its baseline, so it failed the registered +3 decisions or +15 percentage points incremental gate.

The result is not robust to the frozen regularization sensitivity:

L2 lambda	Sim+outcome acc.	+telemetry acc.	Base policy errors	Telemetry policy errors	Base saving	Telemetry saving
0.1	41.67%	75.00%	4	2	invalid	invalid
1.0	66.67%	83.33%	1	0	invalid	11.44%
10.0	83.33%	83.33%	0	0	0.00%	5.98%

Consequently the positive classification delta is neither statistically nor hyperparameter robust.

End-to-end shortlist result

Frontier's simulator-feasible ranking on the tested P1 surface was:

Rank	Cell / anchor	Sim throughput/GPU	Real feasible	Real offered goodput/GPU
1	TP4/MNS64 high	3.0718	yes	3.1250
2	TP1/MNS64 high	2.8823	no	2.9833
3	TP2/MNS64 high	2.8096	yes	2.8750
4	TP4/MNS16 low	2.0866	no	2.1250
5	TP1/MNS8 low	1.9806	yes	2.0333
6	TP2/MNS8 low	1.8637	yes	1.9083

The real oracle was TP4/MNS64 high at 3.125 req/s/GPU. Cost includes an inferred per-cell startup/warm-up/burn-in/cleanup component and the selected anchor; benchmark-only 2-of-3 annotation intervals are removed. Gaps around annotation runs remain in the shared setup term, making this a conservative method-cost estimate.

Frozen policy (k=2)	Online H20-hours	+ prior failure	Real regret	Safety	Saving vs full
Sim top-2 + real final	0.281383	0.301935	0.00%	valid	—
Sim + outcome prefix	0.214664	0.235216	no selected cell	1 false reject	invalid
Sim + outcome + telemetry	0.277370	0.297922	0.00%	valid	1.426%

For k=1, the simulator already picked the oracle and telemetry abstained, so its saving was exactly 0%. For k=3, telemetry remained correct but saved 1.007%. The result therefore does not depend on an unfavorable choice of k.

Across all six cells, a generous replay that removes benchmark-only repeat intervals estimates 0.944244 H20-hours for the full online workflow. The instrumentation policy saved 0.043467 H20-hours: 11.44% of primary trial time, but only 4.603% of end-to-end online cost. Even a post-hoc oracle symmetric threshold cannot make the current telemetry model reach the contribution bar; its best zero-error envelope saves at most 5.69% of inferred online cost. A strong outcome-only model at another post-hoc regularization/threshold can save 16.13%, which further prevents attributing a unique advantage to telemetry. These oracle-threshold numbers are diagnostics only and are not test evidence.

Why the learned verifier did not generalize

The training corpus has only 37 anchors from one workload/SLO task. P1 shows large covariate shift:

• sim+outcome: 12/192 feature values exceed 3 training standard deviations and 4 exceed 5; maximum absolute z-score is 10.36;
• sim+outcome+telemetry: 19/396 exceed 3 and 9 exceed 5;
• the largest shifts include admitted input-length mean (10.36), waiting state (7.77), running maximum (6.38), and decode-batch maximum (6.08).

Coefficient attribution shows that the input-length feature dominates several wrong feasible-anchor logits. Because all training examples share one task, the joint classifier can learn incidental within-task correlation and override a correct simulator prior on TP2/MNS64-high and TP4/MNS64-high. This is a supported diagnosis of model/data insufficiency; it is not a causal proof that one feature alone caused the P1 failure.

More importantly, retuning lambda, threshold, features, or cutoff on P1 and then calling P1 a held-out result would violate calibration/evaluation separation. P1 may now be used only as development data.

Decision and the only defensible reopening condition

Do not run registered P2/P3 with the current model. It failed the predeclared gate on the favorable primary-trial denominator and is even farther from the bar under end-to-end cost. Spending six-task headline GPU budget on the same method would be metric shopping, not replication.

A new route may be opened only as a new hypothesis:

1. Replace the joint classifier with a simulator-residual verifier. The simulator prediction remains an explicit prior; nested outcome-only and telemetry models learn when that prior is wrong, rather than freely relearning feasibility and overriding it under workload shift.
2. Train on multiple complete workload/SLO tasks. SLO thresholds and target pass rate must be explicit inputs; splits are by complete task.
3. Calibrate abstention with task-level risk control. No threshold is selected on a headline task, and “never early decide” is included as the safe outcome-only baseline.
4. Treat Phase 6 and P1 as development only, freeze the residual architecture, features, cutoff, threshold, simulator reading, and k, then use entirely new trace windows for a new gate.

This reopening is justified only if development data show both (a) the simulator's errors are predictable from pre-outcome engine state and (b) a simulator-preserving residual model does not corrupt correct simulator predictions. It is a new project decision, not a continuation automatically authorized by P1.

Benchmark audit

Audit item	Verdict	Severity	Evidence / disposition
Calibration set separate from P1	PASS	—	Phase 6/0311 trained; P1/0312 tested
Strong simulator-aware baseline	PASS	—	Identical Frontier features in both nested models
Sim top-k + real-final E2E baseline	PASS	—	Frozen k=2, tie expansion, measured setup/continuation cost
Multiple independent headline tasks	NEEDS EVIDENCE	Blocking for a positive claim	P1 gate failed; P2 correctly not opened
Statistical significance	NEEDS EVIDENCE	Blocking for a positive claim	n=12 anchors from one task; McNemar p=0.5
Hyperparameter robustness	FAIL	Blocking	Lambda sensitivity changes safety and relative result
Full resource accounting	PASS for P1	—	Failures, startup/warm-up/burn-in, continuation and annotation separated
Avoid post-test retuning	PASS only if route stops	Blocking if violated	P1 is now development-only
Selective winning-workload reporting	PASS	—	Negative P1 and TP/MNS losing cases retained

Overall recommendation: Block the current independent harness contribution claim.

Artifacts

• Registered protocol: docs/fidelity-aware-harness-protocol-20260714.md
• Historical headroom: docs/fidelity-aware-harness-headroom-20260714.md
• Registered P1 analysis: runs/fidelity-headroom/analyze_pilot.py
• Strong P1 analysis: runs/fidelity-headroom/analyze_strong_pilot.py
• E2E shortlist replay: runs/fidelity-headroom/analyze_pilot_e2e.py
• External immutable result root: /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714

Data sanity block

Data	n	Min	Max	Distinct	Invariant
P1 labels	12	0	1	2	7 feasible / 5 infeasible
Primary elapsed seconds	12	19.448	61.435	12	Every five-second prefix is in range
Prefix Layer-1 records	12	332	557	12	Contiguous; zero drops
Exact timestamped outcomes	12 anchors	54	750	11	Monotonic completion timestamps
Simulator pass rate	12	0.1548	1.0	7	Ratios in [0,1]
Strong nested probabilities	24	0.000208	0.809422	24	Ratios in [0,1]
E2E cost components	36	0.001389	0.169653 H20-h	21	Non-negative
GPU attempts	2	0.020552	1.722112 H20-h	2	Aggregate 1.742664 < 3.5
Copied raw files	191	—	153,093,348 bytes total	—	Remote/local aggregate SHA identical

Checked invariants: six cells and twelve anchors; exact request count and request-ID/arrival/length hashes; all cell validation flags true; both labels present; probabilities bounded; costs and counters non-negative; simulator results not all identical; committed simulator rerun 12/12 numerically identical to the exploratory run; no prompt text in public simulator fixtures; no co-resident serving process; final eight GPUs at 0 MiB and 0% utilization. No red flag remains.