# Fidelity-aware harness P1 result

Status: **REGISTERED ROUTE REJECTED; DO NOT OPEN P2/P3 FOR THE CURRENT METHOD**.

Date: 2026-07-14 (Asia/Singapore).

## Outcome

The registered five-second instrumentation-aware verifier did not pass P1.
The stronger simulator-aware comparison also failed the independent
contribution bar.  On the frozen `k=2` end-to-end replay:

- `sim top-k + real final` selected the real oracle with zero regret;
- instrumentation-aware also selected the oracle, but reduced online H20-hours
  by only **1.426%** (1.329% when the prior failed attempt is added to both);
- the required reduction was 30% versus full real final and 20% versus a safe
  outcome-only calibrator;
- the outcome-only calibrator was not safe: it rejected the true best cell, so
  its apparent cost saving is not a deployable comparison.

This rejects the claim that the **current joint logistic verifier**, trained on
one historical workload, gives the harness an independent tuning contribution.
It does not prove that engine telemetry contains no useful signal.  Telemetry
improved held-out classification and removed unsafe decisions, but did not turn
that signal into meaningful end-to-end tuning-cost reduction.

## Frozen setup

- Host: `dash0`, 8 NVIDIA H20 GPUs; cells were serialized and used TP1, TP2,
  or TP4 without co-resident serving jobs.
- Engine/model: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16.
- Workload: held-out `chat_w20260312_1000`, seven disjoint repeat bands,
  60-second replay after 0.1 time scaling, input `[0,8192]`, exactly 128 output
  tokens.
- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, request pass rate at least 0.95.
- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
- Per cell: burn-in, three low-rate repeats, and three high-rate repeats.  The
  first repeat supplied the five-second prefix; 2-of-3 supplied its label.
- Models: the registered pair used config/workload/outcome versus the same
  vector plus Layer-1 engine telemetry.  The strengthened pair additionally
  gave both models identical frozen Frontier throughput, SLO pass-rate, and
  feasibility predictions.
- Policy: accept at `p>=0.95`, reject at `p<=0.05`, otherwise continue the same
  trial.  Model, cutoff, threshold, role order, request hashes, and cap were
  frozen before their applicable evaluation.

The first launch failed its warm-up input-count validation before a measured
anchor.  It cost 0.020552 H20-hours.  The corrected primary attempt cost
1.722112 H20-hours, so aggregate campaign cost was **1.742664 H20-hours**, below
the 3.5 cap.  The fix changed only warm-up validation; formal request counts and
hash checks were unchanged.

## P1 labels are not an artificial easy split

The 12 adjudicated anchor labels contain 7 feasible and 5 infeasible examples.
They are not simply “low feasible, high infeasible”:

- TP2/MNS64 high was feasible in all three repeats;
- TP4/MNS64 low and high were feasible in all six repeats;
- TP4/MNS16 low and high were infeasible in all six repeats.

That last pair creates a large real MNS interaction under an otherwise matched
TP4 configuration.  Frontier correctly predicted TP4/MNS64 high as feasible,
but incorrectly predicted TP4/MNS16 low as feasible.  It also incorrectly
predicted TP1/MNS64 high as feasible.  Overall simulator-only feasibility was
10/12 correct: 83.33% accuracy, with two false-feasible predictions and no
false-infeasible prediction.

The two false-feasible cases expose the intended latent-state problem.  At five
seconds, all 26 completed TP4/MNS16-low requests and all 9 completed
TP1/MNS64-high requests still passed their SLO, although both full anchors were
infeasible.  External outcomes had not yet exposed the future failure; queue,
running-batch, and scheduler state existed before the tail outcome.  This is
mechanistic evidence that instrumentation can be useful, not evidence that the
current learned policy uses it well enough.

## Registered and strengthened prefix results

At the frozen 0.95 policy threshold:

| Comparison | Accuracy | Balanced acc. | Early decisions | False accept | False reject | Valid primary-trial saving |
|---|---:|---:|---:|---:|---:|---:|
| Registered outcome-only | 41.67% | 50.00% | 6/12 | 0 | 2 | invalid |
| Registered + telemetry | 66.67% | 71.43% | 4/12 | 0 | 0 | 11.44% |
| Strong sim + outcome | 66.67% | 68.57% | 5/12 | 0 | 1 | invalid |
| Strong sim + outcome + telemetry | 83.33% | 85.71% | 4/12 | 0 | 0 | 11.44% |

For the strong pair, telemetry was correct on two examples where the baseline
was wrong and lost none; McNemar's exact two-sided p-value is 0.5 at `n=12`.
This is a safety/classification improvement, not a cost contribution.  The
registered instrumentation policy made two fewer early decisions than its
baseline, so it failed the registered `+3 decisions or +15 percentage points`
incremental gate.

The result is not robust to the frozen regularization sensitivity:

| L2 lambda | Sim+outcome acc. | +telemetry acc. | Base policy errors | Telemetry policy errors | Base saving | Telemetry saving |
|---:|---:|---:|---:|---:|---:|---:|
| 0.1 | 41.67% | 75.00% | 4 | 2 | invalid | invalid |
| 1.0 | 66.67% | 83.33% | 1 | 0 | invalid | 11.44% |
| 10.0 | 83.33% | 83.33% | 0 | 0 | 0.00% | 5.98% |

Consequently the positive classification delta is neither statistically nor
hyperparameter robust.

## End-to-end shortlist result

Frontier's simulator-feasible ranking on the tested P1 surface was:

| Rank | Cell / anchor | Sim throughput/GPU | Real feasible | Real offered goodput/GPU |
|---:|---|---:|---:|---:|
| 1 | TP4/MNS64 high | 3.0718 | yes | 3.1250 |
| 2 | TP1/MNS64 high | 2.8823 | no | 2.9833 |
| 3 | TP2/MNS64 high | 2.8096 | yes | 2.8750 |
| 4 | TP4/MNS16 low | 2.0866 | no | 2.1250 |
| 5 | TP1/MNS8 low | 1.9806 | yes | 2.0333 |
| 6 | TP2/MNS8 low | 1.8637 | yes | 1.9083 |

The real oracle was TP4/MNS64 high at 3.125 req/s/GPU.  Cost includes an
inferred per-cell startup/warm-up/burn-in/cleanup component and the selected
anchor; benchmark-only 2-of-3 annotation intervals are removed.  Gaps around
annotation runs remain in the shared setup term, making this a conservative
method-cost estimate.

| Frozen policy (`k=2`) | Online H20-hours | + prior failure | Real regret | Safety | Saving vs full |
|---|---:|---:|---:|---|---:|
| Sim top-2 + real final | 0.281383 | 0.301935 | 0.00% | valid | — |
| Sim + outcome prefix | 0.214664 | 0.235216 | no selected cell | 1 false reject | invalid |
| Sim + outcome + telemetry | 0.277370 | 0.297922 | 0.00% | valid | **1.426%** |

For `k=1`, the simulator already picked the oracle and telemetry abstained, so
its saving was exactly 0%.  For `k=3`, telemetry remained correct but saved
1.007%.  The result therefore does not depend on an unfavorable choice of `k`.

Across all six cells, a generous replay that removes benchmark-only repeat
intervals estimates 0.944244 H20-hours for the full online workflow.  The
instrumentation policy saved 0.043467 H20-hours: 11.44% of primary trial time,
but only **4.603%** of end-to-end online cost.  Even a post-hoc oracle symmetric
threshold cannot make the current telemetry model reach the contribution bar;
its best zero-error envelope saves at most 5.69% of inferred online cost.  A
strong outcome-only model at another post-hoc regularization/threshold can save
16.13%, which further prevents attributing a unique advantage to telemetry.
These oracle-threshold numbers are diagnostics only and are not test evidence.

## Why the learned verifier did not generalize

The training corpus has only 37 anchors from one workload/SLO task.  P1 shows
large covariate shift:

- sim+outcome: 12/192 feature values exceed 3 training standard deviations and
  4 exceed 5; maximum absolute z-score is 10.36;
- sim+outcome+telemetry: 19/396 exceed 3 and 9 exceed 5;
- the largest shifts include admitted input-length mean (10.36), waiting state
  (7.77), running maximum (6.38), and decode-batch maximum (6.08).

Coefficient attribution shows that the input-length feature dominates several
wrong feasible-anchor logits.  Because all training examples share one task,
the joint classifier can learn incidental within-task correlation and override
a correct simulator prior on TP2/MNS64-high and TP4/MNS64-high.  This is a
supported diagnosis of model/data insufficiency; it is not a causal proof that
one feature alone caused the P1 failure.

More importantly, retuning lambda, threshold, features, or cutoff on P1 and
then calling P1 a held-out result would violate calibration/evaluation
separation.  P1 may now be used only as development data.

## Decision and the only defensible reopening condition

Do not run registered P2/P3 with the current model.  It failed the predeclared
gate on the favorable primary-trial denominator and is even farther from the
bar under end-to-end cost.  Spending six-task headline GPU budget on the same
method would be metric shopping, not replication.

A new route may be opened only as a new hypothesis:

1. Replace the joint classifier with a **simulator-residual verifier**.  The
   simulator prediction remains an explicit prior; nested outcome-only and
   telemetry models learn when that prior is wrong, rather than freely
   relearning feasibility and overriding it under workload shift.
2. Train on multiple complete workload/SLO tasks.  SLO thresholds and target
   pass rate must be explicit inputs; splits are by complete task.
3. Calibrate abstention with task-level risk control.  No threshold is selected
   on a headline task, and “never early decide” is included as the safe
   outcome-only baseline.
4. Treat Phase 6 and P1 as development only, freeze the residual architecture,
   features, cutoff, threshold, simulator reading, and `k`, then use entirely
   new trace windows for a new gate.

This reopening is justified only if development data show both (a) the
simulator's errors are predictable from pre-outcome engine state and (b) a
simulator-preserving residual model does not corrupt correct simulator
predictions.  It is a new project decision, not a continuation automatically
authorized by P1.

## Benchmark audit

| Audit item | Verdict | Severity | Evidence / disposition |
|---|---|---|---|
| Calibration set separate from P1 | PASS | — | Phase 6/0311 trained; P1/0312 tested |
| Strong simulator-aware baseline | PASS | — | Identical Frontier features in both nested models |
| Sim top-k + real-final E2E baseline | PASS | — | Frozen `k=2`, tie expansion, measured setup/continuation cost |
| Multiple independent headline tasks | NEEDS EVIDENCE | Blocking for a positive claim | P1 gate failed; P2 correctly not opened |
| Statistical significance | NEEDS EVIDENCE | Blocking for a positive claim | n=12 anchors from one task; McNemar p=0.5 |
| Hyperparameter robustness | FAIL | Blocking | Lambda sensitivity changes safety and relative result |
| Full resource accounting | PASS for P1 | — | Failures, startup/warm-up/burn-in, continuation and annotation separated |
| Avoid post-test retuning | PASS only if route stops | Blocking if violated | P1 is now development-only |
| Selective winning-workload reporting | PASS | — | Negative P1 and TP/MNS losing cases retained |

Overall recommendation: **Block the current independent harness contribution
claim.**

## Artifacts

- Registered protocol: `docs/fidelity-aware-harness-protocol-20260714.md`
- Historical headroom: `docs/fidelity-aware-harness-headroom-20260714.md`
- Registered P1 analysis: `runs/fidelity-headroom/analyze_pilot.py`
- Strong P1 analysis: `runs/fidelity-headroom/analyze_strong_pilot.py`
- E2E shortlist replay: `runs/fidelity-headroom/analyze_pilot_e2e.py`
- External immutable result root:
  `/home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714`

## Data sanity block

| Data | n | Min | Max | Distinct | Invariant |
|---|---:|---:|---:|---:|---|
| P1 labels | 12 | 0 | 1 | 2 | 7 feasible / 5 infeasible |
| Primary elapsed seconds | 12 | 19.448 | 61.435 | 12 | Every five-second prefix is in range |
| Prefix Layer-1 records | 12 | 332 | 557 | 12 | Contiguous; zero drops |
| Exact timestamped outcomes | 12 anchors | 54 | 750 | 11 | Monotonic completion timestamps |
| Simulator pass rate | 12 | 0.1548 | 1.0 | 7 | Ratios in `[0,1]` |
| Strong nested probabilities | 24 | 0.000208 | 0.809422 | 24 | Ratios in `[0,1]` |
| E2E cost components | 36 | 0.001389 | 0.169653 H20-h | 21 | Non-negative |
| GPU attempts | 2 | 0.020552 | 1.722112 H20-h | 2 | Aggregate 1.742664 < 3.5 |
| Copied raw files | 191 | — | 153,093,348 bytes total | — | Remote/local aggregate SHA identical |

Checked invariants: six cells and twelve anchors; exact request count and
request-ID/arrival/length hashes; all cell validation flags true; both labels
present; probabilities bounded; costs and counters non-negative; simulator
results not all identical; committed simulator rerun 12/12 numerically
identical to the exploratory run; no prompt text in public simulator fixtures;
no co-resident serving process; final eight GPUs at 0 MiB and 0% utilization.
No red flag remains.
