243 lines
13 KiB
Markdown
243 lines
13 KiB
Markdown
# Fidelity-aware harness P1 result
|
|
|
|
Status: **REGISTERED ROUTE REJECTED; DO NOT OPEN P2/P3 FOR THE CURRENT METHOD**.
|
|
|
|
Date: 2026-07-14 (Asia/Singapore).
|
|
|
|
## Outcome
|
|
|
|
The registered five-second instrumentation-aware verifier did not pass P1.
|
|
The stronger simulator-aware comparison also failed the independent
|
|
contribution bar. On the frozen `k=2` end-to-end replay:
|
|
|
|
- `sim top-k + real final` selected the real oracle with zero regret;
|
|
- instrumentation-aware also selected the oracle, but reduced online H20-hours
|
|
by only **1.426%** (1.329% when the prior failed attempt is added to both);
|
|
- the required reduction was 30% versus full real final and 20% versus a safe
|
|
outcome-only calibrator;
|
|
- the outcome-only calibrator was not safe: it rejected the true best cell, so
|
|
its apparent cost saving is not a deployable comparison.
|
|
|
|
This rejects the claim that the **current joint logistic verifier**, trained on
|
|
one historical workload, gives the harness an independent tuning contribution.
|
|
It does not prove that engine telemetry contains no useful signal. Telemetry
|
|
improved held-out classification and removed unsafe decisions, but did not turn
|
|
that signal into meaningful end-to-end tuning-cost reduction.
|
|
|
|
## Frozen setup
|
|
|
|
- Host: `dash0`, 8 NVIDIA H20 GPUs; cells were serialized and used TP1, TP2,
|
|
or TP4 without co-resident serving jobs.
|
|
- Engine/model: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16.
|
|
- Workload: held-out `chat_w20260312_1000`, seven disjoint repeat bands,
|
|
60-second replay after 0.1 time scaling, input `[0,8192]`, exactly 128 output
|
|
tokens.
|
|
- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, request pass rate at least 0.95.
|
|
- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
|
|
- Per cell: burn-in, three low-rate repeats, and three high-rate repeats. The
|
|
first repeat supplied the five-second prefix; 2-of-3 supplied its label.
|
|
- Models: the registered pair used config/workload/outcome versus the same
|
|
vector plus Layer-1 engine telemetry. The strengthened pair additionally
|
|
gave both models identical frozen Frontier throughput, SLO pass-rate, and
|
|
feasibility predictions.
|
|
- Policy: accept at `p>=0.95`, reject at `p<=0.05`, otherwise continue the same
|
|
trial. Model, cutoff, threshold, role order, request hashes, and cap were
|
|
frozen before their applicable evaluation.
|
|
|
|
The first launch failed its warm-up input-count validation before a measured
|
|
anchor. It cost 0.020552 H20-hours. The corrected primary attempt cost
|
|
1.722112 H20-hours, so aggregate campaign cost was **1.742664 H20-hours**, below
|
|
the 3.5 cap. The fix changed only warm-up validation; formal request counts and
|
|
hash checks were unchanged.
|
|
|
|
## P1 labels are not an artificial easy split
|
|
|
|
The 12 adjudicated anchor labels contain 7 feasible and 5 infeasible examples.
|
|
They are not simply “low feasible, high infeasible”:
|
|
|
|
- TP2/MNS64 high was feasible in all three repeats;
|
|
- TP4/MNS64 low and high were feasible in all six repeats;
|
|
- TP4/MNS16 low and high were infeasible in all six repeats.
|
|
|
|
That last pair creates a large real MNS interaction under an otherwise matched
|
|
TP4 configuration. Frontier correctly predicted TP4/MNS64 high as feasible,
|
|
but incorrectly predicted TP4/MNS16 low as feasible. It also incorrectly
|
|
predicted TP1/MNS64 high as feasible. Overall simulator-only feasibility was
|
|
10/12 correct: 83.33% accuracy, with two false-feasible predictions and no
|
|
false-infeasible prediction.
|
|
|
|
The two false-feasible cases expose the intended latent-state problem. At five
|
|
seconds, all 26 completed TP4/MNS16-low requests and all 9 completed
|
|
TP1/MNS64-high requests still passed their SLO, although both full anchors were
|
|
infeasible. External outcomes had not yet exposed the future failure; queue,
|
|
running-batch, and scheduler state existed before the tail outcome. This is
|
|
mechanistic evidence that instrumentation can be useful, not evidence that the
|
|
current learned policy uses it well enough.
|
|
|
|
## Registered and strengthened prefix results
|
|
|
|
At the frozen 0.95 policy threshold:
|
|
|
|
| Comparison | Accuracy | Balanced acc. | Early decisions | False accept | False reject | Valid primary-trial saving |
|
|
|---|---:|---:|---:|---:|---:|---:|
|
|
| Registered outcome-only | 41.67% | 50.00% | 6/12 | 0 | 2 | invalid |
|
|
| Registered + telemetry | 66.67% | 71.43% | 4/12 | 0 | 0 | 11.44% |
|
|
| Strong sim + outcome | 66.67% | 68.57% | 5/12 | 0 | 1 | invalid |
|
|
| Strong sim + outcome + telemetry | 83.33% | 85.71% | 4/12 | 0 | 0 | 11.44% |
|
|
|
|
For the strong pair, telemetry was correct on two examples where the baseline
|
|
was wrong and lost none; McNemar's exact two-sided p-value is 0.5 at `n=12`.
|
|
This is a safety/classification improvement, not a cost contribution. The
|
|
registered instrumentation policy made two fewer early decisions than its
|
|
baseline, so it failed the registered `+3 decisions or +15 percentage points`
|
|
incremental gate.
|
|
|
|
The result is not robust to the frozen regularization sensitivity:
|
|
|
|
| L2 lambda | Sim+outcome acc. | +telemetry acc. | Base policy errors | Telemetry policy errors | Base saving | Telemetry saving |
|
|
|---:|---:|---:|---:|---:|---:|---:|
|
|
| 0.1 | 41.67% | 75.00% | 4 | 2 | invalid | invalid |
|
|
| 1.0 | 66.67% | 83.33% | 1 | 0 | invalid | 11.44% |
|
|
| 10.0 | 83.33% | 83.33% | 0 | 0 | 0.00% | 5.98% |
|
|
|
|
Consequently the positive classification delta is neither statistically nor
|
|
hyperparameter robust.
|
|
|
|
## End-to-end shortlist result
|
|
|
|
Frontier's simulator-feasible ranking on the tested P1 surface was:
|
|
|
|
| Rank | Cell / anchor | Sim throughput/GPU | Real feasible | Real offered goodput/GPU |
|
|
|---:|---|---:|---:|---:|
|
|
| 1 | TP4/MNS64 high | 3.0718 | yes | 3.1250 |
|
|
| 2 | TP1/MNS64 high | 2.8823 | no | 2.9833 |
|
|
| 3 | TP2/MNS64 high | 2.8096 | yes | 2.8750 |
|
|
| 4 | TP4/MNS16 low | 2.0866 | no | 2.1250 |
|
|
| 5 | TP1/MNS8 low | 1.9806 | yes | 2.0333 |
|
|
| 6 | TP2/MNS8 low | 1.8637 | yes | 1.9083 |
|
|
|
|
The real oracle was TP4/MNS64 high at 3.125 req/s/GPU. Cost includes an
|
|
inferred per-cell startup/warm-up/burn-in/cleanup component and the selected
|
|
anchor; benchmark-only 2-of-3 annotation intervals are removed. Gaps around
|
|
annotation runs remain in the shared setup term, making this a conservative
|
|
method-cost estimate.
|
|
|
|
| Frozen policy (`k=2`) | Online H20-hours | + prior failure | Real regret | Safety | Saving vs full |
|
|
|---|---:|---:|---:|---|---:|
|
|
| Sim top-2 + real final | 0.281383 | 0.301935 | 0.00% | valid | — |
|
|
| Sim + outcome prefix | 0.214664 | 0.235216 | no selected cell | 1 false reject | invalid |
|
|
| Sim + outcome + telemetry | 0.277370 | 0.297922 | 0.00% | valid | **1.426%** |
|
|
|
|
For `k=1`, the simulator already picked the oracle and telemetry abstained, so
|
|
its saving was exactly 0%. For `k=3`, telemetry remained correct but saved
|
|
1.007%. The result therefore does not depend on an unfavorable choice of `k`.
|
|
|
|
Across all six cells, a generous replay that removes benchmark-only repeat
|
|
intervals estimates 0.944244 H20-hours for the full online workflow. The
|
|
instrumentation policy saved 0.043467 H20-hours: 11.44% of primary trial time,
|
|
but only **4.603%** of end-to-end online cost. Even a post-hoc oracle symmetric
|
|
threshold cannot make the current telemetry model reach the contribution bar;
|
|
its best zero-error envelope saves at most 5.69% of inferred online cost. A
|
|
strong outcome-only model at another post-hoc regularization/threshold can save
|
|
16.13%, which further prevents attributing a unique advantage to telemetry.
|
|
These oracle-threshold numbers are diagnostics only and are not test evidence.
|
|
|
|
## Why the learned verifier did not generalize
|
|
|
|
The training corpus has only 37 anchors from one workload/SLO task. P1 shows
|
|
large covariate shift:
|
|
|
|
- sim+outcome: 12/192 feature values exceed 3 training standard deviations and
|
|
4 exceed 5; maximum absolute z-score is 10.36;
|
|
- sim+outcome+telemetry: 19/396 exceed 3 and 9 exceed 5;
|
|
- the largest shifts include admitted input-length mean (10.36), waiting state
|
|
(7.77), running maximum (6.38), and decode-batch maximum (6.08).
|
|
|
|
Coefficient attribution shows that the input-length feature dominates several
|
|
wrong feasible-anchor logits. Because all training examples share one task,
|
|
the joint classifier can learn incidental within-task correlation and override
|
|
a correct simulator prior on TP2/MNS64-high and TP4/MNS64-high. This is a
|
|
supported diagnosis of model/data insufficiency; it is not a causal proof that
|
|
one feature alone caused the P1 failure.
|
|
|
|
More importantly, retuning lambda, threshold, features, or cutoff on P1 and
|
|
then calling P1 a held-out result would violate calibration/evaluation
|
|
separation. P1 may now be used only as development data.
|
|
|
|
## Decision and the only defensible reopening condition
|
|
|
|
Do not run registered P2/P3 with the current model. It failed the predeclared
|
|
gate on the favorable primary-trial denominator and is even farther from the
|
|
bar under end-to-end cost. Spending six-task headline GPU budget on the same
|
|
method would be metric shopping, not replication.
|
|
|
|
A new route may be opened only as a new hypothesis:
|
|
|
|
1. Replace the joint classifier with a **simulator-residual verifier**. The
|
|
simulator prediction remains an explicit prior; nested outcome-only and
|
|
telemetry models learn when that prior is wrong, rather than freely
|
|
relearning feasibility and overriding it under workload shift.
|
|
2. Train on multiple complete workload/SLO tasks. SLO thresholds and target
|
|
pass rate must be explicit inputs; splits are by complete task.
|
|
3. Calibrate abstention with task-level risk control. No threshold is selected
|
|
on a headline task, and “never early decide” is included as the safe
|
|
outcome-only baseline.
|
|
4. Treat Phase 6 and P1 as development only, freeze the residual architecture,
|
|
features, cutoff, threshold, simulator reading, and `k`, then use entirely
|
|
new trace windows for a new gate.
|
|
|
|
This reopening is justified only if development data show both (a) the
|
|
simulator's errors are predictable from pre-outcome engine state and (b) a
|
|
simulator-preserving residual model does not corrupt correct simulator
|
|
predictions. It is a new project decision, not a continuation automatically
|
|
authorized by P1.
|
|
|
|
## Benchmark audit
|
|
|
|
| Audit item | Verdict | Severity | Evidence / disposition |
|
|
|---|---|---|---|
|
|
| Calibration set separate from P1 | PASS | — | Phase 6/0311 trained; P1/0312 tested |
|
|
| Strong simulator-aware baseline | PASS | — | Identical Frontier features in both nested models |
|
|
| Sim top-k + real-final E2E baseline | PASS | — | Frozen `k=2`, tie expansion, measured setup/continuation cost |
|
|
| Multiple independent headline tasks | NEEDS EVIDENCE | Blocking for a positive claim | P1 gate failed; P2 correctly not opened |
|
|
| Statistical significance | NEEDS EVIDENCE | Blocking for a positive claim | n=12 anchors from one task; McNemar p=0.5 |
|
|
| Hyperparameter robustness | FAIL | Blocking | Lambda sensitivity changes safety and relative result |
|
|
| Full resource accounting | PASS for P1 | — | Failures, startup/warm-up/burn-in, continuation and annotation separated |
|
|
| Avoid post-test retuning | PASS only if route stops | Blocking if violated | P1 is now development-only |
|
|
| Selective winning-workload reporting | PASS | — | Negative P1 and TP/MNS losing cases retained |
|
|
|
|
Overall recommendation: **Block the current independent harness contribution
|
|
claim.**
|
|
|
|
## Artifacts
|
|
|
|
- Registered protocol: `docs/fidelity-aware-harness-protocol-20260714.md`
|
|
- Historical headroom: `docs/fidelity-aware-harness-headroom-20260714.md`
|
|
- Registered P1 analysis: `runs/fidelity-headroom/analyze_pilot.py`
|
|
- Strong P1 analysis: `runs/fidelity-headroom/analyze_strong_pilot.py`
|
|
- E2E shortlist replay: `runs/fidelity-headroom/analyze_pilot_e2e.py`
|
|
- External immutable result root:
|
|
`/home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714`
|
|
|
|
## Data sanity block
|
|
|
|
| Data | n | Min | Max | Distinct | Invariant |
|
|
|---|---:|---:|---:|---:|---|
|
|
| P1 labels | 12 | 0 | 1 | 2 | 7 feasible / 5 infeasible |
|
|
| Primary elapsed seconds | 12 | 19.448 | 61.435 | 12 | Every five-second prefix is in range |
|
|
| Prefix Layer-1 records | 12 | 332 | 557 | 12 | Contiguous; zero drops |
|
|
| Exact timestamped outcomes | 12 anchors | 54 | 750 | 11 | Monotonic completion timestamps |
|
|
| Simulator pass rate | 12 | 0.1548 | 1.0 | 7 | Ratios in `[0,1]` |
|
|
| Strong nested probabilities | 24 | 0.000208 | 0.809422 | 24 | Ratios in `[0,1]` |
|
|
| E2E cost components | 36 | 0.001389 | 0.169653 H20-h | 21 | Non-negative |
|
|
| GPU attempts | 2 | 0.020552 | 1.722112 H20-h | 2 | Aggregate 1.742664 < 3.5 |
|
|
| Copied raw files | 191 | — | 153,093,348 bytes total | — | Remote/local aggregate SHA identical |
|
|
|
|
Checked invariants: six cells and twelve anchors; exact request count and
|
|
request-ID/arrival/length hashes; all cell validation flags true; both labels
|
|
present; probabilities bounded; costs and counters non-negative; simulator
|
|
results not all identical; committed simulator rerun 12/12 numerically
|
|
identical to the exploratory run; no prompt text in public simulator fixtures;
|
|
no co-resident serving process; final eight GPUs at 0 MiB and 0% utilization.
|
|
No red flag remains.
|