Report failed fidelity pilot gate
This commit is contained in:
@@ -1,10 +1,16 @@
|
||||
# Fidelity-aware harness headroom audit
|
||||
|
||||
Status: **PROMISING PREMISE, NO CONTRIBUTION CLAIM**.
|
||||
Status: **HISTORICAL PREMISE DID NOT PASS PROSPECTIVE P1; NO CONTRIBUTION CLAIM**.
|
||||
|
||||
The audit answers whether engine instrumentation has enough incremental signal
|
||||
to justify a prospective experiment. It does not establish generalization.
|
||||
|
||||
Post-run update: the exact-timestamp held-out P1 completed and failed the
|
||||
registered gate. Under the stronger simulator-aware `k=2` end-to-end replay,
|
||||
telemetry preserved zero regret but saved only 1.426% online H20-hours versus
|
||||
sim top-k + real final. The current route is closed; see
|
||||
`docs/fidelity-aware-harness-p1-report-20260714.md`.
|
||||
|
||||
## Simulator shortlist lower bound
|
||||
|
||||
On the frozen 12-cell SimFid task, the strongest calibrated SLO simulator
|
||||
|
||||
242
docs/fidelity-aware-harness-p1-report-20260714.md
Normal file
242
docs/fidelity-aware-harness-p1-report-20260714.md
Normal file
@@ -0,0 +1,242 @@
|
||||
# Fidelity-aware harness P1 result
|
||||
|
||||
Status: **REGISTERED ROUTE REJECTED; DO NOT OPEN P2/P3 FOR THE CURRENT METHOD**.
|
||||
|
||||
Date: 2026-07-14 (Asia/Singapore).
|
||||
|
||||
## Outcome
|
||||
|
||||
The registered five-second instrumentation-aware verifier did not pass P1.
|
||||
The stronger simulator-aware comparison also failed the independent
|
||||
contribution bar. On the frozen `k=2` end-to-end replay:
|
||||
|
||||
- `sim top-k + real final` selected the real oracle with zero regret;
|
||||
- instrumentation-aware also selected the oracle, but reduced online H20-hours
|
||||
by only **1.426%** (1.329% when the prior failed attempt is added to both);
|
||||
- the required reduction was 30% versus full real final and 20% versus a safe
|
||||
outcome-only calibrator;
|
||||
- the outcome-only calibrator was not safe: it rejected the true best cell, so
|
||||
its apparent cost saving is not a deployable comparison.
|
||||
|
||||
This rejects the claim that the **current joint logistic verifier**, trained on
|
||||
one historical workload, gives the harness an independent tuning contribution.
|
||||
It does not prove that engine telemetry contains no useful signal. Telemetry
|
||||
improved held-out classification and removed unsafe decisions, but did not turn
|
||||
that signal into meaningful end-to-end tuning-cost reduction.
|
||||
|
||||
## Frozen setup
|
||||
|
||||
- Host: `dash0`, 8 NVIDIA H20 GPUs; cells were serialized and used TP1, TP2,
|
||||
or TP4 without co-resident serving jobs.
|
||||
- Engine/model: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16.
|
||||
- Workload: held-out `chat_w20260312_1000`, seven disjoint repeat bands,
|
||||
60-second replay after 0.1 time scaling, input `[0,8192]`, exactly 128 output
|
||||
tokens.
|
||||
- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, request pass rate at least 0.95.
|
||||
- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
|
||||
- Per cell: burn-in, three low-rate repeats, and three high-rate repeats. The
|
||||
first repeat supplied the five-second prefix; 2-of-3 supplied its label.
|
||||
- Models: the registered pair used config/workload/outcome versus the same
|
||||
vector plus Layer-1 engine telemetry. The strengthened pair additionally
|
||||
gave both models identical frozen Frontier throughput, SLO pass-rate, and
|
||||
feasibility predictions.
|
||||
- Policy: accept at `p>=0.95`, reject at `p<=0.05`, otherwise continue the same
|
||||
trial. Model, cutoff, threshold, role order, request hashes, and cap were
|
||||
frozen before their applicable evaluation.
|
||||
|
||||
The first launch failed its warm-up input-count validation before a measured
|
||||
anchor. It cost 0.020552 H20-hours. The corrected primary attempt cost
|
||||
1.722112 H20-hours, so aggregate campaign cost was **1.742664 H20-hours**, below
|
||||
the 3.5 cap. The fix changed only warm-up validation; formal request counts and
|
||||
hash checks were unchanged.
|
||||
|
||||
## P1 labels are not an artificial easy split
|
||||
|
||||
The 12 adjudicated anchor labels contain 7 feasible and 5 infeasible examples.
|
||||
They are not simply “low feasible, high infeasible”:
|
||||
|
||||
- TP2/MNS64 high was feasible in all three repeats;
|
||||
- TP4/MNS64 low and high were feasible in all six repeats;
|
||||
- TP4/MNS16 low and high were infeasible in all six repeats.
|
||||
|
||||
That last pair creates a large real MNS interaction under an otherwise matched
|
||||
TP4 configuration. Frontier correctly predicted TP4/MNS64 high as feasible,
|
||||
but incorrectly predicted TP4/MNS16 low as feasible. It also incorrectly
|
||||
predicted TP1/MNS64 high as feasible. Overall simulator-only feasibility was
|
||||
10/12 correct: 83.33% accuracy, with two false-feasible predictions and no
|
||||
false-infeasible prediction.
|
||||
|
||||
The two false-feasible cases expose the intended latent-state problem. At five
|
||||
seconds, all 26 completed TP4/MNS16-low requests and all 9 completed
|
||||
TP1/MNS64-high requests still passed their SLO, although both full anchors were
|
||||
infeasible. External outcomes had not yet exposed the future failure; queue,
|
||||
running-batch, and scheduler state existed before the tail outcome. This is
|
||||
mechanistic evidence that instrumentation can be useful, not evidence that the
|
||||
current learned policy uses it well enough.
|
||||
|
||||
## Registered and strengthened prefix results
|
||||
|
||||
At the frozen 0.95 policy threshold:
|
||||
|
||||
| Comparison | Accuracy | Balanced acc. | Early decisions | False accept | False reject | Valid primary-trial saving |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| Registered outcome-only | 41.67% | 50.00% | 6/12 | 0 | 2 | invalid |
|
||||
| Registered + telemetry | 66.67% | 71.43% | 4/12 | 0 | 0 | 11.44% |
|
||||
| Strong sim + outcome | 66.67% | 68.57% | 5/12 | 0 | 1 | invalid |
|
||||
| Strong sim + outcome + telemetry | 83.33% | 85.71% | 4/12 | 0 | 0 | 11.44% |
|
||||
|
||||
For the strong pair, telemetry was correct on two examples where the baseline
|
||||
was wrong and lost none; McNemar's exact two-sided p-value is 0.5 at `n=12`.
|
||||
This is a safety/classification improvement, not a cost contribution. The
|
||||
registered instrumentation policy made two fewer early decisions than its
|
||||
baseline, so it failed the registered `+3 decisions or +15 percentage points`
|
||||
incremental gate.
|
||||
|
||||
The result is not robust to the frozen regularization sensitivity:
|
||||
|
||||
| L2 lambda | Sim+outcome acc. | +telemetry acc. | Base policy errors | Telemetry policy errors | Base saving | Telemetry saving |
|
||||
|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 0.1 | 41.67% | 75.00% | 4 | 2 | invalid | invalid |
|
||||
| 1.0 | 66.67% | 83.33% | 1 | 0 | invalid | 11.44% |
|
||||
| 10.0 | 83.33% | 83.33% | 0 | 0 | 0.00% | 5.98% |
|
||||
|
||||
Consequently the positive classification delta is neither statistically nor
|
||||
hyperparameter robust.
|
||||
|
||||
## End-to-end shortlist result
|
||||
|
||||
Frontier's simulator-feasible ranking on the tested P1 surface was:
|
||||
|
||||
| Rank | Cell / anchor | Sim throughput/GPU | Real feasible | Real offered goodput/GPU |
|
||||
|---:|---|---:|---:|---:|
|
||||
| 1 | TP4/MNS64 high | 3.0718 | yes | 3.1250 |
|
||||
| 2 | TP1/MNS64 high | 2.8823 | no | 2.9833 |
|
||||
| 3 | TP2/MNS64 high | 2.8096 | yes | 2.8750 |
|
||||
| 4 | TP4/MNS16 low | 2.0866 | no | 2.1250 |
|
||||
| 5 | TP1/MNS8 low | 1.9806 | yes | 2.0333 |
|
||||
| 6 | TP2/MNS8 low | 1.8637 | yes | 1.9083 |
|
||||
|
||||
The real oracle was TP4/MNS64 high at 3.125 req/s/GPU. Cost includes an
|
||||
inferred per-cell startup/warm-up/burn-in/cleanup component and the selected
|
||||
anchor; benchmark-only 2-of-3 annotation intervals are removed. Gaps around
|
||||
annotation runs remain in the shared setup term, making this a conservative
|
||||
method-cost estimate.
|
||||
|
||||
| Frozen policy (`k=2`) | Online H20-hours | + prior failure | Real regret | Safety | Saving vs full |
|
||||
|---|---:|---:|---:|---|---:|
|
||||
| Sim top-2 + real final | 0.281383 | 0.301935 | 0.00% | valid | — |
|
||||
| Sim + outcome prefix | 0.214664 | 0.235216 | no selected cell | 1 false reject | invalid |
|
||||
| Sim + outcome + telemetry | 0.277370 | 0.297922 | 0.00% | valid | **1.426%** |
|
||||
|
||||
For `k=1`, the simulator already picked the oracle and telemetry abstained, so
|
||||
its saving was exactly 0%. For `k=3`, telemetry remained correct but saved
|
||||
1.007%. The result therefore does not depend on an unfavorable choice of `k`.
|
||||
|
||||
Across all six cells, a generous replay that removes benchmark-only repeat
|
||||
intervals estimates 0.944244 H20-hours for the full online workflow. The
|
||||
instrumentation policy saved 0.043467 H20-hours: 11.44% of primary trial time,
|
||||
but only **4.603%** of end-to-end online cost. Even a post-hoc oracle symmetric
|
||||
threshold cannot make the current telemetry model reach the contribution bar;
|
||||
its best zero-error envelope saves at most 5.69% of inferred online cost. A
|
||||
strong outcome-only model at another post-hoc regularization/threshold can save
|
||||
16.13%, which further prevents attributing a unique advantage to telemetry.
|
||||
These oracle-threshold numbers are diagnostics only and are not test evidence.
|
||||
|
||||
## Why the learned verifier did not generalize
|
||||
|
||||
The training corpus has only 37 anchors from one workload/SLO task. P1 shows
|
||||
large covariate shift:
|
||||
|
||||
- sim+outcome: 12/192 feature values exceed 3 training standard deviations and
|
||||
4 exceed 5; maximum absolute z-score is 10.36;
|
||||
- sim+outcome+telemetry: 19/396 exceed 3 and 9 exceed 5;
|
||||
- the largest shifts include admitted input-length mean (10.36), waiting state
|
||||
(7.77), running maximum (6.38), and decode-batch maximum (6.08).
|
||||
|
||||
Coefficient attribution shows that the input-length feature dominates several
|
||||
wrong feasible-anchor logits. Because all training examples share one task,
|
||||
the joint classifier can learn incidental within-task correlation and override
|
||||
a correct simulator prior on TP2/MNS64-high and TP4/MNS64-high. This is a
|
||||
supported diagnosis of model/data insufficiency; it is not a causal proof that
|
||||
one feature alone caused the P1 failure.
|
||||
|
||||
More importantly, retuning lambda, threshold, features, or cutoff on P1 and
|
||||
then calling P1 a held-out result would violate calibration/evaluation
|
||||
separation. P1 may now be used only as development data.
|
||||
|
||||
## Decision and the only defensible reopening condition
|
||||
|
||||
Do not run registered P2/P3 with the current model. It failed the predeclared
|
||||
gate on the favorable primary-trial denominator and is even farther from the
|
||||
bar under end-to-end cost. Spending six-task headline GPU budget on the same
|
||||
method would be metric shopping, not replication.
|
||||
|
||||
A new route may be opened only as a new hypothesis:
|
||||
|
||||
1. Replace the joint classifier with a **simulator-residual verifier**. The
|
||||
simulator prediction remains an explicit prior; nested outcome-only and
|
||||
telemetry models learn when that prior is wrong, rather than freely
|
||||
relearning feasibility and overriding it under workload shift.
|
||||
2. Train on multiple complete workload/SLO tasks. SLO thresholds and target
|
||||
pass rate must be explicit inputs; splits are by complete task.
|
||||
3. Calibrate abstention with task-level risk control. No threshold is selected
|
||||
on a headline task, and “never early decide” is included as the safe
|
||||
outcome-only baseline.
|
||||
4. Treat Phase 6 and P1 as development only, freeze the residual architecture,
|
||||
features, cutoff, threshold, simulator reading, and `k`, then use entirely
|
||||
new trace windows for a new gate.
|
||||
|
||||
This reopening is justified only if development data show both (a) the
|
||||
simulator's errors are predictable from pre-outcome engine state and (b) a
|
||||
simulator-preserving residual model does not corrupt correct simulator
|
||||
predictions. It is a new project decision, not a continuation automatically
|
||||
authorized by P1.
|
||||
|
||||
## Benchmark audit
|
||||
|
||||
| Audit item | Verdict | Severity | Evidence / disposition |
|
||||
|---|---|---|---|
|
||||
| Calibration set separate from P1 | PASS | — | Phase 6/0311 trained; P1/0312 tested |
|
||||
| Strong simulator-aware baseline | PASS | — | Identical Frontier features in both nested models |
|
||||
| Sim top-k + real-final E2E baseline | PASS | — | Frozen `k=2`, tie expansion, measured setup/continuation cost |
|
||||
| Multiple independent headline tasks | NEEDS EVIDENCE | Blocking for a positive claim | P1 gate failed; P2 correctly not opened |
|
||||
| Statistical significance | NEEDS EVIDENCE | Blocking for a positive claim | n=12 anchors from one task; McNemar p=0.5 |
|
||||
| Hyperparameter robustness | FAIL | Blocking | Lambda sensitivity changes safety and relative result |
|
||||
| Full resource accounting | PASS for P1 | — | Failures, startup/warm-up/burn-in, continuation and annotation separated |
|
||||
| Avoid post-test retuning | PASS only if route stops | Blocking if violated | P1 is now development-only |
|
||||
| Selective winning-workload reporting | PASS | — | Negative P1 and TP/MNS losing cases retained |
|
||||
|
||||
Overall recommendation: **Block the current independent harness contribution
|
||||
claim.**
|
||||
|
||||
## Artifacts
|
||||
|
||||
- Registered protocol: `docs/fidelity-aware-harness-protocol-20260714.md`
|
||||
- Historical headroom: `docs/fidelity-aware-harness-headroom-20260714.md`
|
||||
- Registered P1 analysis: `runs/fidelity-headroom/analyze_pilot.py`
|
||||
- Strong P1 analysis: `runs/fidelity-headroom/analyze_strong_pilot.py`
|
||||
- E2E shortlist replay: `runs/fidelity-headroom/analyze_pilot_e2e.py`
|
||||
- External immutable result root:
|
||||
`/home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714`
|
||||
|
||||
## Data sanity block
|
||||
|
||||
| Data | n | Min | Max | Distinct | Invariant |
|
||||
|---|---:|---:|---:|---:|---|
|
||||
| P1 labels | 12 | 0 | 1 | 2 | 7 feasible / 5 infeasible |
|
||||
| Primary elapsed seconds | 12 | 19.448 | 61.435 | 12 | Every five-second prefix is in range |
|
||||
| Prefix Layer-1 records | 12 | 332 | 557 | 12 | Contiguous; zero drops |
|
||||
| Exact timestamped outcomes | 12 anchors | 54 | 750 | 11 | Monotonic completion timestamps |
|
||||
| Simulator pass rate | 12 | 0.1548 | 1.0 | 7 | Ratios in `[0,1]` |
|
||||
| Strong nested probabilities | 24 | 0.000208 | 0.809422 | 24 | Ratios in `[0,1]` |
|
||||
| E2E cost components | 36 | 0.001389 | 0.169653 H20-h | 21 | Non-negative |
|
||||
| GPU attempts | 2 | 0.020552 | 1.722112 H20-h | 2 | Aggregate 1.742664 < 3.5 |
|
||||
| Copied raw files | 191 | — | 153,093,348 bytes total | — | Remote/local aggregate SHA identical |
|
||||
|
||||
Checked invariants: six cells and twelve anchors; exact request count and
|
||||
request-ID/arrival/length hashes; all cell validation flags true; both labels
|
||||
present; probabilities bounded; costs and counters non-negative; simulator
|
||||
results not all identical; committed simulator rerun 12/12 numerically
|
||||
identical to the exploratory run; no prompt text in public simulator fixtures;
|
||||
no co-resident serving process; final eight GPUs at 0 MiB and 0% utilization.
|
||||
No red flag remains.
|
||||
@@ -1,9 +1,18 @@
|
||||
# Fidelity-aware real-verification harness protocol
|
||||
|
||||
Status: **PRE-REGISTERED STAGED EVALUATION; CONTRIBUTION NOT YET ESTABLISHED**.
|
||||
Status: **P1 FAILED; P2/P3 CLOSED FOR THE REGISTERED METHOD; CONTRIBUTION NOT ESTABLISHED**.
|
||||
|
||||
Date frozen: 2026-07-14 (Asia/Singapore).
|
||||
|
||||
Post-run disposition (2026-07-14): P1 completed with valid data but failed its
|
||||
registered incremental gate. The strengthened simulator-aware comparison and
|
||||
end-to-end `k=2` replay also failed: instrumentation was safe and retained zero
|
||||
regret, but reduced online H20-hours by only 1.426% versus sim top-k + real
|
||||
final, against the 30% bar. Outcome-only was unsafe. P2/P3 are therefore not
|
||||
opened for this model. Full results and the permitted reopening condition are
|
||||
in `docs/fidelity-aware-harness-p1-report-20260714.md`; the protocol below is
|
||||
retained unchanged as the pre-run record.
|
||||
|
||||
## Research question and contribution bar
|
||||
|
||||
The harness has an independent systems contribution only if engine-internal
|
||||
|
||||
Reference in New Issue
Block a user