aituner/docs/fidelity-aware-harness-protocol-20260714.md

Fidelity-aware real-verification harness protocol

Status: P1 FAILED; P2/P3 CLOSED FOR THE REGISTERED METHOD; CONTRIBUTION NOT ESTABLISHED.

Date frozen: 2026-07-14 (Asia/Singapore).

Post-run disposition (2026-07-14): P1 completed with valid data but failed its registered incremental gate. The strengthened simulator-aware comparison and end-to-end k=2 replay also failed: instrumentation was safe and retained zero regret, but reduced online H20-hours by only 1.426% versus sim top-k + real final, against the 30% bar. Outcome-only was unsafe. P2/P3 are therefore not opened for this model. Full results and the permitted reopening condition are in docs/fidelity-aware-harness-p1-report-20260714.md; the protocol below is retained unchanged as the pre-run record.

Research question and contribution bar

The harness has an independent systems contribution only if engine-internal instrumentation improves a tuning decision beyond what is already achievable with a simulator shortlist and external benchmark outcomes. The intended claim is therefore deliberately stronger than “telemetry explains a run”:

Given the same simulator ranking, the same candidate order, and the same short real-GPU probe, a learned instrumentation-aware verifier reaches a configuration with at most 5% real SLO-goodput regret using materially fewer H20-hours than both (a) simulator top-k followed by full real evaluation and (b) an outcome-only verifier given exactly the same probe.

The paper-facing gate is:

• at least 20% lower real-verification H20-hours than outcome-only calibration;
• at least 30% lower real-verification H20-hours than simulator top-k plus full real final evaluation;
• paired 95% task-bootstrap confidence interval for the outcome-only cost reduction strictly above zero;
• selected-configuration SLO-goodput regret at most 5% on every headline task;
• no false-safe early accept in the pilot and at most 1% in the expanded suite;
• profiling, warm-up, confirmation, instrumentation, and failed-run costs are included rather than amortized away. An amortized profile-cost view may be reported only as a secondary result.

If these conditions fail, instrumentation remains a debugging facility. It is not an independent tuning-harness contribution.

What is learned, and what is not a rule

The decision target is a stable, repeated real verdict, not a hand-authored diagnosis such as “queue length above N means reject.” Each anchor receives three full real repetitions and a frozen 2-of-3 feasibility label. A nested pair of regularized models predicts that label from a fixed prefix:

• Outcome-only input X: configuration, offered rate, admitted/completed progress, observed TTFT/TPOT margins, failures, and known workload lengths.
• Instrumentation input Z: the same X plus generic engine state: running and waiting queues, decode-batch shape, KV usage, graph mode and padding, prefill share, preemptions, and model-step rate.

Both models use the same L2 logistic family, train split, standardization, regularization, cutoff, and probability threshold. The only experimental difference is Z. The initial family is intentionally simple: a positive result then demonstrates value in the engine signal rather than capacity in a larger learner. A sequence model is admissible only as a later, paired ablation.

Amendment A1: strengthen the calibration baseline before P2

Frozen 2026-07-14 13:08 Asia/Singapore, after P1 launch but before P1 completion or analysis. A baseline audit found that the first frozen P1 models use the simulator only to define candidate order; their feature vectors do not contain the simulator's per-anchor prediction. This is insufficient for the stronger term outcome-only calibration. P1 therefore remains a prospective test of the originally frozen cross-workload predictor, but cannot by itself open a contribution claim.

For P2/P3, both nested models must additionally receive the identical frozen simulator outputs available at that decision: predicted completed throughput per GPU, predicted SLO pass rate, and predicted feasibility. The comparison is consequently sim + config + workload + real outcome prefix versus that exact vector plus real engine state. Simulator features, regularization, cutoff, and thresholds are frozen before any P2 task. If telemetry does not improve this stronger baseline, the harness has no independent contribution.

The same audit also separates algorithm cost from benchmark-oracle cost. Headline method cost includes every action the method would execute online: simulator profiling/calibration, model onboarding, server startup, warm-up, real prefix, continuation after abstention, method-requested confirmation, logging overhead, failures, and cleanup. Exhaustive real-oracle runs and the extra repetitions used only to construct 2-of-3 evaluation labels are common benchmark annotation cost; they are reported separately and charged to no method. A second, deliberately conservative table adds that common cost to all methods. This prevents both hiding real method cost and making the percentage gate mathematically depend on offline ground-truth annotation.

The frozen first policy uses a 5-second prefix, L2 regularization 1.0, and a two-sided abstaining threshold of 0.95: accept at p(feasible)>=0.95, reject at p(feasible)<=0.05, otherwise continue the exact same trial to completion. Threshold and cutoff were selected on the historical training task and are therefore not evidence; all claims come from subsequent held-out tasks.

Fair baselines

Method	Simulator	5-second real prefix	External outcomes	Engine state	Full real continuation
Real-only oracle	no	no	full	optional diagnostic	every candidate/anchor
Sim top-k + real final	yes	included in full run	full	no decision use	every shortlisted candidate/anchor
Outcome-only calibration	yes, including its prediction features	yes	yes	no	only on abstention
Instrumentation-aware	same prediction features	yes	yes	yes	only on abstention

Tie buckets are expanded before top-k. k is selected on training tasks and is fixed on held-out tasks; an oracle per-task k is forbidden. Outcome-only receives all information available outside the engine, including config, workload, and frozen simulator-prediction features. Instrumentation cannot use any record submitted after the cutoff. The full label, confirmation votes, realized simulator error, and later requests are never model features.

Staged experiment

R0: historical premise and headroom audit

The frozen SimFid surface has 12 cells. The strongest calibrated SLO simulator reading has a top tie bucket {TP2/MNS32, TP2/MNS64}; full real evaluation of those two cells already finds the oracle with zero regret. Consequently this single task cannot demonstrate a selection-count advantage: any method needing one real calibration probe and one real final verification has a lower bound of two real cells.

The viable estimand is instead the duration and number of full real frontier evaluations inside a fixed shortlist. Historical Phase-6 prefixes are analyzed only as training/premise data. Their request completion times are reconstructed from arrival, TTFT, TPOT, and token count, so they cannot support a final claim.

P1: exact-timestamp prospective pilot

• Engine/model/hardware: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B, one solo server/client on dash0, NVIDIA H20, TP in {1,2,4}.
• Held-out workload: chat_w20260312_1000, 60-second replay after the frozen 0.1 time scale, raw input [0,8192], exactly 128 output tokens.
• SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, 95% request pass rate.
• Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
• Per cell: one attainable low offered rate near 0.85x the historical v0.24 frontier and one high rate near 1.25x. The exact threshold and selected request hashes are frozen by a CPU preflight before launch.
• Each cell uses a fresh server, the accepted long-request warm-up, one unmeasured full-window burn-in, then three repetitions per rate. Rate order alternates and reverses across cells to prevent a fixed warm-state/order confound.
• The first repetition supplies the exact prefix. All three repetitions supply the 2-of-3 label. Every request records a monotonic completion timestamp; Layer-1 records are cut at the same monotonic boundary.
• Placement is serialized. Co-location is forbidden because Phase 6 observed up to 92.86 percentage-point pass-rate shifts under co-location.
• Hard cap: 3.5 H20-hours, including startup, warm-up, burn-in, all repetitions, failures, and cleanup. Projected cap violation stops before the next cell.

P1 opens P2 only if all data invariants pass and instrumentation-aware has zero false accept/reject, is no worse than outcome-only, and either makes at least three additional correct early decisions or improves total valid trial-cost reduction by at least 15 absolute percentage points. The pilot is a gate, not paper evidence.

P2: held-out task replication

If P1 passes, freeze the model and run at least six independent task groups: three trace windows spanning distinct date/slot combinations and two SLO regimes. No task used for threshold/model selection enters the headline test. The candidate surface is the full 12-cell TP={1,2,4} x MNS={8,16,32,64} surface. Splits are by complete task, never by anchor or request. A task-level paired bootstrap (10,000 repetitions, fixed seed) estimates cost and regret intervals. Non-monotonic or split 2-of-3 anchors remain explicit; no frontier is imputed.

P3: end-to-end shortlist and search replay

For each P2 task, run the same frozen simulator and tie-expanded top-k policy. Replay the real binary/frontier search under all three verification policies: full real, outcome-only, and instrumentation-aware. The policy consumes only prefixes that would have been available at that decision point. Report:

• selected cell and real SLO-goodput regret;
• number of real cells, anchors, and confirmations;
• measured H20-hours and wall time;
• false accept, false reject, and abstention counts;
• profile, startup/warm-up, probe, full-continuation, confirmation, logging, and failure cost breakdowns.

P4: simulator-rank-error attribution

This phase distinguishes an outdated implementation/profile from a structural simulator limitation. For each held-out task compare:

1. the original simulator/profile;
2. a version-matched re-profiled simulator;
3. a trajectory-conditioned run supplied with the realized arrival and request length sequence;
4. outcome-only residual calibration;
5. instrumentation-aware residual calibration.

The engine trace is extended only as needed with a worker-level step UID and CUDA-event duration, because current async submit-to-complete spans overlap and are not GPU step time. Residuals are decomposed into operator-profile error, scheduler/state error, and run-to-run noise. If re-profiling alone restores the ranking, the old 30% loss was an implementation/profile defect. If exact profiles and realized trajectories still mis-rank cells, and the residual is systematically explained by queue/KV/graph/batch state unavailable to the simulator, that is evidence of a structural state-abstraction gap. Correlation alone is not called causal.

Failure modes that reject the route

• Outcome-only matches or beats instrumentation-aware under the same cutoff.
• Instrumentation gains average accuracy but introduces false-safe decisions.
• Gains disappear under task-level rather than request/anchor-level splitting.
• Savings come only from excluding startup, warm-up, profiling, confirmations, or failed trials.
• A different cutoff/threshold must be selected after seeing each test task.
• The simulator top-k baseline already reaches the target with equal or lower total H20-hours.
• Exact instrumentation overhead exceeds 1% throughput or materially changes p95/p99 latency.
• Results depend on TP4 transient/non-monotonic trials and do not replicate on held-out tasks.

Data sanity contract

Every analysis ends with n, min/max, distinct count, label balance, and these invariants: non-negative counters/costs; probabilities and ratios in [0,1]; per-config results not all identical; timestamps monotonic; every prefix record at or before its cutoff; selected request ID/arrival/length hashes stable across repetitions; exact 128-token completion or counted failure; no dropped Layer-1 records; 2-of-3 labels reproducible; no co-resident GPU process; total H20-hours below the hard cap; final GPUs idle. A red flag is reported first and blocks the contribution claim.