aituner/docs/fidelity-aware-harness-protocol-20260714.md

# Fidelity-aware real-verification harness protocol

Status: **P1 FAILED; P2/P3 CLOSED FOR THE REGISTERED METHOD; CONTRIBUTION NOT ESTABLISHED**.

Date frozen: 2026-07-14 (Asia/Singapore).

Post-run disposition (2026-07-14): P1 completed with valid data but failed its
registered incremental gate.  The strengthened simulator-aware comparison and
end-to-end `k=2` replay also failed: instrumentation was safe and retained zero
regret, but reduced online H20-hours by only 1.426% versus sim top-k + real
final, against the 30% bar.  Outcome-only was unsafe.  P2/P3 are therefore not
opened for this model.  Full results and the permitted reopening condition are
in `docs/fidelity-aware-harness-p1-report-20260714.md`; the protocol below is
retained unchanged as the pre-run record.

## Research question and contribution bar

The harness has an independent systems contribution only if engine-internal
instrumentation improves a tuning decision beyond what is already achievable
with a simulator shortlist and external benchmark outcomes.  The intended
claim is therefore deliberately stronger than “telemetry explains a run”:

> Given the same simulator ranking, the same candidate order, and the same
> short real-GPU probe, a learned instrumentation-aware verifier reaches a
> configuration with at most 5% real SLO-goodput regret using materially fewer
> H20-hours than both (a) simulator top-k followed by full real evaluation and
> (b) an outcome-only verifier given exactly the same probe.

The paper-facing gate is:

- at least 20% lower real-verification H20-hours than outcome-only calibration;
- at least 30% lower real-verification H20-hours than simulator top-k plus full
  real final evaluation;
- paired 95% task-bootstrap confidence interval for the outcome-only cost
  reduction strictly above zero;
- selected-configuration SLO-goodput regret at most 5% on every headline task;
- no false-safe early accept in the pilot and at most 1% in the expanded suite;
- profiling, warm-up, confirmation, instrumentation, and failed-run costs are
  included rather than amortized away.  An amortized profile-cost view may be
  reported only as a secondary result.

If these conditions fail, instrumentation remains a debugging facility.  It is
not an independent tuning-harness contribution.

## What is learned, and what is not a rule

The decision target is a stable, repeated real verdict, not a hand-authored
diagnosis such as “queue length above N means reject.”  Each anchor receives
three full real repetitions and a frozen 2-of-3 feasibility label.  A nested
pair of regularized models predicts that label from a fixed prefix:

- **Outcome-only input X:** configuration, offered rate, admitted/completed
  progress, observed TTFT/TPOT margins, failures, and known workload lengths.
- **Instrumentation input Z:** the same X plus generic engine state: running and
  waiting queues, decode-batch shape, KV usage, graph mode and padding, prefill
  share, preemptions, and model-step rate.

Both models use the same L2 logistic family, train split, standardization,
regularization, cutoff, and probability threshold.  The only experimental
difference is Z.  The initial family is intentionally simple: a positive result
then demonstrates value in the engine signal rather than capacity in a larger
learner.  A sequence model is admissible only as a later, paired ablation.

### Amendment A1: strengthen the calibration baseline before P2

Frozen 2026-07-14 13:08 Asia/Singapore, after P1 launch but before P1
completion or analysis.  A baseline audit found that the first frozen P1
models use the simulator only to define candidate order; their feature vectors
do not contain the simulator's per-anchor prediction.  This is insufficient
for the stronger term **outcome-only calibration**.  P1 therefore remains a
prospective test of the originally frozen cross-workload predictor, but cannot
by itself open a contribution claim.

For P2/P3, both nested models must additionally receive the identical frozen
simulator outputs available at that decision: predicted completed throughput
per GPU, predicted SLO pass rate, and predicted feasibility.  The comparison
is consequently `sim + config + workload + real outcome prefix` versus that
exact vector plus real engine state.  Simulator features, regularization,
cutoff, and thresholds are frozen before any P2 task.  If telemetry does not
improve this stronger baseline, the harness has no independent contribution.

The same audit also separates algorithm cost from benchmark-oracle cost.
Headline method cost includes every action the method would execute online:
simulator profiling/calibration, model onboarding, server startup, warm-up,
real prefix, continuation after abstention, method-requested confirmation,
logging overhead, failures, and cleanup.  Exhaustive real-oracle runs and the
extra repetitions used only to construct 2-of-3 evaluation labels are common
benchmark annotation cost; they are reported separately and charged to no
method.  A second, deliberately conservative table adds that common cost to
all methods.  This prevents both hiding real method cost and making the
percentage gate mathematically depend on offline ground-truth annotation.

The frozen first policy uses a 5-second prefix, L2 regularization 1.0, and a
two-sided abstaining threshold of 0.95: accept at `p(feasible)>=0.95`, reject at
`p(feasible)<=0.05`, otherwise continue the exact same trial to completion.
Threshold and cutoff were selected on the historical training task and are
therefore not evidence; all claims come from subsequent held-out tasks.

## Fair baselines

| Method | Simulator | 5-second real prefix | External outcomes | Engine state | Full real continuation |
|---|---:|---:|---:|---:|---:|
| Real-only oracle | no | no | full | optional diagnostic | every candidate/anchor |
| Sim top-k + real final | yes | included in full run | full | no decision use | every shortlisted candidate/anchor |
| Outcome-only calibration | yes, including its prediction features | yes | yes | no | only on abstention |
| Instrumentation-aware | same prediction features | yes | yes | yes | only on abstention |

Tie buckets are expanded before top-k.  `k` is selected on training tasks and
is fixed on held-out tasks; an oracle per-task k is forbidden.  Outcome-only
receives all information available outside the engine, including config,
workload, and frozen simulator-prediction features.  Instrumentation cannot use
any record submitted after the cutoff.  The full label, confirmation votes,
realized simulator error, and later requests are never model features.

## Staged experiment

### R0: historical premise and headroom audit

The frozen SimFid surface has 12 cells.  The strongest calibrated SLO simulator
reading has a top tie bucket `{TP2/MNS32, TP2/MNS64}`; full real evaluation of
those two cells already finds the oracle with zero regret.  Consequently this
single task cannot demonstrate a selection-count advantage: any method needing
one real calibration probe and one real final verification has a lower bound of
two real cells.

The viable estimand is instead the duration and number of full real frontier
evaluations inside a fixed shortlist.  Historical Phase-6 prefixes are analyzed
only as training/premise data.  Their request completion times are reconstructed
from arrival, TTFT, TPOT, and token count, so they cannot support a final claim.

### P1: exact-timestamp prospective pilot

- Engine/model/hardware: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B, one solo
  server/client on dash0, NVIDIA H20, `TP in {1,2,4}`.
- Held-out workload: `chat_w20260312_1000`, 60-second replay after the frozen
  0.1 time scale, raw input `[0,8192]`, exactly 128 output tokens.
- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, 95% request pass rate.
- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
- Per cell: one attainable low offered rate near 0.85x the historical v0.24
  frontier and one high rate near 1.25x.  The exact threshold and selected
  request hashes are frozen by a CPU preflight before launch.
- Each cell uses a fresh server, the accepted long-request warm-up, one
  unmeasured full-window burn-in, then three repetitions per rate.  Rate order
  alternates and reverses across cells to prevent a fixed warm-state/order
  confound.
- The first repetition supplies the exact prefix.  All three repetitions supply
  the 2-of-3 label.  Every request records a monotonic completion timestamp;
  Layer-1 records are cut at the same monotonic boundary.
- Placement is serialized.  Co-location is forbidden because Phase 6 observed
  up to 92.86 percentage-point pass-rate shifts under co-location.
- Hard cap: 3.5 H20-hours, including startup, warm-up, burn-in, all repetitions,
  failures, and cleanup.  Projected cap violation stops before the next cell.

P1 opens P2 only if all data invariants pass and instrumentation-aware has zero
false accept/reject, is no worse than outcome-only, and either makes at least
three additional correct early decisions or improves total valid trial-cost
reduction by at least 15 absolute percentage points.  The pilot is a gate, not
paper evidence.

### P2: held-out task replication

If P1 passes, freeze the model and run at least six independent task groups:
three trace windows spanning distinct date/slot combinations and two SLO
regimes.  No task used for threshold/model selection enters the headline test.
The candidate surface is the full 12-cell `TP={1,2,4} x MNS={8,16,32,64}`
surface.  Splits are by complete task, never by anchor or request.  A task-level
paired bootstrap (10,000 repetitions, fixed seed) estimates cost and regret
intervals.  Non-monotonic or split 2-of-3 anchors remain explicit; no frontier
is imputed.

### P3: end-to-end shortlist and search replay

For each P2 task, run the same frozen simulator and tie-expanded top-k policy.
Replay the real binary/frontier search under all three verification policies:
full real, outcome-only, and instrumentation-aware.  The policy consumes only
prefixes that would have been available at that decision point.  Report:

- selected cell and real SLO-goodput regret;
- number of real cells, anchors, and confirmations;
- measured H20-hours and wall time;
- false accept, false reject, and abstention counts;
- profile, startup/warm-up, probe, full-continuation, confirmation, logging, and
  failure cost breakdowns.

### P4: simulator-rank-error attribution

This phase distinguishes an outdated implementation/profile from a structural
simulator limitation.  For each held-out task compare:

1. the original simulator/profile;
2. a version-matched re-profiled simulator;
3. a trajectory-conditioned run supplied with the realized arrival and request
   length sequence;
4. outcome-only residual calibration;
5. instrumentation-aware residual calibration.

The engine trace is extended only as needed with a worker-level step UID and
CUDA-event duration, because current async submit-to-complete spans overlap and
are not GPU step time.  Residuals are decomposed into operator-profile error,
scheduler/state error, and run-to-run noise.  If re-profiling alone restores the
ranking, the old 30% loss was an implementation/profile defect.  If exact
profiles and realized trajectories still mis-rank cells, and the residual is
systematically explained by queue/KV/graph/batch state unavailable to the
simulator, that is evidence of a structural state-abstraction gap.  Correlation
alone is not called causal.

## Failure modes that reject the route

- Outcome-only matches or beats instrumentation-aware under the same cutoff.
- Instrumentation gains average accuracy but introduces false-safe decisions.
- Gains disappear under task-level rather than request/anchor-level splitting.
- Savings come only from excluding startup, warm-up, profiling, confirmations,
  or failed trials.
- A different cutoff/threshold must be selected after seeing each test task.
- The simulator top-k baseline already reaches the target with equal or lower
  total H20-hours.
- Exact instrumentation overhead exceeds 1% throughput or materially changes
  p95/p99 latency.
- Results depend on TP4 transient/non-monotonic trials and do not replicate on
  held-out tasks.

## Data sanity contract

Every analysis ends with n, min/max, distinct count, label balance, and these
invariants: non-negative counters/costs; probabilities and ratios in `[0,1]`;
per-config results not all identical; timestamps monotonic; every prefix record
at or before its cutoff; selected request ID/arrival/length hashes stable across
repetitions; exact 128-token completion or counted failure; no dropped Layer-1
records; 2-of-3 labels reproducible; no co-resident GPU process; total H20-hours
below the hard cap; final GPUs idle.  A red flag is reported first and blocks
the contribution claim.