aituner/docs/telemetry-residual-tuning-roadmap-20260714.md

Telemetry-conditioned residual tuning roadmap

Status: R0 COMPLETE / FAILED; R1 AND R2 CLOSED FOR THIS MODEL.

Date: 2026-07-14 (Asia/Singapore).

Research question and claim boundary

The question is whether a small number of real engine observations can correct a simulator's task-specific error over unmeasured configurations, and whether that correction reduces the real-GPU cost of finding a high SLO-goodput serving configuration.

The intended headline claim, if the evidence supports it, is:

An engine-state-conditioned residual model turns a simulator prediction into a task-specific posterior over unmeasured serving configurations, allowing a sequential tuner to reach near-oracle SLO-goodput with materially fewer H20-hours than simulator-only and outcome-only tuning.

Classification accuracy, simulator-error diagnosis, and telemetry overhead are supporting evidence. None is an end-to-end tuning contribution by itself.

The following method is closed and will not be revived under another name: per-candidate five-second accept/reject as the headline contribution. The P1 result showed only 1.426% cost reduction in the frozen k=2 workflow.

Two models, one evaluation

Both branches use the same legal candidate set, real measurements, task split, cost accounting, and acquisition function.

Simulator-residual branch (primary)

For measured anchor c_t and unmeasured candidate c':

y_hat(c') = y_real(c_t)
          + [y_sim(c') - y_sim(c_t)]
          + f(state_real(c_t) - state_sim(c_t), c' - c_t, workload, SLO)

The simulator delta is the prior. The learned model may correct it only with training-supported state/config transitions; uncertainty or distribution shift must shrink the correction back toward the simulator prior.

Telemetry-only branch (mandatory)

y_hat(c') = y_real(c_t)
          + g(state_real(c_t), c' - c_t, workload, SLO)

This branch tests whether the simulator is actually necessary. It does not use a hand-authored bottleneck-to-knob rule.

Search policy

Legal configurations are enumerated independently of telemetry. A generic cost-aware acquisition rule ranks candidates from predicted improvement, uncertainty, and measured H20 cost. The current production harness's bottleneck scores, topology-first ordering, and hand-set relief constants are not consumed by either branch. The validator may enforce legality, full-config no-repeat, failure accounting, and resource caps only.

Hypotheses

ID	Hypothesis	Direct test	Failure meaning
H0	Existing artifacts can express a common, direct-measurement state without heuristic labels.	Engine/simulator extractor coverage and invariants.	Route is not currently implementable.
H1	Simulator errors are predictable from engine/simulator state discrepancy at measured anchors.	Task-held-out pairwise inversion correction and new-inversion rate.	Telemetry is diagnostic but cannot correct the surface.
H2	Telemetry alone predicts useful config transitions beyond outcome-only history.	Telemetry-only versus real-outcome-only sequential replay.	Direct telemetry-guided tuning has no independent value.
H3	Residual correction changes actual tuning decisions and cost.	H20-hours to 95% oracle and regret AUC against the strongest safe baseline.	No system contribution even if H1/H2 prediction metrics improve.

Common-state contract

Only directly observed or exactly reconstructed quantities are admitted.

Quantity	vLLM Layer-1	Frontier	R0 status
Scheduled requests / batch size	Per scheduler step	Existing per-batch metric, disabled in P1 output	Common after CPU replay
Scheduled prefill/decode tokens	Per scheduler step	Existing per-batch metrics	Common after CPU replay
Scheduler/batch rate	Monotonic step timestamps	Batch count / simulated duration	Common after CPU replay
Waiting queue area	Time-weighted queue gauge	Sum of request waiting times	Common aggregate
Running request area	Time-weighted running gauge	Sum of E2E minus waiting time	Common aggregate, semantics audited
Preemption count	Per step	Per request	Common
KV usage/headroom	Exact blocks and ratio	Not in committed output	Engine-only until exact reconstruction exists
CUDA graph mode/padding	Exact per step	Not modeled	Engine-only omitted-mechanism signal
Request TTFT/TPOT/pass rate	Exact real outcomes	Exact simulated request metrics	Common outcome, not state

Unavailable fields remain null. They cannot be imputed from a human prefill/decode/queueing label.

Frontier already contains the required detailed batch and timestamped stage-batch ledger output. P1 disabled it for artifact size. R0 replays the same immutable fixtures with the existing output flags enabled; it does not change the simulator model or calibration.

Data separation

• Phase 6 / chat_w20260311_1000: development only.
• P1 / chat_w20260312_1000: development only.
• R1 / chat_w20260313_1000: new development surface.
• R2: trace windows not used for feature, model, threshold, candidate-space, cutoff, or acquisition decisions.
• Splits are by complete workload/SLO task. Anchor- or pair-level random splits are prohibited.
• Sequential-policy seeds measure algorithmic variability; they are not counted as independent system tasks.

The two existing development tasks have an important limitation: the now- available SLO-gated simulator reading already retains the real oracle at its top rank/tie. They therefore cannot establish a positive end-to-end ranking claim. They are used for plumbing, known false-feasible cases, and negative evidence. R1 must be run as an unbiased complete surface, not selected after observing simulator success or failure.

Step-by-step roadmap

R0.1 — Inventory and roadmap

Deliverables:

• this roadmap;
• rolling untracked ONGOING.md;
• exact engine/simulator field and artifact inventory.

Gate: every claimed input has an authoritative file path and provenance.

R0.2 — Common-state plumbing

Deliverables:

• runs/telemetry-residual/common_state.py;
• synthetic correctness tests;
• one exact P1 Frontier replay with individual batch metrics and the full stage-batch ledger enabled;
• paired engine/simulator state summary for the same fixture.

Gate:

• replay request count and SLO scorer exactly agree with the committed replay;
• batch/ledger outputs are non-empty;
• all counters are non-negative, ratios bounded, times monotonic;
• no GPU is visible to Frontier;
• output volume is practical before expanding to twelve replays.

R0.3 — Development residual/headroom audit

Use all frozen P1 primary fixtures and corresponding engine intervals. Produce:

• common-state residuals per anchor;
• simulator-error labels and continuous SLO/goodput residuals;
• ordered source/target diagnostic that removes both config identities from both roles in every training fold;
• oracle upper bound for cross-candidate correction;
• explicit comparison with simulator+outcome and telemetry-only features.

R0 is a feasibility gate, not headline evidence. Proceed to R1 only if:

1. state features are collected with the measured source anchor, vary across cells, and are available before any target config is evaluated;
2. at least one known simulator error has a state discrepancy not exposed by the matched external prefix outcome;
3. a prior-preserving model can correct development errors without introducing a larger number of new errors under regularization sensitivity;
4. an oracle cross-candidate correction has at least 15% sequential tuning-cost headroom under full startup/warm-up accounting.

R0 result and decision

R0 completed without a data-validity red flag, but failed condition 3. The decision is STOP_BEFORE_R1; no H20 job was launched for this route.

• All 12 detailed Frontier CPU replays exactly reproduced their committed SLO scorers. Runtime was 23.943--54.786 seconds per replay, detailed artifacts were 4.12--13.53 MB, CUDA visibility was empty, and there were zero failures.
• The paired surface contains 12 real/sim anchors, two known simulator false-feasible anchors, and 120 legal cross-config ordered transitions. A fold removes both the source and target TP/MNS identity from source and target roles; the two offered-load anchors remain part of the same task.
• Raw Frontier feasibility is 83.33% on the repeated transition view. The structurally correct hybrid model uses r_target = r_source + delta_r; the direct model uses y_target = y_source + delta_y and never reads simulator fields.
• Direct telemetry is not robust relative to real-outcome-only: its accuracy delta over L2 {0.1,1,10,100} is {-0.83,+1.67,0,-4.17} percentage points, and its best absolute accuracy is 54.17%, below the raw simulator's 83.33%.
• Hybrid telemetry raises classification accuracy over the corresponding simulator+outcome transition regression by 1.67--4.17 percentage points, but worsens pass-rate RMSE by 0.141--0.201 and MAE by 0.084--0.125. Its full correction reaches only 46.67--53.33% absolute accuracy.
• Across 24 nonzero (L2, raw-simulator-prior weight) combinations, no model both corrects an existing simulator error without more new errors and avoids worsening RMSE/MAE. Whenever a correction fixes at least one error, it corrupts at least 11 previously correct transitions.
• A perfect correction could skip the frozen simulator rank-2 real final and save 0.043469 H20-hours: 15.45% of the prospective online k=2 cost, or 14.40% when the prior failed launch is charged. On this development task the simulator top-1 already is the real oracle with zero regret, so headroom versus the observed-safe top-1 baseline is 0%.

The result does not prove that engine telemetry is useless. It shows that the current one-task anchor-transition evidence cannot support either a safe simulator-residual tuner or a simulator-free telemetry tuner. A larger model or an R1 run would add capacity/data after a failed gate and is therefore not authorized under this roadmap.

R1 — New development surface

Status: NOT LAUNCHED; CLOSED BY R0.

Frozen starting setup:

• host: dash0, eight NVIDIA H20 GPUs;
• cells run solo; no co-location for SLO verdicts;
• patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16;
• trace: chat_w20260313_1000;
• output tokens: exactly 128;
• SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, pass rate at least 0.95;
• config surface: TP {1,2,4} × MNS {8,16,32,64};
• hard campaign cap: 4 H20-hours.

The load ladder, repetitions, randomized order, exact commands, expected wall time, and artifact paths are frozen only after R0. A resolved echo is required before launch.

R1 passes only if a frozen sequential replay shows at least 15% E2E H20-hour headroom over the strongest safe baseline with final regret at most 5%. R1 is development evidence and cannot be reported as the held-out result.

R2 — Held-out sequential tuning

Status: NOT LAUNCHED; CLOSED BY R0.

Required baselines:

1. random search;
2. real-outcome-only Bayesian/sequential search;
3. Frontier ranking plus real top-k final;
4. simulator plus real-outcome residual;
5. telemetry-only transition tuner;
6. simulator plus telemetry residual tuner;
7. complete real surface as oracle, not as a cost competitor.

Primary metric: end-to-end H20-hours to first reach 95% of the real full-surface SLO-goodput oracle. Secondary metrics are cost-normalized regret AUC, final regret at fixed budgets, oracle false-prune, wall time, and per-task regressions.

The route is successful only if the winning telemetry method reduces the primary cost by at least 20% versus the strongest safe baseline and ends within 5% regret on every headline task. If hybrid beats telemetry-only by at least 10%, simulator residual correction is the primary method. If telemetry-only is within 5% or better, the simulator dependency is removed. If neither clears the contribution bar, the route is closed and telemetry remains a diagnostic facility only.

Cost discipline

• R0 simulator work is CPU-only and must set empty CUDA visibility.
• R1 cannot exceed 4 H20-hours.
• R2 receives no budget until R1 passes.
• Startup, warm-up, burn-in, failed launches, real probes, continuation, and final validation are charged. Benchmark-only annotation repeats are reported separately and cannot disappear from campaign accounting.

Final R0 sanity block

Data	n	Min	Max	Distinct	Checked invariant
Phase 6 cells	12	TP1/MNS8	TP4/MNS64	12	Surface not identical; solo SLO tier authoritative
Phase 6 Layer-1 primary steps	37 streams	343	12,103	37	Contiguous; zero drops
P1 primary anchors	12	infeasible	feasible	2 labels	7 feasible / 5 infeasible
P1 Frontier runtime	12	24.093 s	54.575 s	12	CPU-only; zero failures
Detailed Frontier replay runtime	12	23.943 s	54.786 s	12	Exact committed scorers; CUDA hidden
Detailed artifact bytes	12	4,123,724	13,527,776	12	Non-negative; practical CPU replay size
Cross-config transitions	120	real pass 0.1067	real pass 1.0	6 outcomes	Both endpoint config identities held out
State residual vectors	12	16 fields	16 fields	12 vectors	Finite; no missing common field
R0 E2E cost values	4	0.237914	0.301935 H20-h	4	Non-negative; k=1/2, online/conservative

Checked invariants: non-negative counts and costs; pass rates in [0,1]; simulator results not all identical; exact request count/hash agreement; Layer-1 step continuity and zero drops; no co-resident SLO measurements; no calibration or evaluation split reuse for a future headline claim. No current red flag invalidates R0 plumbing. The R0 tuning gate itself failed because safe prior-preserving correction was absent.