# Telemetry-conditioned residual tuning roadmap Status: **R0 COMPLETE / FAILED; R1 AND R2 CLOSED FOR THIS MODEL**. Date: 2026-07-14 (Asia/Singapore). ## Research question and claim boundary The question is whether a small number of real engine observations can correct a simulator's task-specific error over **unmeasured configurations**, and whether that correction reduces the real-GPU cost of finding a high SLO-goodput serving configuration. The intended headline claim, if the evidence supports it, is: > An engine-state-conditioned residual model turns a simulator prediction into > a task-specific posterior over unmeasured serving configurations, allowing a > sequential tuner to reach near-oracle SLO-goodput with materially fewer > H20-hours than simulator-only and outcome-only tuning. Classification accuracy, simulator-error diagnosis, and telemetry overhead are supporting evidence. None is an end-to-end tuning contribution by itself. The following method is closed and will not be revived under another name: per-candidate five-second accept/reject as the headline contribution. The P1 result showed only 1.426% cost reduction in the frozen `k=2` workflow. ## Two models, one evaluation Both branches use the same legal candidate set, real measurements, task split, cost accounting, and acquisition function. ### Simulator-residual branch (primary) For measured anchor `c_t` and unmeasured candidate `c'`: ```text y_hat(c') = y_real(c_t) + [y_sim(c') - y_sim(c_t)] + f(state_real(c_t) - state_sim(c_t), c' - c_t, workload, SLO) ``` The simulator delta is the prior. The learned model may correct it only with training-supported state/config transitions; uncertainty or distribution shift must shrink the correction back toward the simulator prior. ### Telemetry-only branch (mandatory) ```text y_hat(c') = y_real(c_t) + g(state_real(c_t), c' - c_t, workload, SLO) ``` This branch tests whether the simulator is actually necessary. It does not use a hand-authored bottleneck-to-knob rule. ### Search policy Legal configurations are enumerated independently of telemetry. A generic cost-aware acquisition rule ranks candidates from predicted improvement, uncertainty, and measured H20 cost. The current production harness's bottleneck scores, topology-first ordering, and hand-set relief constants are not consumed by either branch. The validator may enforce legality, full-config no-repeat, failure accounting, and resource caps only. ## Hypotheses | ID | Hypothesis | Direct test | Failure meaning | |---|---|---|---| | H0 | Existing artifacts can express a common, direct-measurement state without heuristic labels. | Engine/simulator extractor coverage and invariants. | Route is not currently implementable. | | H1 | Simulator errors are predictable from engine/simulator state discrepancy at measured anchors. | Task-held-out pairwise inversion correction and new-inversion rate. | Telemetry is diagnostic but cannot correct the surface. | | H2 | Telemetry alone predicts useful config transitions beyond outcome-only history. | Telemetry-only versus real-outcome-only sequential replay. | Direct telemetry-guided tuning has no independent value. | | H3 | Residual correction changes actual tuning decisions and cost. | H20-hours to 95% oracle and regret AUC against the strongest safe baseline. | No system contribution even if H1/H2 prediction metrics improve. | ## Common-state contract Only directly observed or exactly reconstructed quantities are admitted. | Quantity | vLLM Layer-1 | Frontier | R0 status | |---|---|---|---| | Scheduled requests / batch size | Per scheduler step | Existing per-batch metric, disabled in P1 output | Common after CPU replay | | Scheduled prefill/decode tokens | Per scheduler step | Existing per-batch metrics | Common after CPU replay | | Scheduler/batch rate | Monotonic step timestamps | Batch count / simulated duration | Common after CPU replay | | Waiting queue area | Time-weighted queue gauge | Sum of request waiting times | Common aggregate | | Running request area | Time-weighted running gauge | Sum of E2E minus waiting time | Common aggregate, semantics audited | | Preemption count | Per step | Per request | Common | | KV usage/headroom | Exact blocks and ratio | Not in committed output | Engine-only until exact reconstruction exists | | CUDA graph mode/padding | Exact per step | Not modeled | Engine-only omitted-mechanism signal | | Request TTFT/TPOT/pass rate | Exact real outcomes | Exact simulated request metrics | Common outcome, not state | Unavailable fields remain null. They cannot be imputed from a human `prefill/decode/queueing` label. Frontier already contains the required detailed batch and timestamped stage-batch ledger output. P1 disabled it for artifact size. R0 replays the same immutable fixtures with the existing output flags enabled; it does not change the simulator model or calibration. ## Data separation - Phase 6 / `chat_w20260311_1000`: development only. - P1 / `chat_w20260312_1000`: development only. - R1 / `chat_w20260313_1000`: new development surface. - R2: trace windows not used for feature, model, threshold, candidate-space, cutoff, or acquisition decisions. - Splits are by complete workload/SLO task. Anchor- or pair-level random splits are prohibited. - Sequential-policy seeds measure algorithmic variability; they are not counted as independent system tasks. The two existing development tasks have an important limitation: the now- available SLO-gated simulator reading already retains the real oracle at its top rank/tie. They therefore cannot establish a positive end-to-end ranking claim. They are used for plumbing, known false-feasible cases, and negative evidence. R1 must be run as an unbiased complete surface, not selected after observing simulator success or failure. ## Step-by-step roadmap ### R0.1 — Inventory and roadmap Deliverables: - this roadmap; - rolling untracked `ONGOING.md`; - exact engine/simulator field and artifact inventory. Gate: every claimed input has an authoritative file path and provenance. ### R0.2 — Common-state plumbing Deliverables: - `runs/telemetry-residual/common_state.py`; - synthetic correctness tests; - one exact P1 Frontier replay with individual batch metrics and the full stage-batch ledger enabled; - paired engine/simulator state summary for the same fixture. Gate: - replay request count and SLO scorer exactly agree with the committed replay; - batch/ledger outputs are non-empty; - all counters are non-negative, ratios bounded, times monotonic; - no GPU is visible to Frontier; - output volume is practical before expanding to twelve replays. ### R0.3 — Development residual/headroom audit Use all frozen P1 primary fixtures and corresponding engine intervals. Produce: - common-state residuals per anchor; - simulator-error labels and continuous SLO/goodput residuals; - ordered source/target diagnostic that removes both config identities from both roles in every training fold; - oracle upper bound for cross-candidate correction; - explicit comparison with simulator+outcome and telemetry-only features. R0 is a feasibility gate, not headline evidence. Proceed to R1 only if: 1. state features are collected with the measured source anchor, vary across cells, and are available before any target config is evaluated; 2. at least one known simulator error has a state discrepancy not exposed by the matched external prefix outcome; 3. a prior-preserving model can correct development errors without introducing a larger number of new errors under regularization sensitivity; 4. an oracle cross-candidate correction has at least 15% sequential tuning-cost headroom under full startup/warm-up accounting. ### R0 result and decision R0 completed without a data-validity red flag, but failed condition 3. The decision is **STOP_BEFORE_R1**; no H20 job was launched for this route. - All 12 detailed Frontier CPU replays exactly reproduced their committed SLO scorers. Runtime was 23.943--54.786 seconds per replay, detailed artifacts were 4.12--13.53 MB, CUDA visibility was empty, and there were zero failures. - The paired surface contains 12 real/sim anchors, two known simulator false-feasible anchors, and 120 legal cross-config ordered transitions. A fold removes both the source and target TP/MNS identity from source and target roles; the two offered-load anchors remain part of the same task. - Raw Frontier feasibility is 83.33% on the repeated transition view. The structurally correct hybrid model uses `r_target = r_source + delta_r`; the direct model uses `y_target = y_source + delta_y` and never reads simulator fields. - Direct telemetry is not robust relative to real-outcome-only: its accuracy delta over L2 `{0.1,1,10,100}` is `{-0.83,+1.67,0,-4.17}` percentage points, and its best absolute accuracy is 54.17%, below the raw simulator's 83.33%. - Hybrid telemetry raises classification accuracy over the corresponding simulator+outcome transition regression by 1.67--4.17 percentage points, but worsens pass-rate RMSE by 0.141--0.201 and MAE by 0.084--0.125. Its full correction reaches only 46.67--53.33% absolute accuracy. - Across 24 nonzero `(L2, raw-simulator-prior weight)` combinations, no model both corrects an existing simulator error without more new errors and avoids worsening RMSE/MAE. Whenever a correction fixes at least one error, it corrupts at least 11 previously correct transitions. - A perfect correction could skip the frozen simulator rank-2 real final and save 0.043469 H20-hours: 15.45% of the prospective online `k=2` cost, or 14.40% when the prior failed launch is charged. On this development task the simulator top-1 already is the real oracle with zero regret, so headroom versus the observed-safe top-1 baseline is 0%. The result does not prove that engine telemetry is useless. It shows that the current one-task anchor-transition evidence cannot support either a safe simulator-residual tuner or a simulator-free telemetry tuner. A larger model or an R1 run would add capacity/data after a failed gate and is therefore not authorized under this roadmap. ### R1 — New development surface Status: **NOT LAUNCHED; CLOSED BY R0**. Frozen starting setup: - host: dash0, eight NVIDIA H20 GPUs; - cells run solo; no co-location for SLO verdicts; - patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16; - trace: `chat_w20260313_1000`; - output tokens: exactly 128; - SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, pass rate at least 0.95; - config surface: TP `{1,2,4}` × MNS `{8,16,32,64}`; - hard campaign cap: 4 H20-hours. The load ladder, repetitions, randomized order, exact commands, expected wall time, and artifact paths are frozen only after R0. A resolved echo is required before launch. R1 passes only if a frozen sequential replay shows at least 15% E2E H20-hour headroom over the strongest safe baseline with final regret at most 5%. R1 is development evidence and cannot be reported as the held-out result. ### R2 — Held-out sequential tuning Status: **NOT LAUNCHED; CLOSED BY R0**. Required baselines: 1. random search; 2. real-outcome-only Bayesian/sequential search; 3. Frontier ranking plus real top-k final; 4. simulator plus real-outcome residual; 5. telemetry-only transition tuner; 6. simulator plus telemetry residual tuner; 7. complete real surface as oracle, not as a cost competitor. Primary metric: end-to-end H20-hours to first reach 95% of the real full-surface SLO-goodput oracle. Secondary metrics are cost-normalized regret AUC, final regret at fixed budgets, oracle false-prune, wall time, and per-task regressions. The route is successful only if the winning telemetry method reduces the primary cost by at least 20% versus the strongest safe baseline and ends within 5% regret on every headline task. If hybrid beats telemetry-only by at least 10%, simulator residual correction is the primary method. If telemetry-only is within 5% or better, the simulator dependency is removed. If neither clears the contribution bar, the route is closed and telemetry remains a diagnostic facility only. ## Cost discipline - R0 simulator work is CPU-only and must set empty CUDA visibility. - R1 cannot exceed 4 H20-hours. - R2 receives no budget until R1 passes. - Startup, warm-up, burn-in, failed launches, real probes, continuation, and final validation are charged. Benchmark-only annotation repeats are reported separately and cannot disappear from campaign accounting. ## Final R0 sanity block | Data | n | Min | Max | Distinct | Checked invariant | |---|---:|---:|---:|---:|---| | Phase 6 cells | 12 | TP1/MNS8 | TP4/MNS64 | 12 | Surface not identical; solo SLO tier authoritative | | Phase 6 Layer-1 primary steps | 37 streams | 343 | 12,103 | 37 | Contiguous; zero drops | | P1 primary anchors | 12 | infeasible | feasible | 2 labels | 7 feasible / 5 infeasible | | P1 Frontier runtime | 12 | 24.093 s | 54.575 s | 12 | CPU-only; zero failures | | Detailed Frontier replay runtime | 12 | 23.943 s | 54.786 s | 12 | Exact committed scorers; CUDA hidden | | Detailed artifact bytes | 12 | 4,123,724 | 13,527,776 | 12 | Non-negative; practical CPU replay size | | Cross-config transitions | 120 | real pass 0.1067 | real pass 1.0 | 6 outcomes | Both endpoint config identities held out | | State residual vectors | 12 | 16 fields | 16 fields | 12 vectors | Finite; no missing common field | | R0 E2E cost values | 4 | 0.237914 | 0.301935 H20-h | 4 | Non-negative; `k=1/2`, online/conservative | Checked invariants: non-negative counts and costs; pass rates in `[0,1]`; simulator results not all identical; exact request count/hash agreement; Layer-1 step continuity and zero drops; no co-resident SLO measurements; no calibration or evaluation split reuse for a future headline claim. No current red flag invalidates R0 plumbing. The R0 tuning gate itself failed because safe prior-preserving correction was absent.