aituner/docs/telemetry-residual-tuning-roadmap-20260714.md

# Telemetry-conditioned residual tuning roadmap

Status: **R0 COMPLETE / FAILED; R1 AND R2 CLOSED FOR THIS MODEL**.

Date: 2026-07-14 (Asia/Singapore).

## Research question and claim boundary

The question is whether a small number of real engine observations can correct
a simulator's task-specific error over **unmeasured configurations**, and
whether that correction reduces the real-GPU cost of finding a high
SLO-goodput serving configuration.

The intended headline claim, if the evidence supports it, is:

> An engine-state-conditioned residual model turns a simulator prediction into
> a task-specific posterior over unmeasured serving configurations, allowing a
> sequential tuner to reach near-oracle SLO-goodput with materially fewer
> H20-hours than simulator-only and outcome-only tuning.

Classification accuracy, simulator-error diagnosis, and telemetry overhead are
supporting evidence.  None is an end-to-end tuning contribution by itself.

The following method is closed and will not be revived under another name:
per-candidate five-second accept/reject as the headline contribution.  The P1
result showed only 1.426% cost reduction in the frozen `k=2` workflow.

## Two models, one evaluation

Both branches use the same legal candidate set, real measurements, task split,
cost accounting, and acquisition function.

### Simulator-residual branch (primary)

For measured anchor `c_t` and unmeasured candidate `c'`:

```text
y_hat(c') = y_real(c_t)
          + [y_sim(c') - y_sim(c_t)]
          + f(state_real(c_t) - state_sim(c_t), c' - c_t, workload, SLO)
```

The simulator delta is the prior.  The learned model may correct it only with
training-supported state/config transitions; uncertainty or distribution shift
must shrink the correction back toward the simulator prior.

### Telemetry-only branch (mandatory)

```text
y_hat(c') = y_real(c_t)
          + g(state_real(c_t), c' - c_t, workload, SLO)
```

This branch tests whether the simulator is actually necessary.  It does not
use a hand-authored bottleneck-to-knob rule.

### Search policy

Legal configurations are enumerated independently of telemetry.  A generic
cost-aware acquisition rule ranks candidates from predicted improvement,
uncertainty, and measured H20 cost.  The current production harness's
bottleneck scores, topology-first ordering, and hand-set relief constants are
not consumed by either branch.  The validator may enforce legality,
full-config no-repeat, failure accounting, and resource caps only.

## Hypotheses

| ID | Hypothesis | Direct test | Failure meaning |
|---|---|---|---|
| H0 | Existing artifacts can express a common, direct-measurement state without heuristic labels. | Engine/simulator extractor coverage and invariants. | Route is not currently implementable. |
| H1 | Simulator errors are predictable from engine/simulator state discrepancy at measured anchors. | Task-held-out pairwise inversion correction and new-inversion rate. | Telemetry is diagnostic but cannot correct the surface. |
| H2 | Telemetry alone predicts useful config transitions beyond outcome-only history. | Telemetry-only versus real-outcome-only sequential replay. | Direct telemetry-guided tuning has no independent value. |
| H3 | Residual correction changes actual tuning decisions and cost. | H20-hours to 95% oracle and regret AUC against the strongest safe baseline. | No system contribution even if H1/H2 prediction metrics improve. |

## Common-state contract

Only directly observed or exactly reconstructed quantities are admitted.

| Quantity | vLLM Layer-1 | Frontier | R0 status |
|---|---|---|---|
| Scheduled requests / batch size | Per scheduler step | Existing per-batch metric, disabled in P1 output | Common after CPU replay |
| Scheduled prefill/decode tokens | Per scheduler step | Existing per-batch metrics | Common after CPU replay |
| Scheduler/batch rate | Monotonic step timestamps | Batch count / simulated duration | Common after CPU replay |
| Waiting queue area | Time-weighted queue gauge | Sum of request waiting times | Common aggregate |
| Running request area | Time-weighted running gauge | Sum of E2E minus waiting time | Common aggregate, semantics audited |
| Preemption count | Per step | Per request | Common |
| KV usage/headroom | Exact blocks and ratio | Not in committed output | Engine-only until exact reconstruction exists |
| CUDA graph mode/padding | Exact per step | Not modeled | Engine-only omitted-mechanism signal |
| Request TTFT/TPOT/pass rate | Exact real outcomes | Exact simulated request metrics | Common outcome, not state |

Unavailable fields remain null.  They cannot be imputed from a human
`prefill/decode/queueing` label.

Frontier already contains the required detailed batch and timestamped
stage-batch ledger output.  P1 disabled it for artifact size.  R0 replays the
same immutable fixtures with the existing output flags enabled; it does not
change the simulator model or calibration.

## Data separation

- Phase 6 / `chat_w20260311_1000`: development only.
- P1 / `chat_w20260312_1000`: development only.
- R1 / `chat_w20260313_1000`: new development surface.
- R2: trace windows not used for feature, model, threshold, candidate-space,
  cutoff, or acquisition decisions.
- Splits are by complete workload/SLO task.  Anchor- or pair-level random
  splits are prohibited.
- Sequential-policy seeds measure algorithmic variability; they are not
  counted as independent system tasks.

The two existing development tasks have an important limitation: the now-
available SLO-gated simulator reading already retains the real oracle at its
top rank/tie.  They therefore cannot establish a positive end-to-end ranking
claim.  They are used for plumbing, known false-feasible cases, and negative
evidence.  R1 must be run as an unbiased complete surface, not selected after
observing simulator success or failure.

## Step-by-step roadmap

### R0.1 — Inventory and roadmap

Deliverables:

- this roadmap;
- rolling untracked `ONGOING.md`;
- exact engine/simulator field and artifact inventory.

Gate: every claimed input has an authoritative file path and provenance.

### R0.2 — Common-state plumbing

Deliverables:

- `runs/telemetry-residual/common_state.py`;
- synthetic correctness tests;
- one exact P1 Frontier replay with individual batch metrics and the full
  stage-batch ledger enabled;
- paired engine/simulator state summary for the same fixture.

Gate:

- replay request count and SLO scorer exactly agree with the committed replay;
- batch/ledger outputs are non-empty;
- all counters are non-negative, ratios bounded, times monotonic;
- no GPU is visible to Frontier;
- output volume is practical before expanding to twelve replays.

### R0.3 — Development residual/headroom audit

Use all frozen P1 primary fixtures and corresponding engine intervals.  Produce:

- common-state residuals per anchor;
- simulator-error labels and continuous SLO/goodput residuals;
- ordered source/target diagnostic that removes both config identities from
  both roles in every training fold;
- oracle upper bound for cross-candidate correction;
- explicit comparison with simulator+outcome and telemetry-only features.

R0 is a feasibility gate, not headline evidence.  Proceed to R1 only if:

1. state features are collected with the measured source anchor, vary across
   cells, and are available before any target config is evaluated;
2. at least one known simulator error has a state discrepancy not exposed by
   the matched external prefix outcome;
3. a prior-preserving model can correct development errors without introducing
   a larger number of new errors under regularization sensitivity;
4. an oracle cross-candidate correction has at least 15% sequential tuning-cost
   headroom under full startup/warm-up accounting.

### R0 result and decision

R0 completed without a data-validity red flag, but failed condition 3.  The
decision is **STOP_BEFORE_R1**; no H20 job was launched for this route.

- All 12 detailed Frontier CPU replays exactly reproduced their committed SLO
  scorers.  Runtime was 23.943--54.786 seconds per replay, detailed artifacts
  were 4.12--13.53 MB, CUDA visibility was empty, and there were zero failures.
- The paired surface contains 12 real/sim anchors, two known simulator
  false-feasible anchors, and 120 legal cross-config ordered transitions.  A
  fold removes both the source and target TP/MNS identity from source and
  target roles; the two offered-load anchors remain part of the same task.
- Raw Frontier feasibility is 83.33% on the repeated transition view.  The
  structurally correct hybrid model uses
  `r_target = r_source + delta_r`; the direct model uses
  `y_target = y_source + delta_y` and never reads simulator fields.
- Direct telemetry is not robust relative to real-outcome-only: its accuracy
  delta over L2 `{0.1,1,10,100}` is `{-0.83,+1.67,0,-4.17}` percentage points,
  and its best absolute accuracy is 54.17%, below the raw simulator's 83.33%.
- Hybrid telemetry raises classification accuracy over the corresponding
  simulator+outcome transition regression by 1.67--4.17 percentage points,
  but worsens pass-rate RMSE by 0.141--0.201 and MAE by 0.084--0.125.  Its full
  correction reaches only 46.67--53.33% absolute accuracy.
- Across 24 nonzero `(L2, raw-simulator-prior weight)` combinations, no model
  both corrects an existing simulator error without more new errors and avoids
  worsening RMSE/MAE.  Whenever a correction fixes at least one error, it
  corrupts at least 11 previously correct transitions.
- A perfect correction could skip the frozen simulator rank-2 real final and
  save 0.043469 H20-hours: 15.45% of the prospective online `k=2` cost, or
  14.40% when the prior failed launch is charged.  On this development task the
  simulator top-1 already is the real oracle with zero regret, so headroom
  versus the observed-safe top-1 baseline is 0%.

The result does not prove that engine telemetry is useless.  It shows that the
current one-task anchor-transition evidence cannot support either a safe
simulator-residual tuner or a simulator-free telemetry tuner.  A larger model
or an R1 run would add capacity/data after a failed gate and is therefore not
authorized under this roadmap.

### R1 — New development surface

Status: **NOT LAUNCHED; CLOSED BY R0**.

Frozen starting setup:

- host: dash0, eight NVIDIA H20 GPUs;
- cells run solo; no co-location for SLO verdicts;
- patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16;
- trace: `chat_w20260313_1000`;
- output tokens: exactly 128;
- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, pass rate at least 0.95;
- config surface: TP `{1,2,4}` × MNS `{8,16,32,64}`;
- hard campaign cap: 4 H20-hours.

The load ladder, repetitions, randomized order, exact commands, expected wall
time, and artifact paths are frozen only after R0.  A resolved echo is required
before launch.

R1 passes only if a frozen sequential replay shows at least 15% E2E H20-hour
headroom over the strongest safe baseline with final regret at most 5%.  R1 is
development evidence and cannot be reported as the held-out result.

### R2 — Held-out sequential tuning

Status: **NOT LAUNCHED; CLOSED BY R0**.

Required baselines:

1. random search;
2. real-outcome-only Bayesian/sequential search;
3. Frontier ranking plus real top-k final;
4. simulator plus real-outcome residual;
5. telemetry-only transition tuner;
6. simulator plus telemetry residual tuner;
7. complete real surface as oracle, not as a cost competitor.

Primary metric: end-to-end H20-hours to first reach 95% of the real full-surface
SLO-goodput oracle.  Secondary metrics are cost-normalized regret AUC, final
regret at fixed budgets, oracle false-prune, wall time, and per-task regressions.

The route is successful only if the winning telemetry method reduces the
primary cost by at least 20% versus the strongest safe baseline and ends within
5% regret on every headline task.  If hybrid beats telemetry-only by at least
10%, simulator residual correction is the primary method.  If telemetry-only
is within 5% or better, the simulator dependency is removed.  If neither clears
the contribution bar, the route is closed and telemetry remains a diagnostic
facility only.

## Cost discipline

- R0 simulator work is CPU-only and must set empty CUDA visibility.
- R1 cannot exceed 4 H20-hours.
- R2 receives no budget until R1 passes.
- Startup, warm-up, burn-in, failed launches, real probes, continuation, and
  final validation are charged.  Benchmark-only annotation repeats are
  reported separately and cannot disappear from campaign accounting.

## Final R0 sanity block

| Data | n | Min | Max | Distinct | Checked invariant |
|---|---:|---:|---:|---:|---|
| Phase 6 cells | 12 | TP1/MNS8 | TP4/MNS64 | 12 | Surface not identical; solo SLO tier authoritative |
| Phase 6 Layer-1 primary steps | 37 streams | 343 | 12,103 | 37 | Contiguous; zero drops |
| P1 primary anchors | 12 | infeasible | feasible | 2 labels | 7 feasible / 5 infeasible |
| P1 Frontier runtime | 12 | 24.093 s | 54.575 s | 12 | CPU-only; zero failures |
| Detailed Frontier replay runtime | 12 | 23.943 s | 54.786 s | 12 | Exact committed scorers; CUDA hidden |
| Detailed artifact bytes | 12 | 4,123,724 | 13,527,776 | 12 | Non-negative; practical CPU replay size |
| Cross-config transitions | 120 | real pass 0.1067 | real pass 1.0 | 6 outcomes | Both endpoint config identities held out |
| State residual vectors | 12 | 16 fields | 16 fields | 12 vectors | Finite; no missing common field |
| R0 E2E cost values | 4 | 0.237914 | 0.301935 H20-h | 4 | Non-negative; `k=1/2`, online/conservative |

Checked invariants: non-negative counts and costs; pass rates in `[0,1]`;
simulator results not all identical; exact request count/hash agreement; Layer-1
step continuity and zero drops; no co-resident SLO measurements; no calibration
or evaluation split reuse for a future headline claim.  No current red flag
invalidates R0 plumbing.  The R0 tuning gate itself failed because safe
prior-preserving correction was absent.