Audit telemetry residual tuning premise
This commit is contained in:
286
docs/telemetry-residual-tuning-roadmap-20260714.md
Normal file
286
docs/telemetry-residual-tuning-roadmap-20260714.md
Normal file
@@ -0,0 +1,286 @@
|
||||
# Telemetry-conditioned residual tuning roadmap
|
||||
|
||||
Status: **R0 COMPLETE / FAILED; R1 AND R2 CLOSED FOR THIS MODEL**.
|
||||
|
||||
Date: 2026-07-14 (Asia/Singapore).
|
||||
|
||||
## Research question and claim boundary
|
||||
|
||||
The question is whether a small number of real engine observations can correct
|
||||
a simulator's task-specific error over **unmeasured configurations**, and
|
||||
whether that correction reduces the real-GPU cost of finding a high
|
||||
SLO-goodput serving configuration.
|
||||
|
||||
The intended headline claim, if the evidence supports it, is:
|
||||
|
||||
> An engine-state-conditioned residual model turns a simulator prediction into
|
||||
> a task-specific posterior over unmeasured serving configurations, allowing a
|
||||
> sequential tuner to reach near-oracle SLO-goodput with materially fewer
|
||||
> H20-hours than simulator-only and outcome-only tuning.
|
||||
|
||||
Classification accuracy, simulator-error diagnosis, and telemetry overhead are
|
||||
supporting evidence. None is an end-to-end tuning contribution by itself.
|
||||
|
||||
The following method is closed and will not be revived under another name:
|
||||
per-candidate five-second accept/reject as the headline contribution. The P1
|
||||
result showed only 1.426% cost reduction in the frozen `k=2` workflow.
|
||||
|
||||
## Two models, one evaluation
|
||||
|
||||
Both branches use the same legal candidate set, real measurements, task split,
|
||||
cost accounting, and acquisition function.
|
||||
|
||||
### Simulator-residual branch (primary)
|
||||
|
||||
For measured anchor `c_t` and unmeasured candidate `c'`:
|
||||
|
||||
```text
|
||||
y_hat(c') = y_real(c_t)
|
||||
+ [y_sim(c') - y_sim(c_t)]
|
||||
+ f(state_real(c_t) - state_sim(c_t), c' - c_t, workload, SLO)
|
||||
```
|
||||
|
||||
The simulator delta is the prior. The learned model may correct it only with
|
||||
training-supported state/config transitions; uncertainty or distribution shift
|
||||
must shrink the correction back toward the simulator prior.
|
||||
|
||||
### Telemetry-only branch (mandatory)
|
||||
|
||||
```text
|
||||
y_hat(c') = y_real(c_t)
|
||||
+ g(state_real(c_t), c' - c_t, workload, SLO)
|
||||
```
|
||||
|
||||
This branch tests whether the simulator is actually necessary. It does not
|
||||
use a hand-authored bottleneck-to-knob rule.
|
||||
|
||||
### Search policy
|
||||
|
||||
Legal configurations are enumerated independently of telemetry. A generic
|
||||
cost-aware acquisition rule ranks candidates from predicted improvement,
|
||||
uncertainty, and measured H20 cost. The current production harness's
|
||||
bottleneck scores, topology-first ordering, and hand-set relief constants are
|
||||
not consumed by either branch. The validator may enforce legality,
|
||||
full-config no-repeat, failure accounting, and resource caps only.
|
||||
|
||||
## Hypotheses
|
||||
|
||||
| ID | Hypothesis | Direct test | Failure meaning |
|
||||
|---|---|---|---|
|
||||
| H0 | Existing artifacts can express a common, direct-measurement state without heuristic labels. | Engine/simulator extractor coverage and invariants. | Route is not currently implementable. |
|
||||
| H1 | Simulator errors are predictable from engine/simulator state discrepancy at measured anchors. | Task-held-out pairwise inversion correction and new-inversion rate. | Telemetry is diagnostic but cannot correct the surface. |
|
||||
| H2 | Telemetry alone predicts useful config transitions beyond outcome-only history. | Telemetry-only versus real-outcome-only sequential replay. | Direct telemetry-guided tuning has no independent value. |
|
||||
| H3 | Residual correction changes actual tuning decisions and cost. | H20-hours to 95% oracle and regret AUC against the strongest safe baseline. | No system contribution even if H1/H2 prediction metrics improve. |
|
||||
|
||||
## Common-state contract
|
||||
|
||||
Only directly observed or exactly reconstructed quantities are admitted.
|
||||
|
||||
| Quantity | vLLM Layer-1 | Frontier | R0 status |
|
||||
|---|---|---|---|
|
||||
| Scheduled requests / batch size | Per scheduler step | Existing per-batch metric, disabled in P1 output | Common after CPU replay |
|
||||
| Scheduled prefill/decode tokens | Per scheduler step | Existing per-batch metrics | Common after CPU replay |
|
||||
| Scheduler/batch rate | Monotonic step timestamps | Batch count / simulated duration | Common after CPU replay |
|
||||
| Waiting queue area | Time-weighted queue gauge | Sum of request waiting times | Common aggregate |
|
||||
| Running request area | Time-weighted running gauge | Sum of E2E minus waiting time | Common aggregate, semantics audited |
|
||||
| Preemption count | Per step | Per request | Common |
|
||||
| KV usage/headroom | Exact blocks and ratio | Not in committed output | Engine-only until exact reconstruction exists |
|
||||
| CUDA graph mode/padding | Exact per step | Not modeled | Engine-only omitted-mechanism signal |
|
||||
| Request TTFT/TPOT/pass rate | Exact real outcomes | Exact simulated request metrics | Common outcome, not state |
|
||||
|
||||
Unavailable fields remain null. They cannot be imputed from a human
|
||||
`prefill/decode/queueing` label.
|
||||
|
||||
Frontier already contains the required detailed batch and timestamped
|
||||
stage-batch ledger output. P1 disabled it for artifact size. R0 replays the
|
||||
same immutable fixtures with the existing output flags enabled; it does not
|
||||
change the simulator model or calibration.
|
||||
|
||||
## Data separation
|
||||
|
||||
- Phase 6 / `chat_w20260311_1000`: development only.
|
||||
- P1 / `chat_w20260312_1000`: development only.
|
||||
- R1 / `chat_w20260313_1000`: new development surface.
|
||||
- R2: trace windows not used for feature, model, threshold, candidate-space,
|
||||
cutoff, or acquisition decisions.
|
||||
- Splits are by complete workload/SLO task. Anchor- or pair-level random
|
||||
splits are prohibited.
|
||||
- Sequential-policy seeds measure algorithmic variability; they are not
|
||||
counted as independent system tasks.
|
||||
|
||||
The two existing development tasks have an important limitation: the now-
|
||||
available SLO-gated simulator reading already retains the real oracle at its
|
||||
top rank/tie. They therefore cannot establish a positive end-to-end ranking
|
||||
claim. They are used for plumbing, known false-feasible cases, and negative
|
||||
evidence. R1 must be run as an unbiased complete surface, not selected after
|
||||
observing simulator success or failure.
|
||||
|
||||
## Step-by-step roadmap
|
||||
|
||||
### R0.1 — Inventory and roadmap
|
||||
|
||||
Deliverables:
|
||||
|
||||
- this roadmap;
|
||||
- rolling untracked `ONGOING.md`;
|
||||
- exact engine/simulator field and artifact inventory.
|
||||
|
||||
Gate: every claimed input has an authoritative file path and provenance.
|
||||
|
||||
### R0.2 — Common-state plumbing
|
||||
|
||||
Deliverables:
|
||||
|
||||
- `runs/telemetry-residual/common_state.py`;
|
||||
- synthetic correctness tests;
|
||||
- one exact P1 Frontier replay with individual batch metrics and the full
|
||||
stage-batch ledger enabled;
|
||||
- paired engine/simulator state summary for the same fixture.
|
||||
|
||||
Gate:
|
||||
|
||||
- replay request count and SLO scorer exactly agree with the committed replay;
|
||||
- batch/ledger outputs are non-empty;
|
||||
- all counters are non-negative, ratios bounded, times monotonic;
|
||||
- no GPU is visible to Frontier;
|
||||
- output volume is practical before expanding to twelve replays.
|
||||
|
||||
### R0.3 — Development residual/headroom audit
|
||||
|
||||
Use all frozen P1 primary fixtures and corresponding engine intervals. Produce:
|
||||
|
||||
- common-state residuals per anchor;
|
||||
- simulator-error labels and continuous SLO/goodput residuals;
|
||||
- ordered source/target diagnostic that removes both config identities from
|
||||
both roles in every training fold;
|
||||
- oracle upper bound for cross-candidate correction;
|
||||
- explicit comparison with simulator+outcome and telemetry-only features.
|
||||
|
||||
R0 is a feasibility gate, not headline evidence. Proceed to R1 only if:
|
||||
|
||||
1. state features are collected with the measured source anchor, vary across
|
||||
cells, and are available before any target config is evaluated;
|
||||
2. at least one known simulator error has a state discrepancy not exposed by
|
||||
the matched external prefix outcome;
|
||||
3. a prior-preserving model can correct development errors without introducing
|
||||
a larger number of new errors under regularization sensitivity;
|
||||
4. an oracle cross-candidate correction has at least 15% sequential tuning-cost
|
||||
headroom under full startup/warm-up accounting.
|
||||
|
||||
### R0 result and decision
|
||||
|
||||
R0 completed without a data-validity red flag, but failed condition 3. The
|
||||
decision is **STOP_BEFORE_R1**; no H20 job was launched for this route.
|
||||
|
||||
- All 12 detailed Frontier CPU replays exactly reproduced their committed SLO
|
||||
scorers. Runtime was 23.943--54.786 seconds per replay, detailed artifacts
|
||||
were 4.12--13.53 MB, CUDA visibility was empty, and there were zero failures.
|
||||
- The paired surface contains 12 real/sim anchors, two known simulator
|
||||
false-feasible anchors, and 120 legal cross-config ordered transitions. A
|
||||
fold removes both the source and target TP/MNS identity from source and
|
||||
target roles; the two offered-load anchors remain part of the same task.
|
||||
- Raw Frontier feasibility is 83.33% on the repeated transition view. The
|
||||
structurally correct hybrid model uses
|
||||
`r_target = r_source + delta_r`; the direct model uses
|
||||
`y_target = y_source + delta_y` and never reads simulator fields.
|
||||
- Direct telemetry is not robust relative to real-outcome-only: its accuracy
|
||||
delta over L2 `{0.1,1,10,100}` is `{-0.83,+1.67,0,-4.17}` percentage points,
|
||||
and its best absolute accuracy is 54.17%, below the raw simulator's 83.33%.
|
||||
- Hybrid telemetry raises classification accuracy over the corresponding
|
||||
simulator+outcome transition regression by 1.67--4.17 percentage points,
|
||||
but worsens pass-rate RMSE by 0.141--0.201 and MAE by 0.084--0.125. Its full
|
||||
correction reaches only 46.67--53.33% absolute accuracy.
|
||||
- Across 24 nonzero `(L2, raw-simulator-prior weight)` combinations, no model
|
||||
both corrects an existing simulator error without more new errors and avoids
|
||||
worsening RMSE/MAE. Whenever a correction fixes at least one error, it
|
||||
corrupts at least 11 previously correct transitions.
|
||||
- A perfect correction could skip the frozen simulator rank-2 real final and
|
||||
save 0.043469 H20-hours: 15.45% of the prospective online `k=2` cost, or
|
||||
14.40% when the prior failed launch is charged. On this development task the
|
||||
simulator top-1 already is the real oracle with zero regret, so headroom
|
||||
versus the observed-safe top-1 baseline is 0%.
|
||||
|
||||
The result does not prove that engine telemetry is useless. It shows that the
|
||||
current one-task anchor-transition evidence cannot support either a safe
|
||||
simulator-residual tuner or a simulator-free telemetry tuner. A larger model
|
||||
or an R1 run would add capacity/data after a failed gate and is therefore not
|
||||
authorized under this roadmap.
|
||||
|
||||
### R1 — New development surface
|
||||
|
||||
Status: **NOT LAUNCHED; CLOSED BY R0**.
|
||||
|
||||
Frozen starting setup:
|
||||
|
||||
- host: dash0, eight NVIDIA H20 GPUs;
|
||||
- cells run solo; no co-location for SLO verdicts;
|
||||
- patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16;
|
||||
- trace: `chat_w20260313_1000`;
|
||||
- output tokens: exactly 128;
|
||||
- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, pass rate at least 0.95;
|
||||
- config surface: TP `{1,2,4}` × MNS `{8,16,32,64}`;
|
||||
- hard campaign cap: 4 H20-hours.
|
||||
|
||||
The load ladder, repetitions, randomized order, exact commands, expected wall
|
||||
time, and artifact paths are frozen only after R0. A resolved echo is required
|
||||
before launch.
|
||||
|
||||
R1 passes only if a frozen sequential replay shows at least 15% E2E H20-hour
|
||||
headroom over the strongest safe baseline with final regret at most 5%. R1 is
|
||||
development evidence and cannot be reported as the held-out result.
|
||||
|
||||
### R2 — Held-out sequential tuning
|
||||
|
||||
Status: **NOT LAUNCHED; CLOSED BY R0**.
|
||||
|
||||
Required baselines:
|
||||
|
||||
1. random search;
|
||||
2. real-outcome-only Bayesian/sequential search;
|
||||
3. Frontier ranking plus real top-k final;
|
||||
4. simulator plus real-outcome residual;
|
||||
5. telemetry-only transition tuner;
|
||||
6. simulator plus telemetry residual tuner;
|
||||
7. complete real surface as oracle, not as a cost competitor.
|
||||
|
||||
Primary metric: end-to-end H20-hours to first reach 95% of the real full-surface
|
||||
SLO-goodput oracle. Secondary metrics are cost-normalized regret AUC, final
|
||||
regret at fixed budgets, oracle false-prune, wall time, and per-task regressions.
|
||||
|
||||
The route is successful only if the winning telemetry method reduces the
|
||||
primary cost by at least 20% versus the strongest safe baseline and ends within
|
||||
5% regret on every headline task. If hybrid beats telemetry-only by at least
|
||||
10%, simulator residual correction is the primary method. If telemetry-only
|
||||
is within 5% or better, the simulator dependency is removed. If neither clears
|
||||
the contribution bar, the route is closed and telemetry remains a diagnostic
|
||||
facility only.
|
||||
|
||||
## Cost discipline
|
||||
|
||||
- R0 simulator work is CPU-only and must set empty CUDA visibility.
|
||||
- R1 cannot exceed 4 H20-hours.
|
||||
- R2 receives no budget until R1 passes.
|
||||
- Startup, warm-up, burn-in, failed launches, real probes, continuation, and
|
||||
final validation are charged. Benchmark-only annotation repeats are
|
||||
reported separately and cannot disappear from campaign accounting.
|
||||
|
||||
## Final R0 sanity block
|
||||
|
||||
| Data | n | Min | Max | Distinct | Checked invariant |
|
||||
|---|---:|---:|---:|---:|---|
|
||||
| Phase 6 cells | 12 | TP1/MNS8 | TP4/MNS64 | 12 | Surface not identical; solo SLO tier authoritative |
|
||||
| Phase 6 Layer-1 primary steps | 37 streams | 343 | 12,103 | 37 | Contiguous; zero drops |
|
||||
| P1 primary anchors | 12 | infeasible | feasible | 2 labels | 7 feasible / 5 infeasible |
|
||||
| P1 Frontier runtime | 12 | 24.093 s | 54.575 s | 12 | CPU-only; zero failures |
|
||||
| Detailed Frontier replay runtime | 12 | 23.943 s | 54.786 s | 12 | Exact committed scorers; CUDA hidden |
|
||||
| Detailed artifact bytes | 12 | 4,123,724 | 13,527,776 | 12 | Non-negative; practical CPU replay size |
|
||||
| Cross-config transitions | 120 | real pass 0.1067 | real pass 1.0 | 6 outcomes | Both endpoint config identities held out |
|
||||
| State residual vectors | 12 | 16 fields | 16 fields | 12 vectors | Finite; no missing common field |
|
||||
| R0 E2E cost values | 4 | 0.237914 | 0.301935 H20-h | 4 | Non-negative; `k=1/2`, online/conservative |
|
||||
|
||||
Checked invariants: non-negative counts and costs; pass rates in `[0,1]`;
|
||||
simulator results not all identical; exact request count/hash agreement; Layer-1
|
||||
step continuity and zero drops; no co-resident SLO measurements; no calibration
|
||||
or evaluation split reuse for a future headline claim. No current red flag
|
||||
invalidates R0 plumbing. The R0 tuning gate itself failed because safe
|
||||
prior-preserving correction was absent.
|
||||
Reference in New Issue
Block a user