Active intervention + measurement v0 protocol

Date: 2026-07-15 (Asia/Singapore)

Status: FROZEN BEFORE THE chat_w20260313_1000 GPU RUN.

Research question

This experiment asks whether a tuner conditioned on direct engine-state trajectories can choose both a measurement horizon and a coupled configuration intervention with lower real-GPU cost than the same tuner using only external prefix outcomes.

The contribution is not the controller, legality checks, telemetry collection, or the ridge model. The route remains open only if engine state changes an actual decision and reduces cost-to-near-oracle on unseen workloads.

Development result that motivates, but does not pass, the route

The frozen trace-12 dataset contains 72 examples: six source decisions, four measurement checkpoints, and noop/MNS/MBBT actions. Features are direct continuous Layer-1 state summaries; cap-exclusive and bottleneck labels are excluded. Leave-one-repetition-out sequential replay uses the same model, candidate set, confidence rule, and checkpoint set for both modes.

The external-outcome policy and telemetry policy both put all six decisions within 2% regret. Outcome-only selected a mean 262.5-second source measurement and cost 3.750 replay H20-hours across the six replayed decisions; telemetry selected 275 seconds and cost 3.833 H20-hours. Telemetry therefore increased the replay lower-bound cost by 2.22%, with no regret reduction. This is a negative result. It does not settle the question because the dataset has only two source regimes, one source is at the offered ceiling, and there is no joint MNS+MBBT action.

Sanity: n=6 decisions; regret min=0, max=0.009412, distinct=3; source cutoff min=150s, max=300s, distinct=3 across the two policies; all costs are non-negative, regrets are in [0,1], target results are not all identical, and the six decisions are complete exact-workload pairs.

Frozen prospective setup

Host: dash0, 8 NVIDIA H20 GPUs available; each TP4 server runs alone on GPUs 0-3. Co-location is prohibited for SLO verdicts.
Engine: patched vLLM 0.24.1.dev3+opprof from /home/admin/cpfs/wjh/vllm-opprof-phase3 in /tmp/wjh/venvs/vllm-0.20.0-cu129.
Model: /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B, BF16.
Workload: unseen chat_w20260313_1000; input 0-8192; output exactly 128; replay scale 0.5; 300-second arrival window.
Three disjoint repetitions: source rows are assigned by a deterministic SHA-256 modulo-3 partition before input filtering. Each repetition selects approximately 3300 requests, or 2.75 requests/s/GPU at TP4.
SLO: at least 95% pass; stepped TTFT 2/4/6 seconds; TPOT at most 50 ms.
Checkpoints: 75, 150, 225, and 300 seconds.
Full 2x2 surface:
- source: MNS=32, MBBT=4096;
- MNS action: 64,4096;
- MBBT action: 32,8192;
- joint action: 64,8192;
- noop retains the source.
Four config sessions are serialized. Each session uses a fresh server, warm-up, burn-in, and counter-rotated repetition order.
Expected campaign cost: 4.6-5.5 H20-hours; hard cap: 6.0 H20-hours; expected wall time: 75-100 minutes.

The source is executed first. The frozen telemetry policy selects the next real config session; all remaining cells are then measured only to construct the exact finite-surface oracle. Oracle annotation after the selected action is reported separately from tuner cost.

Frozen policies

Both policies fit the paired treatment effect

target normalized SLO-goodput - source normalized SLO-goodput

from source config, full config delta, offered load, and external prefix outcomes. The telemetry policy additionally receives fixed direct Layer-1 summaries and their interactions with delta_log2(MNS) and delta_log2(MBBT). It does not receive a bottleneck label or a diagnosis-to-knob rule.

At each checkpoint, jackknife models produce an effect distribution for noop, MNS, MBBT, and joint actions. Measurement stops at the earliest second consecutive checkpoint with the same confident best action; otherwise it uses the full 300 seconds. Confidence requires a predicted margin of at least 0.02 and the best lower bound to exceed the second-best upper bound. If the final choice is not confident, the next run is the positive-UCB action, explicitly marked as a diagnostic intervention. The exact same rule is used for the outcome-only baseline.

Hypotheses and gates

H1: action value

Engine state must change the selected intervention or its ranking and reduce real action regret. Prediction error or bottleneck-label accuracy is not a success metric.

H2: measurement value

Engine state must select a shorter stable source measurement without increasing action regret. A shorter reconstructed prefix is only a trigger; it is not an actual GPU-cost claim until an early-terminated confirmation run measures startup, warm-up, drain, and cleanup.

H3: end-to-end cost

Primary development metric is H20-hours to first reach a configuration within 2% of the exact median-goodput oracle. The outcome-only and telemetry policies use the same measured config costs and differ only in source information.

At least 10% prospective replay cost reduction, telemetry regret at most 2%, and no outcome-only-to-telemetry harm triggers an actual early-stop confirmation.
At least 20% measured all-in H20-hour reduction is required for a contribution claim. This one task can only establish development feasibility; a paper claim additionally requires task-held-out replication.
Source median normalized goodput at or above 0.98 stops the surface before target runs because the workload has no material improvement headroom.
Any hash mismatch, missing/censored result, telemetry drop, non-monotonic phase, negative cost, ratio outside [0,1], or all-identical config outcomes is a red flag and stops analysis.

If the 10% trigger fails, this route is closed for the current engine-state representation. The experimental control plane is not retained as a fallback research contribution.

6.0 KiB Raw Blame History