130 lines
6.0 KiB
Markdown
130 lines
6.0 KiB
Markdown
# Active intervention + measurement v0 protocol
|
|
|
|
Date: 2026-07-15 (Asia/Singapore)
|
|
|
|
Status: **FROZEN BEFORE THE `chat_w20260313_1000` GPU RUN**.
|
|
|
|
## Research question
|
|
|
|
This experiment asks whether a tuner conditioned on direct engine-state
|
|
trajectories can choose both a measurement horizon and a coupled configuration
|
|
intervention with lower real-GPU cost than the same tuner using only external
|
|
prefix outcomes.
|
|
|
|
The contribution is not the controller, legality checks, telemetry collection,
|
|
or the ridge model. The route remains open only if engine state changes an
|
|
actual decision and reduces cost-to-near-oracle on unseen workloads.
|
|
|
|
## Development result that motivates, but does not pass, the route
|
|
|
|
The frozen trace-12 dataset contains 72 examples: six source decisions, four
|
|
measurement checkpoints, and `noop/MNS/MBBT` actions. Features are direct
|
|
continuous Layer-1 state summaries; cap-exclusive and bottleneck labels are
|
|
excluded. Leave-one-repetition-out sequential replay uses the same model,
|
|
candidate set, confidence rule, and checkpoint set for both modes.
|
|
|
|
The external-outcome policy and telemetry policy both put all six decisions
|
|
within 2% regret. Outcome-only selected a mean 262.5-second source measurement
|
|
and cost 3.750 replay H20-hours across the six replayed decisions; telemetry
|
|
selected 275 seconds and cost 3.833 H20-hours. Telemetry therefore increased
|
|
the replay lower-bound cost by 2.22%, with no regret reduction. This is a
|
|
negative result. It does not settle the question because the dataset has only
|
|
two source regimes, one source is at the offered ceiling, and there is no joint
|
|
MNS+MBBT action.
|
|
|
|
Sanity: n=6 decisions; regret min=0, max=0.009412, distinct=3; source cutoff
|
|
min=150s, max=300s, distinct=3 across the two policies; all costs are
|
|
non-negative, regrets are in `[0,1]`, target results are not all identical, and
|
|
the six decisions are complete exact-workload pairs.
|
|
|
|
## Frozen prospective setup
|
|
|
|
- Host: `dash0`, 8 NVIDIA H20 GPUs available; each TP4 server runs alone on
|
|
GPUs 0-3. Co-location is prohibited for SLO verdicts.
|
|
- Engine: patched vLLM `0.24.1.dev3+opprof` from
|
|
`/home/admin/cpfs/wjh/vllm-opprof-phase3` in
|
|
`/tmp/wjh/venvs/vllm-0.20.0-cu129`.
|
|
- Model: `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, BF16.
|
|
- Workload: unseen `chat_w20260313_1000`; input 0-8192; output exactly 128;
|
|
replay scale 0.5; 300-second arrival window.
|
|
- Three disjoint repetitions: source rows are assigned by a deterministic
|
|
SHA-256 modulo-3 partition before input filtering. Each repetition selects
|
|
approximately 3300 requests, or 2.75 requests/s/GPU at TP4.
|
|
- SLO: at least 95% pass; stepped TTFT 2/4/6 seconds; TPOT at most 50 ms.
|
|
- Checkpoints: 75, 150, 225, and 300 seconds.
|
|
- Full 2x2 surface:
|
|
- source: `MNS=32, MBBT=4096`;
|
|
- MNS action: `64,4096`;
|
|
- MBBT action: `32,8192`;
|
|
- joint action: `64,8192`;
|
|
- `noop` retains the source.
|
|
- Four config sessions are serialized. Each session uses a fresh server,
|
|
warm-up, burn-in, and counter-rotated repetition order.
|
|
- Expected campaign cost: 4.6-5.5 H20-hours; hard cap: 6.0 H20-hours;
|
|
expected wall time: 75-100 minutes.
|
|
|
|
The source is executed first. The frozen telemetry policy selects the next
|
|
real config session; all remaining cells are then measured only to construct
|
|
the exact finite-surface oracle. Oracle annotation after the selected action
|
|
is reported separately from tuner cost.
|
|
|
|
## Frozen policies
|
|
|
|
Both policies fit the paired treatment effect
|
|
|
|
```text
|
|
target normalized SLO-goodput - source normalized SLO-goodput
|
|
```
|
|
|
|
from source config, full config delta, offered load, and external prefix
|
|
outcomes. The telemetry policy additionally receives fixed direct Layer-1
|
|
summaries and their interactions with `delta_log2(MNS)` and
|
|
`delta_log2(MBBT)`. It does not receive a bottleneck label or a
|
|
diagnosis-to-knob rule.
|
|
|
|
At each checkpoint, jackknife models produce an effect distribution for
|
|
`noop`, MNS, MBBT, and joint actions. Measurement stops at the earliest second
|
|
consecutive checkpoint with the same confident best action; otherwise it uses
|
|
the full 300 seconds. Confidence requires a predicted margin of at least 0.02
|
|
and the best lower bound to exceed the second-best upper bound. If the final
|
|
choice is not confident, the next run is the positive-UCB action, explicitly
|
|
marked as a diagnostic intervention. The exact same rule is used for the
|
|
outcome-only baseline.
|
|
|
|
## Hypotheses and gates
|
|
|
|
### H1: action value
|
|
|
|
Engine state must change the selected intervention or its ranking and reduce
|
|
real action regret. Prediction error or bottleneck-label accuracy is not a
|
|
success metric.
|
|
|
|
### H2: measurement value
|
|
|
|
Engine state must select a shorter stable source measurement without increasing
|
|
action regret. A shorter reconstructed prefix is only a trigger; it is not an
|
|
actual GPU-cost claim until an early-terminated confirmation run measures
|
|
startup, warm-up, drain, and cleanup.
|
|
|
|
### H3: end-to-end cost
|
|
|
|
Primary development metric is H20-hours to first reach a configuration within
|
|
2% of the exact median-goodput oracle. The outcome-only and telemetry policies
|
|
use the same measured config costs and differ only in source information.
|
|
|
|
- At least 10% prospective replay cost reduction, telemetry regret at most 2%,
|
|
and no outcome-only-to-telemetry harm triggers an actual early-stop
|
|
confirmation.
|
|
- At least 20% measured all-in H20-hour reduction is required for a contribution
|
|
claim. This one task can only establish development feasibility; a paper
|
|
claim additionally requires task-held-out replication.
|
|
- Source median normalized goodput at or above 0.98 stops the surface before
|
|
target runs because the workload has no material improvement headroom.
|
|
- Any hash mismatch, missing/censored result, telemetry drop, non-monotonic
|
|
phase, negative cost, ratio outside `[0,1]`, or all-identical config outcomes
|
|
is a red flag and stops analysis.
|
|
|
|
If the 10% trigger fails, this route is closed for the current engine-state
|
|
representation. The experimental control plane is not retained as a fallback
|
|
research contribution.
|