Add prospective active intervention experiment

docs/active-intervention-v0-protocol-20260715.md

@@ -0,0 +1,129 @@
+# Active intervention + measurement v0 protocol
+
+Date: 2026-07-15 (Asia/Singapore)
+
+Status: **FROZEN BEFORE THE `chat_w20260313_1000` GPU RUN**.
+
+## Research question
+
+This experiment asks whether a tuner conditioned on direct engine-state
+trajectories can choose both a measurement horizon and a coupled configuration
+intervention with lower real-GPU cost than the same tuner using only external
+prefix outcomes.
+
+The contribution is not the controller, legality checks, telemetry collection,
+or the ridge model.  The route remains open only if engine state changes an
+actual decision and reduces cost-to-near-oracle on unseen workloads.
+
+## Development result that motivates, but does not pass, the route
+
+The frozen trace-12 dataset contains 72 examples: six source decisions, four
+measurement checkpoints, and `noop/MNS/MBBT` actions.  Features are direct
+continuous Layer-1 state summaries; cap-exclusive and bottleneck labels are
+excluded.  Leave-one-repetition-out sequential replay uses the same model,
+candidate set, confidence rule, and checkpoint set for both modes.
+
+The external-outcome policy and telemetry policy both put all six decisions
+within 2% regret.  Outcome-only selected a mean 262.5-second source measurement
+and cost 3.750 replay H20-hours across the six replayed decisions; telemetry
+selected 275 seconds and cost 3.833 H20-hours.  Telemetry therefore increased
+the replay lower-bound cost by 2.22%, with no regret reduction.  This is a
+negative result.  It does not settle the question because the dataset has only
+two source regimes, one source is at the offered ceiling, and there is no joint
+MNS+MBBT action.
+
+Sanity: n=6 decisions; regret min=0, max=0.009412, distinct=3; source cutoff
+min=150s, max=300s, distinct=3 across the two policies; all costs are
+non-negative, regrets are in `[0,1]`, target results are not all identical, and
+the six decisions are complete exact-workload pairs.
+
+## Frozen prospective setup
+
+- Host: `dash0`, 8 NVIDIA H20 GPUs available; each TP4 server runs alone on
+  GPUs 0-3.  Co-location is prohibited for SLO verdicts.
+- Engine: patched vLLM `0.24.1.dev3+opprof` from
+  `/home/admin/cpfs/wjh/vllm-opprof-phase3` in
+  `/tmp/wjh/venvs/vllm-0.20.0-cu129`.
+- Model: `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, BF16.
+- Workload: unseen `chat_w20260313_1000`; input 0-8192; output exactly 128;
+  replay scale 0.5; 300-second arrival window.
+- Three disjoint repetitions: source rows are assigned by a deterministic
+  SHA-256 modulo-3 partition before input filtering.  Each repetition selects
+  approximately 3300 requests, or 2.75 requests/s/GPU at TP4.
+- SLO: at least 95% pass; stepped TTFT 2/4/6 seconds; TPOT at most 50 ms.
+- Checkpoints: 75, 150, 225, and 300 seconds.
+- Full 2x2 surface:
+  - source: `MNS=32, MBBT=4096`;
+  - MNS action: `64,4096`;
+  - MBBT action: `32,8192`;
+  - joint action: `64,8192`;
+  - `noop` retains the source.
+- Four config sessions are serialized.  Each session uses a fresh server,
+  warm-up, burn-in, and counter-rotated repetition order.
+- Expected campaign cost: 4.6-5.5 H20-hours; hard cap: 6.0 H20-hours;
+  expected wall time: 75-100 minutes.
+
+The source is executed first.  The frozen telemetry policy selects the next
+real config session; all remaining cells are then measured only to construct
+the exact finite-surface oracle.  Oracle annotation after the selected action
+is reported separately from tuner cost.
+
+## Frozen policies
+
+Both policies fit the paired treatment effect
+
+```text
+target normalized SLO-goodput - source normalized SLO-goodput
+```
+
+from source config, full config delta, offered load, and external prefix
+outcomes.  The telemetry policy additionally receives fixed direct Layer-1
+summaries and their interactions with `delta_log2(MNS)` and
+`delta_log2(MBBT)`.  It does not receive a bottleneck label or a
+diagnosis-to-knob rule.
+
+At each checkpoint, jackknife models produce an effect distribution for
+`noop`, MNS, MBBT, and joint actions.  Measurement stops at the earliest second
+consecutive checkpoint with the same confident best action; otherwise it uses
+the full 300 seconds.  Confidence requires a predicted margin of at least 0.02
+and the best lower bound to exceed the second-best upper bound.  If the final
+choice is not confident, the next run is the positive-UCB action, explicitly
+marked as a diagnostic intervention.  The exact same rule is used for the
+outcome-only baseline.
+
+## Hypotheses and gates
+
+### H1: action value
+
+Engine state must change the selected intervention or its ranking and reduce
+real action regret.  Prediction error or bottleneck-label accuracy is not a
+success metric.
+
+### H2: measurement value
+
+Engine state must select a shorter stable source measurement without increasing
+action regret.  A shorter reconstructed prefix is only a trigger; it is not an
+actual GPU-cost claim until an early-terminated confirmation run measures
+startup, warm-up, drain, and cleanup.
+
+### H3: end-to-end cost
+
+Primary development metric is H20-hours to first reach a configuration within
+2% of the exact median-goodput oracle.  The outcome-only and telemetry policies
+use the same measured config costs and differ only in source information.
+
+- At least 10% prospective replay cost reduction, telemetry regret at most 2%,
+  and no outcome-only-to-telemetry harm triggers an actual early-stop
+  confirmation.
+- At least 20% measured all-in H20-hour reduction is required for a contribution
+  claim.  This one task can only establish development feasibility; a paper
+  claim additionally requires task-held-out replication.
+- Source median normalized goodput at or above 0.98 stops the surface before
+  target runs because the workload has no material improvement headroom.
+- Any hash mismatch, missing/censored result, telemetry drop, non-monotonic
+  phase, negative cost, ratio outside `[0,1]`, or all-identical config outcomes
+  is a red flag and stops analysis.
+
+If the 10% trigger fails, this route is closed for the current engine-state
+representation.  The experimental control plane is not retained as a fallback
+research contribution.