Report held-out active intervention result

docs/active-intervention-v0-results-20260715.md

@@ -0,0 +1,106 @@
+# Active intervention v0: held-out trace-13 result
+
+Date: 2026-07-15 (Asia/Singapore)
+
+Decision: **close the passive-telemetry treatment-effect route**.  The held-out
+campaign produced no telemetry-induced action change, measurement reduction, or
+GPU-cost reduction.  It did show that the engine state contained the correct
+action-specific mechanism; the current feature model failed to use it.
+
+## Headline result
+
+The outcome-only and telemetry policies both measured the source for 300
+seconds, selected `joint=(MNS64,MBBT8192)`, and produced the same complete
+acquisition order.  Both reached the exact finite-surface oracle after the
+first intervention at a reconstructed all-in lower-bound cost of 2.4284
+H20-hours.  Telemetry GPU-cost reduction was therefore exactly 0%, below the
+10% confirmation trigger and 20% contribution gate.  No actual early-stop
+confirmation was launched.
+
+The complete annotation campaign cost 5.0379 H20-hours, below the 6.0 H20-hour
+hard cap.  It ran 12 uncensored real-GPU outcomes: four configs, three disjoint
+request partitions, and a fresh server per config.
+
+## Exact response surface
+
+Median normalized SLO-goodput was:
+
+| Config | Rep values | Median |
+|---|---|---:|
+| `MNS32,MBBT4096` source | 0.40091 / 0.39788 / 0.42061 | 0.40091 |
+| `MNS64,MBBT4096` | 1.00000 / 0.99970 / 1.00000 | 1.00000 |
+| `MNS32,MBBT8192` | 0.44394 / 0.41515 / 0.42606 | 0.42606 |
+| `MNS64,MBBT8192` joint | 1.00000 / 1.00000 / 1.00000 | 1.00000 |
+
+Increasing MNS alone was sufficient and joint was redundant.  Increasing MBBT
+alone improved the median by only 0.02515, versus 0.59909 for MNS.  This is a
+strong non-additive action response, not a setting where independently tuning
+the knobs and merging their improvements is valid.
+
+## What the telemetry actually said
+
+Across 41,086 source scheduler records, 93.12% of steps had waiting work,
+85.36% were MNS-exclusive binding, 1.11% were MBBT-exclusive, mean running-slot
+utilization was 97.39%, mean token-budget utilization was 15.69%, mean KV usage
+was 2.75%, and there were no preemptions.
+
+The intervention transition agreed with that state:
+
+| Config | Waiting | MNS-exclusive | MBBT-exclusive | Median goodput |
+|---|---:|---:|---:|---:|
+| source | 93.12% | 85.36% | 1.11% | 0.40091 |
+| MNS only | 5.38% | 0% | 5.38% | 1.00000 |
+| MBBT only | 91.19% | 91.09% | 0.04% | 0.42606 |
+| joint | 0.89% | 0% | 0.89% | 1.00000 |
+
+Thus this experiment does **not** support the claim that engine telemetry lacks
+tuning information.  It rejects the narrower claim that adding passive state
+summaries to the current small-data ridge policy converts that information into
+lower tuning cost.
+
+## Why the learned policy failed
+
+At 300 seconds, the telemetry model predicted joint, MNS, and MBBT effects of
+0.35190, 0.26118, and 0.09686.  The actual median effects were 0.59909,
+0.59909, and 0.02515.  Telemetry therefore made the nonexistent joint-over-MNS
+gap larger: 0.09072 predicted versus 0 actual; the outcome-only model predicted
+0.03188.
+
+The failure has three concrete causes:
+
+1. The six training decisions contain no joint intervention.  The
+   `delta_product` feature has no support, so joint ranking is extrapolation.
+2. Passive raw summaries do not represent the counterfactual scheduler work
+   unlocked by each action.  Capacity-normalized MNS pressure was visible, but
+   the model was not structurally required to map it to MNS marginal value.
+3. The policy maximizes predicted effect.  It does not identify the smallest
+   epsilon-optimal intervention or price unsupported action complexity.
+
+## Research implication
+
+Do not retain the harness or the passive telemetry model as a contribution.
+The next defensible route is engine-native, action-conditional counterfactual
+instrumentation: at a real scheduling state, shadow-replay the exact scheduler
+decision under an MNS relaxation, MBBT relaxation, and their joint relaxation,
+then expose the incremental queued work admitted by each action.  Real paired
+interventions calibrate how those one-step shadow effects map to E2E SLO
+goodput.  This is distinct from a hand-written cap-to-knob rule and from a
+full-system simulator: it reuses the exact live queue, scheduler, and cache
+state while simulating only the local decision boundary.
+
+That route should be evaluated against outcome-only search, the present passive
+telemetry model, a cap-hit expert rule, and a full simulator.  The paper-level
+gate remains at least 20% measured H20-hour reduction to a 2%-oracle config on
+task-held-out workloads with at most 2% regret.
+
+## Sanity
+
+Surface outcomes: n=12, min=0.39788, max=1.0, distinct=8.  Session costs: n=4,
+min=1.1702, max=1.3566 H20-hours, distinct=4.  Scheduler-record counts: n=4,
+min=37,001, max=41,348, distinct=4.  All counters and costs were non-negative;
+all ratios were in `[0,1]`; request hashes matched; all 12 runs were uncensored;
+the controller and four sessions completed; and config outcomes were not all
+identical.  No red flags were found.
+
+Machine-readable summary: `runs/active-intervention-v0/trace13-results.json`.
+Raw immutable root: `/home/admin/cpfs/wjh/active-intervention-prospective-20260715`.