107 lines
5.0 KiB
Markdown
107 lines
5.0 KiB
Markdown
# Active intervention v0: held-out trace-13 result
|
|
|
|
Date: 2026-07-15 (Asia/Singapore)
|
|
|
|
Decision: **close the passive-telemetry treatment-effect route**. The held-out
|
|
campaign produced no telemetry-induced action change, measurement reduction, or
|
|
GPU-cost reduction. It did show that the engine state contained the correct
|
|
action-specific mechanism; the current feature model failed to use it.
|
|
|
|
## Headline result
|
|
|
|
The outcome-only and telemetry policies both measured the source for 300
|
|
seconds, selected `joint=(MNS64,MBBT8192)`, and produced the same complete
|
|
acquisition order. Both reached the exact finite-surface oracle after the
|
|
first intervention at a reconstructed all-in lower-bound cost of 2.4284
|
|
H20-hours. Telemetry GPU-cost reduction was therefore exactly 0%, below the
|
|
10% confirmation trigger and 20% contribution gate. No actual early-stop
|
|
confirmation was launched.
|
|
|
|
The complete annotation campaign cost 5.0379 H20-hours, below the 6.0 H20-hour
|
|
hard cap. It ran 12 uncensored real-GPU outcomes: four configs, three disjoint
|
|
request partitions, and a fresh server per config.
|
|
|
|
## Exact response surface
|
|
|
|
Median normalized SLO-goodput was:
|
|
|
|
| Config | Rep values | Median |
|
|
|---|---|---:|
|
|
| `MNS32,MBBT4096` source | 0.40091 / 0.39788 / 0.42061 | 0.40091 |
|
|
| `MNS64,MBBT4096` | 1.00000 / 0.99970 / 1.00000 | 1.00000 |
|
|
| `MNS32,MBBT8192` | 0.44394 / 0.41515 / 0.42606 | 0.42606 |
|
|
| `MNS64,MBBT8192` joint | 1.00000 / 1.00000 / 1.00000 | 1.00000 |
|
|
|
|
Increasing MNS alone was sufficient and joint was redundant. Increasing MBBT
|
|
alone improved the median by only 0.02515, versus 0.59909 for MNS. This is a
|
|
strong non-additive action response, not a setting where independently tuning
|
|
the knobs and merging their improvements is valid.
|
|
|
|
## What the telemetry actually said
|
|
|
|
Across 41,086 source scheduler records, 93.12% of steps had waiting work,
|
|
85.36% were MNS-exclusive binding, 1.11% were MBBT-exclusive, mean running-slot
|
|
utilization was 97.39%, mean token-budget utilization was 15.69%, mean KV usage
|
|
was 2.75%, and there were no preemptions.
|
|
|
|
The intervention transition agreed with that state:
|
|
|
|
| Config | Waiting | MNS-exclusive | MBBT-exclusive | Median goodput |
|
|
|---|---:|---:|---:|---:|
|
|
| source | 93.12% | 85.36% | 1.11% | 0.40091 |
|
|
| MNS only | 5.38% | 0% | 5.38% | 1.00000 |
|
|
| MBBT only | 91.19% | 91.09% | 0.04% | 0.42606 |
|
|
| joint | 0.89% | 0% | 0.89% | 1.00000 |
|
|
|
|
Thus this experiment does **not** support the claim that engine telemetry lacks
|
|
tuning information. It rejects the narrower claim that adding passive state
|
|
summaries to the current small-data ridge policy converts that information into
|
|
lower tuning cost.
|
|
|
|
## Why the learned policy failed
|
|
|
|
At 300 seconds, the telemetry model predicted joint, MNS, and MBBT effects of
|
|
0.35190, 0.26118, and 0.09686. The actual median effects were 0.59909,
|
|
0.59909, and 0.02515. Telemetry therefore made the nonexistent joint-over-MNS
|
|
gap larger: 0.09072 predicted versus 0 actual; the outcome-only model predicted
|
|
0.03188.
|
|
|
|
The failure has three concrete causes:
|
|
|
|
1. The six training decisions contain no joint intervention. The
|
|
`delta_product` feature has no support, so joint ranking is extrapolation.
|
|
2. Passive raw summaries do not represent the counterfactual scheduler work
|
|
unlocked by each action. Capacity-normalized MNS pressure was visible, but
|
|
the model was not structurally required to map it to MNS marginal value.
|
|
3. The policy maximizes predicted effect. It does not identify the smallest
|
|
epsilon-optimal intervention or price unsupported action complexity.
|
|
|
|
## Research implication
|
|
|
|
Do not retain the harness or the passive telemetry model as a contribution.
|
|
The next defensible route is engine-native, action-conditional counterfactual
|
|
instrumentation: at a real scheduling state, shadow-replay the exact scheduler
|
|
decision under an MNS relaxation, MBBT relaxation, and their joint relaxation,
|
|
then expose the incremental queued work admitted by each action. Real paired
|
|
interventions calibrate how those one-step shadow effects map to E2E SLO
|
|
goodput. This is distinct from a hand-written cap-to-knob rule and from a
|
|
full-system simulator: it reuses the exact live queue, scheduler, and cache
|
|
state while simulating only the local decision boundary.
|
|
|
|
That route should be evaluated against outcome-only search, the present passive
|
|
telemetry model, a cap-hit expert rule, and a full simulator. The paper-level
|
|
gate remains at least 20% measured H20-hour reduction to a 2%-oracle config on
|
|
task-held-out workloads with at most 2% regret.
|
|
|
|
## Sanity
|
|
|
|
Surface outcomes: n=12, min=0.39788, max=1.0, distinct=8. Session costs: n=4,
|
|
min=1.1702, max=1.3566 H20-hours, distinct=4. Scheduler-record counts: n=4,
|
|
min=37,001, max=41,348, distinct=4. All counters and costs were non-negative;
|
|
all ratios were in `[0,1]`; request hashes matched; all 12 runs were uncensored;
|
|
the controller and four sessions completed; and config outcomes were not all
|
|
identical. No red flags were found.
|
|
|
|
Machine-readable summary: `runs/active-intervention-v0/trace13-results.json`.
|
|
Raw immutable root: `/home/admin/cpfs/wjh/active-intervention-prospective-20260715`.
|