Active intervention v0: held-out trace-13 result

Date: 2026-07-15 (Asia/Singapore)

Decision: close the passive-telemetry treatment-effect route. The held-out campaign produced no telemetry-induced action change, measurement reduction, or GPU-cost reduction. It did show that the engine state contained the correct action-specific mechanism; the current feature model failed to use it.

Headline result

The outcome-only and telemetry policies both measured the source for 300 seconds, selected joint=(MNS64,MBBT8192), and produced the same complete acquisition order. Both reached the exact finite-surface oracle after the first intervention at a reconstructed all-in lower-bound cost of 2.4284 H20-hours. Telemetry GPU-cost reduction was therefore exactly 0%, below the 10% confirmation trigger and 20% contribution gate. No actual early-stop confirmation was launched.

The complete annotation campaign cost 5.0379 H20-hours, below the 6.0 H20-hour hard cap. It ran 12 uncensored real-GPU outcomes: four configs, three disjoint request partitions, and a fresh server per config.

Exact response surface

Median normalized SLO-goodput was:

Config	Rep values	Median
`MNS32,MBBT4096` source	0.40091 / 0.39788 / 0.42061	0.40091
`MNS64,MBBT4096`	1.00000 / 0.99970 / 1.00000	1.00000
`MNS32,MBBT8192`	0.44394 / 0.41515 / 0.42606	0.42606
`MNS64,MBBT8192` joint	1.00000 / 1.00000 / 1.00000	1.00000

Increasing MNS alone was sufficient and joint was redundant. Increasing MBBT alone improved the median by only 0.02515, versus 0.59909 for MNS. This is a strong non-additive action response, not a setting where independently tuning the knobs and merging their improvements is valid.

What the telemetry actually said

Across 41,086 source scheduler records, 93.12% of steps had waiting work, 85.36% were MNS-exclusive binding, 1.11% were MBBT-exclusive, mean running-slot utilization was 97.39%, mean token-budget utilization was 15.69%, mean KV usage was 2.75%, and there were no preemptions.

The intervention transition agreed with that state:

Config	Waiting	MNS-exclusive	MBBT-exclusive	Median goodput
source	93.12%	85.36%	1.11%	0.40091
MNS only	5.38%	0%	5.38%	1.00000
MBBT only	91.19%	91.09%	0.04%	0.42606
joint	0.89%	0%	0.89%	1.00000

Thus this experiment does not support the claim that engine telemetry lacks tuning information. It rejects the narrower claim that adding passive state summaries to the current small-data ridge policy converts that information into lower tuning cost.

Why the learned policy failed

At 300 seconds, the telemetry model predicted joint, MNS, and MBBT effects of 0.35190, 0.26118, and 0.09686. The actual median effects were 0.59909, 0.59909, and 0.02515. Telemetry therefore made the nonexistent joint-over-MNS gap larger: 0.09072 predicted versus 0 actual; the outcome-only model predicted 0.03188.

The failure has three concrete causes:

The six training decisions contain no joint intervention. The delta_product feature has no support, so joint ranking is extrapolation.
Passive raw summaries do not represent the counterfactual scheduler work unlocked by each action. Capacity-normalized MNS pressure was visible, but the model was not structurally required to map it to MNS marginal value.
The policy maximizes predicted effect. It does not identify the smallest epsilon-optimal intervention or price unsupported action complexity.

Research implication

Do not retain the harness or the passive telemetry model as a contribution. The next defensible route is engine-native, action-conditional counterfactual instrumentation: at a real scheduling state, shadow-replay the exact scheduler decision under an MNS relaxation, MBBT relaxation, and their joint relaxation, then expose the incremental queued work admitted by each action. Real paired interventions calibrate how those one-step shadow effects map to E2E SLO goodput. This is distinct from a hand-written cap-to-knob rule and from a full-system simulator: it reuses the exact live queue, scheduler, and cache state while simulating only the local decision boundary.

That route should be evaluated against outcome-only search, the present passive telemetry model, a cap-hit expert rule, and a full simulator. The paper-level gate remains at least 20% measured H20-hour reduction to a 2%-oracle config on task-held-out workloads with at most 2% regret.

Sanity

Surface outcomes: n=12, min=0.39788, max=1.0, distinct=8. Session costs: n=4, min=1.1702, max=1.3566 H20-hours, distinct=4. Scheduler-record counts: n=4, min=37,001, max=41,348, distinct=4. All counters and costs were non-negative; all ratios were in [0,1]; request hashes matched; all 12 runs were uncensored; the controller and four sessions completed; and config outcomes were not all identical. No red flags were found.

Machine-readable summary: runs/active-intervention-v0/trace13-results.json. Raw immutable root: /home/admin/cpfs/wjh/active-intervention-prospective-20260715.

5.0 KiB Raw Blame History