aituner/docs/intervention-response-v3-results-20260714.md

Phase-aware telemetry intervention-response v3 results

Date: 2026-07-14 (Asia/Singapore).

Decision: STOP_NO_INCREMENTAL_TUNING_SIGNAL.

Claim tested

After the replay has developed queue, batch, and KV state, does increasing MNS from 16 to 64 create telemetry responses that exceed workload-repeat noise, and does any such response identify whether the action repairs the full-run SLO better than external prefix outcomes alone?

The first clause passed. The second clause failed. Long-window telemetry is mechanistically informative, but this pilot does not support its necessity for tuning this action.

Setup

• dash0 GPU 0-3: four NVIDIA H20 GPUs; Qwen3-30B-A3B; patched vLLM 0.24.1.dev3+opprof; TP=4.
• Action: MNS 16 -> 64 with exact request, arrival, and input-length hashes.
• Trace: chat_w20260312_1000, replay-time scale 0.5, 300 seconds.
• Loads: 1.5 and 2.125 requests/s/GPU; three disjoint bands; endpoint order A/B, B/A, A/B; load order low/mid, mid/low, low/mid.
• Five cumulative checkpoints: 30, 75, 150, 225, and 300 seconds; four non-overlapping quarter blocks.
• Six fresh-server sessions, 12 measured runs, six action pairs, and eight same-config repeat pairs.

The prior three-load attempt is not part of the result. Its MNS=16 workload at 3.125 requests/s/GPU could not drain by the 450-second client timeout and produced no high-load result. V3 reran every retained point from scratch.

End-to-end outcome

All three low-load pairs remained feasible (true->true, label 0). All three pressure-load pairs changed from infeasible to feasible (false->true, label 1). MNS=16 pressure pass rates were 0.5604, 0.3145, and 0.2635; all three MNS=64 pressure runs reached 1.0. This yielded a balanced 3/3 action label set.

Mechanism result

No telemetry feature passed the action-versus-repeat gate at 10% or 25%. At 50%, graph padding first passed. At both 75% and 100%, graph padding and queue waiting passed, satisfying the requirement for two features at the same pair of adjacent checkpoints and with consistent directions in both load regimes.

Feature	Direction for MNS 16->64	75% effect/repeat median	75% above repeat p95	100% effect/repeat median	100% above repeat p95
queue_waiting_mean	lower	1346.22	3/6	898.70	3/6
graph_padding_fraction	higher	5.00	4/6	5.74	5/6

The very large queue effect/median-repeat ratios should not be read alone: its repeat p95 was much larger than its repeat median, so the independent p95 coverage criterion remained binding. Full-window queue-waiting deltas were -0.19 to -0.32 at low load and -20.99 to -32.16 at pressure load. Graph padding increased in every pair, by 0.00133-0.00217 at low load and 0.00705-0.00873 at pressure load.

The mechanism is therefore a real tradeoff: larger MNS reduces queueing, especially under pressure, while increasing CUDA-graph padding.

Tuning-signal result

Leave-one-repetition-out balanced accuracy was evaluated against the best external prefix-outcome feature at every predeclared checkpoint.

Replay phase	Best external BA	Best telemetry BA	Incremental telemetry gate
10%	0.833	0.833	fail
25%	1.000	1.000	fail
50%	0.833	1.000	pass at this checkpoint only
75%	1.000	1.000	fail
100%	1.000	1.000	fail

At 50%, several telemetry features exceeded the external baseline by at least 0.15, including the phase-stable mechanism feature graph_padding_fraction. The advantage did not hold at either adjacent checkpoint. Consequently no feature passed the frozen two-adjacent-phase requirement.

The important ordering is that external TTFT already classified the action perfectly at 25%, whereas the robust two-feature mechanism response did not emerge until 75%-100%. In this setup telemetry explains why MNS helps, but it does not provide earlier or more reliable action selection than direct prefix outcomes.

Research conclusion

The 5/10-second negative result was indeed too narrow. It only ruled out an ultra-early telemetry verifier; it did not rule out engine-state information. The 300-second pilot finds a clear and reproducible queueing-versus-padding response.

However, this does not rescue the direct telemetry-guided tuning claim. For this action and workload, the external signal is already as good or better before the telemetry mechanism becomes stable. The project should therefore not claim that engine instrumentation is necessary for tuning on this evidence, and should not open an E2E policy test from this pilot.

The narrower simulator-residual route remains logically open: telemetry may explain why a simulator misranks real configurations even when direct online outcomes can guide a tuner. That is a different hypothesis and was not tested here.

Change and verification

Change: absolute 5/10-second prefixes were replaced by phase-aware 30/75/150/ 225/300-second analysis; SLO early stop was disabled; full Layer-1 coverage, hash, request-accounting, controller, and stream/footer gates were added. The mechanism gate was corrected to require two features at the same adjacent phase pair.

Expected effect: distinguish “telemetry has not developed yet” from “telemetry does not identify or improve the action decision.”

Verification: five local analysis/controller test suites passed; remote manifest preflight and command dry-run passed; six serialized sessions passed all stream invariants; the analyzer was rerun and produced byte-identical output.

Result: mechanism evidence passed; incremental tuning evidence failed. Audit SHA256: 45f6f248712f9cbd3ed72036837ff6dc5b5c14c0f2eb6ba5cd5daceb1aa4ddb7.

Remaining risk: this is a development pilot with one model, one TP, one action, two retained loads, three request bands, and six action labels. It is adequate to reject opening the next direct-policy stage, not to establish a universal negative claim about telemetry.

Research-validity audit

Check	Verdict	Evidence / boundary
Real system and E2E outcome	PASS	Real H20/vLLM replay; full SLO outcome accompanies mechanism telemetry.
Matched action baseline	PASS	Exact request/arrival/length hashes for MNS 16 and 64; external prefix outcome is the decision baseline.
Repeats and order effects	PASS for pilot	Three disjoint bands; A/B, B/A, A/B endpoint order; counter-rotated load order.
Selective load removal	PASS with narrowed claim	The 3.125 load produced no result before any action comparison; the failure and cost are retained, and all kept points were freshly rerun.
Significance/generalization	NEEDS EVIDENCE for a paper claim	Only three bands, one model, one TP, one action, and six labels. This is explicitly a stage gate.
Calibration versus evaluation	NEEDS EVIDENCE for a positive policy claim	Frozen gates and leave-one-band-out folds reduce leakage, but a new workload/model hold-out is still required.
Platform/reproducibility	PASS	Commit, commands, manifest, controller state, platform fingerprint, raw remote paths, and audit hashes are recorded.

Data sanity

• Measured runs: n=12; elapsed 300.346-317.012 seconds; 12 distinct; pass rate 0.2635-1.0 with 4 distinct values; selected requests 1800-2550 with 2 distinct values.
• Sessions: n=6; 0.8413-0.8631 H20-hours; 6 distinct; Layer-1 records 58,465-64,776; 6 distinct. V3 cost was 5.0924 H20-hours; total including the invalid attempt was 6.4505, below the 8.0 cap.
• Labels: n=6; min/max 0/1; 2 distinct. Action pairs were 6 and repeat pairs were 8 at every checkpoint.
• Coverage-gap observations: n=60; start gaps 0.0427-0.1247 seconds; end gaps 0.00014-0.1705; maximum internal gaps 0.1695-0.6528, all below one second.
• Checked invariants: exact pair hashes and counts, all runs uncensored, full request accounting, monotonic admitted/completed coverage, monotonic Layer-1 timestamps, nonnegative counters, bounded ratios, non-identical per-config states, contiguous step indices, zero drops, footer/sidecar agreement, no controller failures, all sessions complete, GPU idle after completion. No red flags were found.