aituner/docs/action-aware-constraint-pilot-v2-results-20260714.md

Action-aware constraint pilot v2 results

Date: 2026-07-14 (Asia/Singapore).

Decision: STOP_WORKLOAD_NOT_CROSSED.

The pilot produced one valid positive regime and one invalid-for-effect-size regime. It supports continuing a narrower action-response investigation, but it does not justify an end-to-end telemetry-guided tuner claim or new engine instrumentation yet.

Question tested

Given only a completed source run, can existing engine telemetry distinguish which of two one-knob interventions will improve SLO-goodput more?

The frozen score counted scheduler steps with backlog where either MNS or MBBT was exclusively at its configured limit. It made two pre-intervention predictions on the same workload and offered load:

• Regime A: source (MNS=16, MBBT=8192) predicts increasing MNS to 64 over increasing MBBT to 16384.
• Regime B: source (MNS=64, MBBT=2048) predicts increasing MBBT to 8192 over increasing MNS to 128.

The primary outcome was 300-second SLO-goodput. A predicted action had to beat the alternative on every paired request band by at least 10% of that band's source goodput. Telemetry direction also had to remain stable at 25%, 50%, 75%, and 100% of the replay.

Setup

• Host: dash0, GPU 0-3 used exclusively; four NVIDIA H20 GPUs; TP=4. GPU 4-7 remained idle to avoid co-location effects.
• Model: Qwen3-30B-A3B BF16 at /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B.
• Runtime: patched vLLM 0.24.1.dev3+g668cfb7e2, source commit 4b253fd, with OpProf Layer-1 telemetry.
• Workload: chat_w20260312_1000, 2.125 requests/s/GPU, 300-second arrival window, 128 output tokens, three disjoint request bands.
• Five fresh-server configurations, three measured runs each, counter-rotated repetition order, 16-request warm-up, and a 510-request/60-second burn-in. SLO early stopping was disabled.
• Exact request-id, arrival-order, and input-length hashes matched for every paired comparison.

The authoritative run root is /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714. The final audit is pilot-audit-final.json, SHA256 7ebe080fcc4970bef423bc587253d157e75aed1ea8b410bd37770c17708135ab.

End-to-end result

Regime A: strong MNS constraint

Rep	Source 16/8192	MBBT action 16/16384	MNS action 64/8192	MNS-only source steps	MBBT-only source steps	(MNS action - MBBT action) / source
1	4.710	7.687	8.500	79.85%	0.038%	17.27%
2	2.803	4.150	8.500	94.61%	0.005%	155.17%
3	2.227	3.943	8.500	79.87%	0.005%	204.64%

Units are SLO-goodput requests/s except the percentage columns. The source prediction was stable at every phase checkpoint and correct in all three paired bands. The predicted MNS action cleared the frozen 10% material-margin gate in all three bands.

This is a real but limited positive result. The source was an extreme case: MNS was full on nearly every scheduler step that retained backlog, TTFT p50 was 1.64-5.61 seconds, KV usage remained below 2.45%, and no preemption occurred. An expert or a simple rule could identify this case without a learned tuner.

Regime B: MBBT direction at an outcome ceiling

| Rep | Source 64/2048 | MNS action 128/2048 | MBBT action 64/8192 | MBBT-only source steps | (MBBT action - MNS action) / source | |---:|---:|---:|---:|---:| | 1 | 8.423 | 8.420 | 8.500 | 10.90% | 0.950% | | 2 | 8.447 | 8.500 | 8.500 | 12.03% | 0% | | 3 | 8.500 | 8.497 | 8.500 | 8.79% | 0.039% |

The telemetry direction was stable in all phase checkpoints. The predicted action won twice and tied once, while the wrong MNS action left the MBBT-only state intact. However, the source already delivered 99.10-100% of the offered 8.5 requests/s. Even a perfect action could not reach the preregistered 10% margin. Regime B therefore does not test material weak-signal value; it is a workload-selection failure, not evidence that telemetry does or does not help near a decision boundary.

The missing preflight condition is mathematical. For an effect threshold delta and offered goodput ceiling G, the source must satisfy source <= G / (1 + delta). Here G=8.5 and delta=0.10, so any source above 7.727 requests/s cannot possibly pass before either target is measured.

Why the exclusive-limit rule is incomplete

The alternative MBBT action improved Regime A by 48.0%, 63.2%, and 77.1% over the source even though MBBT was almost never the exclusive backlog constraint. This rules out the binary interpretation "a knob that is not exclusively at its cap cannot help."

Existing richer telemetry provides a plausible mechanism:

Rep	Split-prefill requests, source -> MBBT action	Prefill steps, source -> MBBT action	Prefill requests/step, source -> MBBT action	Prefix-hit rate, source -> MBBT action
1	41 -> 2	2324 -> 2022	1.071 -> 1.249	13.851% -> 13.747%
2	7 -> 0	2410 -> 2291	1.012 -> 1.076	13.078% -> 12.988%
3	12 -> 1	2377 -> 2270	1.018 -> 1.096	13.613% -> 13.604%

Increasing MBBT allows more prefill work to be packed into one iteration and nearly eliminates split prefills. Under MNS=16, this can reduce the number of iterations for which long prompts occupy scarce running slots. Prefix-cache hit rates differ by at most 0.104 percentage points, and exact workload hashes match, so neither explains the gain. Step-duration p99 also remains similar; one 1.127-second decode-step outlier appears in a_mbbt/rep2, but the same action direction occurs in all three bands.

This is a mechanism-consistent explanation, not a completed causal decomposition. MBBT simultaneously changes total per-iteration token budget, per-request chunk size, and multi-request packing. Instrumentation observes their joint response but cannot separate those effects without another intervention.

Instrumentation decision

Do not add a new engine patch for this mechanism yet. The existing OpProf stream already records submit/complete timestamps, prefill/decode composition, chunked-prefill categories, prefix hits, queues, KV usage, and CUDA graph mode. Those fields are sufficient to identify the interaction missed by the initial exclusive-limit rule.

The next narrow mechanism ablation is available in the current vLLM runtime:

1. (MNS=16, MBBT=8192, long-prefill-threshold=0) is the current source.
2. (MNS=16, MBBT=16384, long-prefill-threshold=8192) keeps individual long chunks at 8192 while increasing total packing budget.
3. (MNS=16, MBBT=16384, long-prefill-threshold=0) is the current MBBT action.

The runtime exposes --long-prefill-token-threshold; with a threshold of 8192, the second arm separates total packing headroom from the larger per-request chunk allowed by the third arm. A formal test must rerun all three arms fresh with counter-rotated order rather than reuse today's endpoints.

Correct tuning-research route

The pilot does not support turning the frozen equality checks into a larger rule tree. The supported route is intervention-calibrated, action-conditioned system identification:

source event sequence + normalized config delta
    -> predicted distribution of Delta SLO-goodput and evaluation cost
    -> uncertainty-aware next-config selection

The policy input should retain continuous distributions and phase evolution: queue/running residency, MNS and token slack, prefill/decode composition, partial-prefill occupancy, step time, KV state, and graph behavior. Human bottleneck labels and hand-authored if queue then increase MNS mappings are not policy inputs. Mechanism summaries remain audit and interpretation tools; the action response is learned from paired real interventions.

The harness has a narrower, non-heuristic role:

• define legal configurations and exact paired workloads;
• reject source points without outcome headroom before a full sweep;
• randomize/counter-rotate execution order and preserve failures/cost;
• validate stream coverage, hashes, request accounting, and censoring;
• expose target outcomes only after a source-only prediction is frozen;
• evaluate fixed-budget regret and H20-hours, not explanation quality alone.

The next tuning experiment should use non-extreme, non-ceiling source points and a local two-dimensional MNS/MBBT neighborhood. A short run may screen load only; every inferential telemetry and outcome result remains a 300-second run. At least one held-out workload must be reserved before choosing features or thresholds.

Primary evaluation is H20-hours/trials to reach 95% of the real local oracle and cost-normalized regret AUC. Required baselines are random search, config/outcome-only sequential search, the current rule heuristic, and the same action-response model with telemetry removed. Action-ranking accuracy is supporting evidence only.

What this pilot establishes and does not establish

Established:

• Long-window engine state can make a correct, phase-stable action-family prediction in an extreme MNS-constrained regime.
• A naive queue > 0 -> increase MNS rule would choose an ineffective action in Regime B; action-conditioned state distinguishes the mechanism direction, although the measured effect is immaterial at the selected load.
• Binary exclusive-cap attribution misses a substantial MNS/MBBT interaction; existing chunk/step telemetry reveals a plausible explanation.
• Source outcome headroom must be an explicit experiment admission gate.

Not established:

• telemetry improves an end-to-end tuner over an outcome-only baseline;
• weak or mixed constraints can be ranked with material gain;
• the response transfers across workloads, models, TP, or hardware;
• new engine instrumentation is necessary;
• the chunking/packing breakdown is causal rather than mechanism-consistent.

Change and verification

Reproduction:

python3 runs/action-aware-v0/test_pilot.py

python3 runs/action-aware-v0/analyze_pilot.py \
  --run-root /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/runs/pilot \
  --manifest runs/action-aware-v0/pilot-manifest-v2.json \
  --output /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/pilot-audit-final.json

The fresh GPU run used AITuner commit c5ab073; asynchronous coverage was corrected in 3facb18; reproducible mechanism summaries were added in 2af22db. The raw run is unchanged across those analyzer-only commits.

Change: added a crossed real-intervention controller and audit, fixed the burn-in result gate, corrected asynchronous per-step coverage accounting, and added reproducible step/chunk/prefix mechanism summaries.

Expected effect: distinguish descriptive telemetry from source-only action predictions that survive paired real interventions.

Verification: local and remote action-aware test suites pass; all five sessions completed; all stream/footer and request-accounting invariants pass; the final analyzer was run twice and produced byte-identical output.

Result: Regime A passes; Regime B is invalid for the frozen effect-size test; the global decision is STOP_WORKLOAD_NOT_CROSSED.

Remaining risk: one model, one TP, one trace family, three bands, two action families, and deliberately constructed endpoints are development evidence only. The strong positive regime is too obvious to support a paper claim.

Data sanity

• Measured runs: n=15; elapsed 300.610-317.350 seconds; 15 distinct. Pass rate min/max 0.2620/1.0 with 11 distinct values; SLO-goodput min/max 2.2267/8.5 requests/s with 11 distinct values.
• Telemetry intervals: n=15; records min/max 13,621/23,711; 15 distinct. Start gaps min/max 0.0412/0.1227 seconds; end gaps 0.00053/0.0649; uncovered internal gaps 0/0.3231. One submit gap reached 1.1190 seconds but was fully covered by a 1.1269-second recorded execution; contiguous indices and zero drops were preserved.
• Sessions: n=5; cost min/max 1.1691/1.2730 H20-hours; 5 distinct. V2 cost was 6.0862 H20-hours; V0/V1/V2 total was 6.7906, below the 8.0 cap.
• Regime-A split-prefill observations: n=6; min/max 0/41 requests; 6 distinct. Prefix-hit rates: n=6; min/max 0.12988/0.13851; 6 distinct.
• Checked invariants: non-negative counters and durations; ratios in [0,1]; exact request, arrival, and length hashes; 2550/2550 request accounting per measured run; uncensored outcomes; outcomes across configurations not all identical; five complete streams; monotonic timestamps; contiguous step indices; zero drops; footer/sidecar agreement; chunk-token accounting; bounded prefix hits; no OOM, controller error, or residual GPU allocation. No unresolved red flag remains. The three identical shared goodputs are reported as the offered-load ceiling, not treated as independent performance variation.