From 8c930ba3a1f6e7b6481ff0a310e45a30c877988e Mon Sep 17 00:00:00 2001 From: Gahow Wang Date: Tue, 14 Jul 2026 22:34:26 +0800 Subject: [PATCH] Report action-aware constraint pilot results --- ...re-constraint-pilot-v2-results-20260714.md | 260 ++++++++++++++++++ 1 file changed, 260 insertions(+) create mode 100644 docs/action-aware-constraint-pilot-v2-results-20260714.md diff --git a/docs/action-aware-constraint-pilot-v2-results-20260714.md b/docs/action-aware-constraint-pilot-v2-results-20260714.md new file mode 100644 index 0000000..b0f1ed0 --- /dev/null +++ b/docs/action-aware-constraint-pilot-v2-results-20260714.md @@ -0,0 +1,260 @@ +# Action-aware constraint pilot v2 results + +Date: 2026-07-14 (Asia/Singapore). + +Decision: **`STOP_WORKLOAD_NOT_CROSSED`**. + +The pilot produced one valid positive regime and one invalid-for-effect-size +regime. It supports continuing a narrower action-response investigation, but +it does not justify an end-to-end telemetry-guided tuner claim or new engine +instrumentation yet. + +## Question tested + +Given only a completed source run, can existing engine telemetry distinguish +which of two one-knob interventions will improve SLO-goodput more? + +The frozen score counted scheduler steps with backlog where either MNS or MBBT +was exclusively at its configured limit. It made two pre-intervention +predictions on the same workload and offered load: + +- Regime A: source `(MNS=16, MBBT=8192)` predicts increasing MNS to 64 over + increasing MBBT to 16384. +- Regime B: source `(MNS=64, MBBT=2048)` predicts increasing MBBT to 8192 over + increasing MNS to 128. + +The primary outcome was 300-second SLO-goodput. A predicted action had to beat +the alternative on every paired request band by at least 10% of that band's +source goodput. Telemetry direction also had to remain stable at 25%, 50%, +75%, and 100% of the replay. + +## Setup + +- Host: `dash0`, GPU 0-3 used exclusively; four NVIDIA H20 GPUs; TP=4. GPU + 4-7 remained idle to avoid co-location effects. +- Model: Qwen3-30B-A3B BF16 at + `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`. +- Runtime: patched vLLM `0.24.1.dev3+g668cfb7e2`, source commit `4b253fd`, with + OpProf Layer-1 telemetry. +- Workload: `chat_w20260312_1000`, 2.125 requests/s/GPU, 300-second arrival + window, 128 output tokens, three disjoint request bands. +- Five fresh-server configurations, three measured runs each, counter-rotated + repetition order, 16-request warm-up, and a 510-request/60-second burn-in. + SLO early stopping was disabled. +- Exact request-id, arrival-order, and input-length hashes matched for every + paired comparison. + +The authoritative run root is +`/home/admin/cpfs/wjh/action-aware-constraint-v2-20260714`. The final audit is +`pilot-audit-final.json`, SHA256 +`7ebe080fcc4970bef423bc587253d157e75aed1ea8b410bd37770c17708135ab`. + +## End-to-end result + +### Regime A: strong MNS constraint + +| Rep | Source 16/8192 | MBBT action 16/16384 | MNS action 64/8192 | MNS-only source steps | MBBT-only source steps | `(MNS action - MBBT action) / source` | +|---:|---:|---:|---:|---:|---:|---:| +| 1 | 4.710 | 7.687 | 8.500 | 79.85% | 0.038% | 17.27% | +| 2 | 2.803 | 4.150 | 8.500 | 94.61% | 0.005% | 155.17% | +| 3 | 2.227 | 3.943 | 8.500 | 79.87% | 0.005% | 204.64% | + +Units are SLO-goodput requests/s except the percentage columns. The source +prediction was stable at every phase checkpoint and correct in all three +paired bands. The predicted MNS action cleared the frozen 10% material-margin +gate in all three bands. + +This is a real but limited positive result. The source was an extreme case: +MNS was full on nearly every scheduler step that retained backlog, TTFT p50 was +1.64-5.61 seconds, KV usage remained below 2.45%, and no preemption occurred. +An expert or a simple rule could identify this case without a learned tuner. + +### Regime B: MBBT direction at an outcome ceiling + +| Rep | Source 64/2048 | MNS action 128/2048 | MBBT action 64/8192 | MBBT-only source steps | `(MBBT action - MNS action) / source` | +|---:|---:|---:|---:|---:| +| 1 | 8.423 | 8.420 | 8.500 | 10.90% | 0.950% | +| 2 | 8.447 | 8.500 | 8.500 | 12.03% | 0% | +| 3 | 8.500 | 8.497 | 8.500 | 8.79% | 0.039% | + +The telemetry direction was stable in all phase checkpoints. The predicted +action won twice and tied once, while the wrong MNS action left the MBBT-only +state intact. However, the source already delivered 99.10-100% of the offered +8.5 requests/s. Even a perfect action could not reach the preregistered 10% +margin. Regime B therefore does not test material weak-signal value; it is a +workload-selection failure, not evidence that telemetry does or does not help +near a decision boundary. + +The missing preflight condition is mathematical. For an effect threshold +`delta` and offered goodput ceiling `G`, the source must satisfy +`source <= G / (1 + delta)`. Here `G=8.5` and `delta=0.10`, so any source above +7.727 requests/s cannot possibly pass before either target is measured. + +## Why the exclusive-limit rule is incomplete + +The alternative MBBT action improved Regime A by 48.0%, 63.2%, and 77.1% over +the source even though MBBT was almost never the exclusive backlog constraint. +This rules out the binary interpretation "a knob that is not exclusively at +its cap cannot help." + +Existing richer telemetry provides a plausible mechanism: + +| Rep | Split-prefill requests, source -> MBBT action | Prefill steps, source -> MBBT action | Prefill requests/step, source -> MBBT action | Prefix-hit rate, source -> MBBT action | +|---:|---:|---:|---:|---:| +| 1 | 41 -> 2 | 2324 -> 2022 | 1.071 -> 1.249 | 13.851% -> 13.747% | +| 2 | 7 -> 0 | 2410 -> 2291 | 1.012 -> 1.076 | 13.078% -> 12.988% | +| 3 | 12 -> 1 | 2377 -> 2270 | 1.018 -> 1.096 | 13.613% -> 13.604% | + +Increasing MBBT allows more prefill work to be packed into one iteration and +nearly eliminates split prefills. Under MNS=16, this can reduce the number of +iterations for which long prompts occupy scarce running slots. Prefix-cache +hit rates differ by at most 0.104 percentage points, and exact workload hashes +match, so neither explains the gain. Step-duration p99 also remains similar; +one 1.127-second decode-step outlier appears in `a_mbbt/rep2`, but the same +action direction occurs in all three bands. + +This is a mechanism-consistent explanation, not a completed causal +decomposition. MBBT simultaneously changes total per-iteration token budget, +per-request chunk size, and multi-request packing. Instrumentation observes +their joint response but cannot separate those effects without another +intervention. + +## Instrumentation decision + +Do **not** add a new engine patch for this mechanism yet. The existing OpProf +stream already records submit/complete timestamps, prefill/decode composition, +chunked-prefill categories, prefix hits, queues, KV usage, and CUDA graph mode. +Those fields are sufficient to identify the interaction missed by the initial +exclusive-limit rule. + +The next narrow mechanism ablation is available in the current vLLM runtime: + +1. `(MNS=16, MBBT=8192, long-prefill-threshold=0)` is the current source. +2. `(MNS=16, MBBT=16384, long-prefill-threshold=8192)` keeps individual long + chunks at 8192 while increasing total packing budget. +3. `(MNS=16, MBBT=16384, long-prefill-threshold=0)` is the current MBBT action. + +The runtime exposes `--long-prefill-token-threshold`; with a threshold of 8192, +the second arm separates total packing headroom from the larger per-request +chunk allowed by the third arm. A formal test must rerun all three arms fresh +with counter-rotated order rather than reuse today's endpoints. + +## Correct tuning-research route + +The pilot does not support turning the frozen equality checks into a larger +rule tree. The supported route is **intervention-calibrated, action-conditioned +system identification**: + +```text +source event sequence + normalized config delta + -> predicted distribution of Delta SLO-goodput and evaluation cost + -> uncertainty-aware next-config selection +``` + +The policy input should retain continuous distributions and phase evolution: +queue/running residency, MNS and token slack, prefill/decode composition, +partial-prefill occupancy, step time, KV state, and graph behavior. Human +bottleneck labels and hand-authored `if queue then increase MNS` mappings are +not policy inputs. Mechanism summaries remain audit and interpretation tools; +the action response is learned from paired real interventions. + +The harness has a narrower, non-heuristic role: + +- define legal configurations and exact paired workloads; +- reject source points without outcome headroom before a full sweep; +- randomize/counter-rotate execution order and preserve failures/cost; +- validate stream coverage, hashes, request accounting, and censoring; +- expose target outcomes only after a source-only prediction is frozen; +- evaluate fixed-budget regret and H20-hours, not explanation quality alone. + +The next tuning experiment should use non-extreme, non-ceiling source points +and a local two-dimensional MNS/MBBT neighborhood. A short run may screen load +only; every inferential telemetry and outcome result remains a 300-second run. +At least one held-out workload must be reserved before choosing features or +thresholds. + +Primary evaluation is H20-hours/trials to reach 95% of the real local oracle +and cost-normalized regret AUC. Required baselines are random search, +config/outcome-only sequential search, the current rule heuristic, and the +same action-response model with telemetry removed. Action-ranking accuracy is +supporting evidence only. + +## What this pilot establishes and does not establish + +Established: + +- Long-window engine state can make a correct, phase-stable action-family + prediction in an extreme MNS-constrained regime. +- A naive `queue > 0 -> increase MNS` rule would choose an ineffective action + in Regime B; action-conditioned state distinguishes the mechanism direction, + although the measured effect is immaterial at the selected load. +- Binary exclusive-cap attribution misses a substantial MNS/MBBT interaction; + existing chunk/step telemetry reveals a plausible explanation. +- Source outcome headroom must be an explicit experiment admission gate. + +Not established: + +- telemetry improves an end-to-end tuner over an outcome-only baseline; +- weak or mixed constraints can be ranked with material gain; +- the response transfers across workloads, models, TP, or hardware; +- new engine instrumentation is necessary; +- the chunking/packing breakdown is causal rather than mechanism-consistent. + +## Change and verification + +Reproduction: + +```bash +python3 runs/action-aware-v0/test_pilot.py + +python3 runs/action-aware-v0/analyze_pilot.py \ + --run-root /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/runs/pilot \ + --manifest runs/action-aware-v0/pilot-manifest-v2.json \ + --output /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/pilot-audit-final.json +``` + +The fresh GPU run used AITuner commit `c5ab073`; asynchronous coverage was +corrected in `3facb18`; reproducible mechanism summaries were added in +`2af22db`. The raw run is unchanged across those analyzer-only commits. + +Change: added a crossed real-intervention controller and audit, fixed the +burn-in result gate, corrected asynchronous per-step coverage accounting, and +added reproducible step/chunk/prefix mechanism summaries. + +Expected effect: distinguish descriptive telemetry from source-only action +predictions that survive paired real interventions. + +Verification: local and remote action-aware test suites pass; all five sessions +completed; all stream/footer and request-accounting invariants pass; the final +analyzer was run twice and produced byte-identical output. + +Result: Regime A passes; Regime B is invalid for the frozen effect-size test; +the global decision is `STOP_WORKLOAD_NOT_CROSSED`. + +Remaining risk: one model, one TP, one trace family, three bands, two action +families, and deliberately constructed endpoints are development evidence +only. The strong positive regime is too obvious to support a paper claim. + +## Data sanity + +- Measured runs: n=15; elapsed 300.610-317.350 seconds; 15 distinct. Pass + rate min/max 0.2620/1.0 with 11 distinct values; SLO-goodput min/max + 2.2267/8.5 requests/s with 11 distinct values. +- Telemetry intervals: n=15; records min/max 13,621/23,711; 15 distinct. + Start gaps min/max 0.0412/0.1227 seconds; end gaps 0.00053/0.0649; uncovered + internal gaps 0/0.3231. One submit gap reached 1.1190 seconds but was fully + covered by a 1.1269-second recorded execution; contiguous indices and zero + drops were preserved. +- Sessions: n=5; cost min/max 1.1691/1.2730 H20-hours; 5 distinct. V2 cost + was 6.0862 H20-hours; V0/V1/V2 total was 6.7906, below the 8.0 cap. +- Regime-A split-prefill observations: n=6; min/max 0/41 requests; 6 distinct. + Prefix-hit rates: n=6; min/max 0.12988/0.13851; 6 distinct. +- Checked invariants: non-negative counters and durations; ratios in `[0,1]`; + exact request, arrival, and length hashes; 2550/2550 request accounting per + measured run; uncensored outcomes; outcomes across configurations not all + identical; + five complete streams; monotonic timestamps; contiguous step indices; zero + drops; footer/sidecar agreement; chunk-token accounting; bounded prefix hits; + no OOM, controller error, or residual GPU allocation. No unresolved red + flag remains. The three identical shared goodputs are reported as the + offered-load ceiling, not treated as independent performance variation.