Report action-aware constraint pilot results

docs/action-aware-constraint-pilot-v2-results-20260714.md

@@ -0,0 +1,260 @@
+# Action-aware constraint pilot v2 results
+
+Date: 2026-07-14 (Asia/Singapore).
+
+Decision: **`STOP_WORKLOAD_NOT_CROSSED`**.
+
+The pilot produced one valid positive regime and one invalid-for-effect-size
+regime.  It supports continuing a narrower action-response investigation, but
+it does not justify an end-to-end telemetry-guided tuner claim or new engine
+instrumentation yet.
+
+## Question tested
+
+Given only a completed source run, can existing engine telemetry distinguish
+which of two one-knob interventions will improve SLO-goodput more?
+
+The frozen score counted scheduler steps with backlog where either MNS or MBBT
+was exclusively at its configured limit.  It made two pre-intervention
+predictions on the same workload and offered load:
+
+- Regime A: source `(MNS=16, MBBT=8192)` predicts increasing MNS to 64 over
+  increasing MBBT to 16384.
+- Regime B: source `(MNS=64, MBBT=2048)` predicts increasing MBBT to 8192 over
+  increasing MNS to 128.
+
+The primary outcome was 300-second SLO-goodput.  A predicted action had to beat
+the alternative on every paired request band by at least 10% of that band's
+source goodput.  Telemetry direction also had to remain stable at 25%, 50%,
+75%, and 100% of the replay.
+
+## Setup
+
+- Host: `dash0`, GPU 0-3 used exclusively; four NVIDIA H20 GPUs; TP=4.  GPU
+  4-7 remained idle to avoid co-location effects.
+- Model: Qwen3-30B-A3B BF16 at
+  `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`.
+- Runtime: patched vLLM `0.24.1.dev3+g668cfb7e2`, source commit `4b253fd`, with
+  OpProf Layer-1 telemetry.
+- Workload: `chat_w20260312_1000`, 2.125 requests/s/GPU, 300-second arrival
+  window, 128 output tokens, three disjoint request bands.
+- Five fresh-server configurations, three measured runs each, counter-rotated
+  repetition order, 16-request warm-up, and a 510-request/60-second burn-in.
+  SLO early stopping was disabled.
+- Exact request-id, arrival-order, and input-length hashes matched for every
+  paired comparison.
+
+The authoritative run root is
+`/home/admin/cpfs/wjh/action-aware-constraint-v2-20260714`.  The final audit is
+`pilot-audit-final.json`, SHA256
+`7ebe080fcc4970bef423bc587253d157e75aed1ea8b410bd37770c17708135ab`.
+
+## End-to-end result
+
+### Regime A: strong MNS constraint
+
+| Rep | Source 16/8192 | MBBT action 16/16384 | MNS action 64/8192 | MNS-only source steps | MBBT-only source steps | `(MNS action - MBBT action) / source` |
+|---:|---:|---:|---:|---:|---:|---:|
+| 1 | 4.710 | 7.687 | 8.500 | 79.85% | 0.038% | 17.27% |
+| 2 | 2.803 | 4.150 | 8.500 | 94.61% | 0.005% | 155.17% |
+| 3 | 2.227 | 3.943 | 8.500 | 79.87% | 0.005% | 204.64% |
+
+Units are SLO-goodput requests/s except the percentage columns.  The source
+prediction was stable at every phase checkpoint and correct in all three
+paired bands.  The predicted MNS action cleared the frozen 10% material-margin
+gate in all three bands.
+
+This is a real but limited positive result.  The source was an extreme case:
+MNS was full on nearly every scheduler step that retained backlog, TTFT p50 was
+1.64-5.61 seconds, KV usage remained below 2.45%, and no preemption occurred.
+An expert or a simple rule could identify this case without a learned tuner.
+
+### Regime B: MBBT direction at an outcome ceiling
+
+| Rep | Source 64/2048 | MNS action 128/2048 | MBBT action 64/8192 | MBBT-only source steps | `(MBBT action - MNS action) / source` |
+|---:|---:|---:|---:|---:|
+| 1 | 8.423 | 8.420 | 8.500 | 10.90% | 0.950% |
+| 2 | 8.447 | 8.500 | 8.500 | 12.03% | 0% |
+| 3 | 8.500 | 8.497 | 8.500 | 8.79% | 0.039% |
+
+The telemetry direction was stable in all phase checkpoints.  The predicted
+action won twice and tied once, while the wrong MNS action left the MBBT-only
+state intact.  However, the source already delivered 99.10-100% of the offered
+8.5 requests/s.  Even a perfect action could not reach the preregistered 10%
+margin.  Regime B therefore does not test material weak-signal value; it is a
+workload-selection failure, not evidence that telemetry does or does not help
+near a decision boundary.
+
+The missing preflight condition is mathematical.  For an effect threshold
+`delta` and offered goodput ceiling `G`, the source must satisfy
+`source <= G / (1 + delta)`.  Here `G=8.5` and `delta=0.10`, so any source above
+7.727 requests/s cannot possibly pass before either target is measured.
+
+## Why the exclusive-limit rule is incomplete
+
+The alternative MBBT action improved Regime A by 48.0%, 63.2%, and 77.1% over
+the source even though MBBT was almost never the exclusive backlog constraint.
+This rules out the binary interpretation "a knob that is not exclusively at
+its cap cannot help."
+
+Existing richer telemetry provides a plausible mechanism:
+
+| Rep | Split-prefill requests, source -> MBBT action | Prefill steps, source -> MBBT action | Prefill requests/step, source -> MBBT action | Prefix-hit rate, source -> MBBT action |
+|---:|---:|---:|---:|---:|
+| 1 | 41 -> 2 | 2324 -> 2022 | 1.071 -> 1.249 | 13.851% -> 13.747% |
+| 2 | 7 -> 0 | 2410 -> 2291 | 1.012 -> 1.076 | 13.078% -> 12.988% |
+| 3 | 12 -> 1 | 2377 -> 2270 | 1.018 -> 1.096 | 13.613% -> 13.604% |
+
+Increasing MBBT allows more prefill work to be packed into one iteration and
+nearly eliminates split prefills.  Under MNS=16, this can reduce the number of
+iterations for which long prompts occupy scarce running slots.  Prefix-cache
+hit rates differ by at most 0.104 percentage points, and exact workload hashes
+match, so neither explains the gain.  Step-duration p99 also remains similar;
+one 1.127-second decode-step outlier appears in `a_mbbt/rep2`, but the same
+action direction occurs in all three bands.
+
+This is a mechanism-consistent explanation, not a completed causal
+decomposition.  MBBT simultaneously changes total per-iteration token budget,
+per-request chunk size, and multi-request packing.  Instrumentation observes
+their joint response but cannot separate those effects without another
+intervention.
+
+## Instrumentation decision
+
+Do **not** add a new engine patch for this mechanism yet.  The existing OpProf
+stream already records submit/complete timestamps, prefill/decode composition,
+chunked-prefill categories, prefix hits, queues, KV usage, and CUDA graph mode.
+Those fields are sufficient to identify the interaction missed by the initial
+exclusive-limit rule.
+
+The next narrow mechanism ablation is available in the current vLLM runtime:
+
+1. `(MNS=16, MBBT=8192, long-prefill-threshold=0)` is the current source.
+2. `(MNS=16, MBBT=16384, long-prefill-threshold=8192)` keeps individual long
+   chunks at 8192 while increasing total packing budget.
+3. `(MNS=16, MBBT=16384, long-prefill-threshold=0)` is the current MBBT action.
+
+The runtime exposes `--long-prefill-token-threshold`; with a threshold of 8192,
+the second arm separates total packing headroom from the larger per-request
+chunk allowed by the third arm.  A formal test must rerun all three arms fresh
+with counter-rotated order rather than reuse today's endpoints.
+
+## Correct tuning-research route
+
+The pilot does not support turning the frozen equality checks into a larger
+rule tree.  The supported route is **intervention-calibrated, action-conditioned
+system identification**:
+
+```text
+source event sequence + normalized config delta
+    -> predicted distribution of Delta SLO-goodput and evaluation cost
+    -> uncertainty-aware next-config selection
+```
+
+The policy input should retain continuous distributions and phase evolution:
+queue/running residency, MNS and token slack, prefill/decode composition,
+partial-prefill occupancy, step time, KV state, and graph behavior.  Human
+bottleneck labels and hand-authored `if queue then increase MNS` mappings are
+not policy inputs.  Mechanism summaries remain audit and interpretation tools;
+the action response is learned from paired real interventions.
+
+The harness has a narrower, non-heuristic role:
+
+- define legal configurations and exact paired workloads;
+- reject source points without outcome headroom before a full sweep;
+- randomize/counter-rotate execution order and preserve failures/cost;
+- validate stream coverage, hashes, request accounting, and censoring;
+- expose target outcomes only after a source-only prediction is frozen;
+- evaluate fixed-budget regret and H20-hours, not explanation quality alone.
+
+The next tuning experiment should use non-extreme, non-ceiling source points
+and a local two-dimensional MNS/MBBT neighborhood.  A short run may screen load
+only; every inferential telemetry and outcome result remains a 300-second run.
+At least one held-out workload must be reserved before choosing features or
+thresholds.
+
+Primary evaluation is H20-hours/trials to reach 95% of the real local oracle
+and cost-normalized regret AUC.  Required baselines are random search,
+config/outcome-only sequential search, the current rule heuristic, and the
+same action-response model with telemetry removed.  Action-ranking accuracy is
+supporting evidence only.
+
+## What this pilot establishes and does not establish
+
+Established:
+
+- Long-window engine state can make a correct, phase-stable action-family
+  prediction in an extreme MNS-constrained regime.
+- A naive `queue > 0 -> increase MNS` rule would choose an ineffective action
+  in Regime B; action-conditioned state distinguishes the mechanism direction,
+  although the measured effect is immaterial at the selected load.
+- Binary exclusive-cap attribution misses a substantial MNS/MBBT interaction;
+  existing chunk/step telemetry reveals a plausible explanation.
+- Source outcome headroom must be an explicit experiment admission gate.
+
+Not established:
+
+- telemetry improves an end-to-end tuner over an outcome-only baseline;
+- weak or mixed constraints can be ranked with material gain;
+- the response transfers across workloads, models, TP, or hardware;
+- new engine instrumentation is necessary;
+- the chunking/packing breakdown is causal rather than mechanism-consistent.
+
+## Change and verification
+
+Reproduction:
+
+```bash
+python3 runs/action-aware-v0/test_pilot.py
+
+python3 runs/action-aware-v0/analyze_pilot.py \
+  --run-root /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/runs/pilot \
+  --manifest runs/action-aware-v0/pilot-manifest-v2.json \
+  --output /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/pilot-audit-final.json
+```
+
+The fresh GPU run used AITuner commit `c5ab073`; asynchronous coverage was
+corrected in `3facb18`; reproducible mechanism summaries were added in
+`2af22db`.  The raw run is unchanged across those analyzer-only commits.
+
+Change: added a crossed real-intervention controller and audit, fixed the
+burn-in result gate, corrected asynchronous per-step coverage accounting, and
+added reproducible step/chunk/prefix mechanism summaries.
+
+Expected effect: distinguish descriptive telemetry from source-only action
+predictions that survive paired real interventions.
+
+Verification: local and remote action-aware test suites pass; all five sessions
+completed; all stream/footer and request-accounting invariants pass; the final
+analyzer was run twice and produced byte-identical output.
+
+Result: Regime A passes; Regime B is invalid for the frozen effect-size test;
+the global decision is `STOP_WORKLOAD_NOT_CROSSED`.
+
+Remaining risk: one model, one TP, one trace family, three bands, two action
+families, and deliberately constructed endpoints are development evidence
+only.  The strong positive regime is too obvious to support a paper claim.
+
+## Data sanity
+
+- Measured runs: n=15; elapsed 300.610-317.350 seconds; 15 distinct.  Pass
+  rate min/max 0.2620/1.0 with 11 distinct values; SLO-goodput min/max
+  2.2267/8.5 requests/s with 11 distinct values.
+- Telemetry intervals: n=15; records min/max 13,621/23,711; 15 distinct.
+  Start gaps min/max 0.0412/0.1227 seconds; end gaps 0.00053/0.0649; uncovered
+  internal gaps 0/0.3231.  One submit gap reached 1.1190 seconds but was fully
+  covered by a 1.1269-second recorded execution; contiguous indices and zero
+  drops were preserved.
+- Sessions: n=5; cost min/max 1.1691/1.2730 H20-hours; 5 distinct.  V2 cost
+  was 6.0862 H20-hours; V0/V1/V2 total was 6.7906, below the 8.0 cap.
+- Regime-A split-prefill observations: n=6; min/max 0/41 requests; 6 distinct.
+  Prefix-hit rates: n=6; min/max 0.12988/0.13851; 6 distinct.
+- Checked invariants: non-negative counters and durations; ratios in `[0,1]`;
+  exact request, arrival, and length hashes; 2550/2550 request accounting per
+  measured run; uncensored outcomes; outcomes across configurations not all
+  identical;
+  five complete streams; monotonic timestamps; contiguous step indices; zero
+  drops; footer/sidecar agreement; chunk-token accounting; bounded prefix hits;
+  no OOM, controller error, or residual GPU allocation.  No unresolved red
+  flag remains.  The three identical shared goodputs are reported as the
+  offered-load ceiling, not treated as independent performance variation.