# Action-aware constraint pilot v2 results

Date: 2026-07-14 (Asia/Singapore).

Decision: **`STOP_WORKLOAD_NOT_CROSSED`**.

The pilot produced one valid positive regime and one invalid-for-effect-size
regime.  It supports continuing a narrower action-response investigation, but
it does not justify an end-to-end telemetry-guided tuner claim or new engine
instrumentation yet.

## Question tested

Given only a completed source run, can existing engine telemetry distinguish
which of two one-knob interventions will improve SLO-goodput more?

The frozen score counted scheduler steps with backlog where either MNS or MBBT
was exclusively at its configured limit.  It made two pre-intervention
predictions on the same workload and offered load:

- Regime A: source `(MNS=16, MBBT=8192)` predicts increasing MNS to 64 over
  increasing MBBT to 16384.
- Regime B: source `(MNS=64, MBBT=2048)` predicts increasing MBBT to 8192 over
  increasing MNS to 128.

The primary outcome was 300-second SLO-goodput.  A predicted action had to beat
the alternative on every paired request band by at least 10% of that band's
source goodput.  Telemetry direction also had to remain stable at 25%, 50%,
75%, and 100% of the replay.

## Setup

- Host: `dash0`, GPU 0-3 used exclusively; four NVIDIA H20 GPUs; TP=4.  GPU
  4-7 remained idle to avoid co-location effects.
- Model: Qwen3-30B-A3B BF16 at
  `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`.
- Runtime: patched vLLM `0.24.1.dev3+g668cfb7e2`, source commit `4b253fd`, with
  OpProf Layer-1 telemetry.
- Workload: `chat_w20260312_1000`, 2.125 requests/s/GPU, 300-second arrival
  window, 128 output tokens, three disjoint request bands.
- Five fresh-server configurations, three measured runs each, counter-rotated
  repetition order, 16-request warm-up, and a 510-request/60-second burn-in.
  SLO early stopping was disabled.
- Exact request-id, arrival-order, and input-length hashes matched for every
  paired comparison.

The authoritative run root is
`/home/admin/cpfs/wjh/action-aware-constraint-v2-20260714`.  The final audit is
`pilot-audit-final.json`, SHA256
`7ebe080fcc4970bef423bc587253d157e75aed1ea8b410bd37770c17708135ab`.

## End-to-end result

### Regime A: strong MNS constraint

| Rep | Source 16/8192 | MBBT action 16/16384 | MNS action 64/8192 | MNS-only source steps | MBBT-only source steps | `(MNS action - MBBT action) / source` |
|---:|---:|---:|---:|---:|---:|---:|
| 1 | 4.710 | 7.687 | 8.500 | 79.85% | 0.038% | 17.27% |
| 2 | 2.803 | 4.150 | 8.500 | 94.61% | 0.005% | 155.17% |
| 3 | 2.227 | 3.943 | 8.500 | 79.87% | 0.005% | 204.64% |

Units are SLO-goodput requests/s except the percentage columns.  The source
prediction was stable at every phase checkpoint and correct in all three
paired bands.  The predicted MNS action cleared the frozen 10% material-margin
gate in all three bands.

This is a real but limited positive result.  The source was an extreme case:
MNS was full on nearly every scheduler step that retained backlog, TTFT p50 was
1.64-5.61 seconds, KV usage remained below 2.45%, and no preemption occurred.
An expert or a simple rule could identify this case without a learned tuner.

### Regime B: MBBT direction at an outcome ceiling

| Rep | Source 64/2048 | MNS action 128/2048 | MBBT action 64/8192 | MBBT-only source steps | `(MBBT action - MNS action) / source` |
|---:|---:|---:|---:|---:|
| 1 | 8.423 | 8.420 | 8.500 | 10.90% | 0.950% |
| 2 | 8.447 | 8.500 | 8.500 | 12.03% | 0% |
| 3 | 8.500 | 8.497 | 8.500 | 8.79% | 0.039% |

The telemetry direction was stable in all phase checkpoints.  The predicted
action won twice and tied once, while the wrong MNS action left the MBBT-only
state intact.  However, the source already delivered 99.10-100% of the offered
8.5 requests/s.  Even a perfect action could not reach the preregistered 10%
margin.  Regime B therefore does not test material weak-signal value; it is a
workload-selection failure, not evidence that telemetry does or does not help
near a decision boundary.

The missing preflight condition is mathematical.  For an effect threshold
`delta` and offered goodput ceiling `G`, the source must satisfy
`source <= G / (1 + delta)`.  Here `G=8.5` and `delta=0.10`, so any source above
7.727 requests/s cannot possibly pass before either target is measured.

## Why the exclusive-limit rule is incomplete

The alternative MBBT action improved Regime A by 48.0%, 63.2%, and 77.1% over
the source even though MBBT was almost never the exclusive backlog constraint.
This rules out the binary interpretation "a knob that is not exclusively at
its cap cannot help."

Existing richer telemetry provides a plausible mechanism:

| Rep | Split-prefill requests, source -> MBBT action | Prefill steps, source -> MBBT action | Prefill requests/step, source -> MBBT action | Prefix-hit rate, source -> MBBT action |
|---:|---:|---:|---:|---:|
| 1 | 41 -> 2 | 2324 -> 2022 | 1.071 -> 1.249 | 13.851% -> 13.747% |
| 2 | 7 -> 0 | 2410 -> 2291 | 1.012 -> 1.076 | 13.078% -> 12.988% |
| 3 | 12 -> 1 | 2377 -> 2270 | 1.018 -> 1.096 | 13.613% -> 13.604% |

Increasing MBBT allows more prefill work to be packed into one iteration and
nearly eliminates split prefills.  Under MNS=16, this can reduce the number of
iterations for which long prompts occupy scarce running slots.  Prefix-cache
hit rates differ by at most 0.104 percentage points, and exact workload hashes
match, so neither explains the gain.  Step-duration p99 also remains similar;
one 1.127-second decode-step outlier appears in `a_mbbt/rep2`, but the same
action direction occurs in all three bands.

This is a mechanism-consistent explanation, not a completed causal
decomposition.  MBBT simultaneously changes total per-iteration token budget,
per-request chunk size, and multi-request packing.  Instrumentation observes
their joint response but cannot separate those effects without another
intervention.

## Instrumentation decision

Do **not** add a new engine patch for this mechanism yet.  The existing OpProf
stream already records submit/complete timestamps, prefill/decode composition,
chunked-prefill categories, prefix hits, queues, KV usage, and CUDA graph mode.
Those fields are sufficient to identify the interaction missed by the initial
exclusive-limit rule.

The next narrow mechanism ablation is available in the current vLLM runtime:

1. `(MNS=16, MBBT=8192, long-prefill-threshold=0)` is the current source.
2. `(MNS=16, MBBT=16384, long-prefill-threshold=8192)` keeps individual long
   chunks at 8192 while increasing total packing budget.
3. `(MNS=16, MBBT=16384, long-prefill-threshold=0)` is the current MBBT action.

The runtime exposes `--long-prefill-token-threshold`; with a threshold of 8192,
the second arm separates total packing headroom from the larger per-request
chunk allowed by the third arm.  A formal test must rerun all three arms fresh
with counter-rotated order rather than reuse today's endpoints.

## Correct tuning-research route

The pilot does not support turning the frozen equality checks into a larger
rule tree.  The supported route is **intervention-calibrated, action-conditioned
system identification**:

```text
source event sequence + normalized config delta
    -> predicted distribution of Delta SLO-goodput and evaluation cost
    -> uncertainty-aware next-config selection
```

The policy input should retain continuous distributions and phase evolution:
queue/running residency, MNS and token slack, prefill/decode composition,
partial-prefill occupancy, step time, KV state, and graph behavior.  Human
bottleneck labels and hand-authored `if queue then increase MNS` mappings are
not policy inputs.  Mechanism summaries remain audit and interpretation tools;
the action response is learned from paired real interventions.

The harness has a narrower, non-heuristic role:

- define legal configurations and exact paired workloads;
- reject source points without outcome headroom before a full sweep;
- randomize/counter-rotate execution order and preserve failures/cost;
- validate stream coverage, hashes, request accounting, and censoring;
- expose target outcomes only after a source-only prediction is frozen;
- evaluate fixed-budget regret and H20-hours, not explanation quality alone.

The next tuning experiment should use non-extreme, non-ceiling source points
and a local two-dimensional MNS/MBBT neighborhood.  A short run may screen load
only; every inferential telemetry and outcome result remains a 300-second run.
At least one held-out workload must be reserved before choosing features or
thresholds.

Primary evaluation is H20-hours/trials to reach 95% of the real local oracle
and cost-normalized regret AUC.  Required baselines are random search,
config/outcome-only sequential search, the current rule heuristic, and the
same action-response model with telemetry removed.  Action-ranking accuracy is
supporting evidence only.

## What this pilot establishes and does not establish

Established:

- Long-window engine state can make a correct, phase-stable action-family
  prediction in an extreme MNS-constrained regime.
- A naive `queue > 0 -> increase MNS` rule would choose an ineffective action
  in Regime B; action-conditioned state distinguishes the mechanism direction,
  although the measured effect is immaterial at the selected load.
- Binary exclusive-cap attribution misses a substantial MNS/MBBT interaction;
  existing chunk/step telemetry reveals a plausible explanation.
- Source outcome headroom must be an explicit experiment admission gate.

Not established:

- telemetry improves an end-to-end tuner over an outcome-only baseline;
- weak or mixed constraints can be ranked with material gain;
- the response transfers across workloads, models, TP, or hardware;
- new engine instrumentation is necessary;
- the chunking/packing breakdown is causal rather than mechanism-consistent.

## Change and verification

Reproduction:

```bash
python3 runs/action-aware-v0/test_pilot.py

python3 runs/action-aware-v0/analyze_pilot.py \
  --run-root /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/runs/pilot \
  --manifest runs/action-aware-v0/pilot-manifest-v2.json \
  --output /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/pilot-audit-final.json
```

The fresh GPU run used AITuner commit `c5ab073`; asynchronous coverage was
corrected in `3facb18`; reproducible mechanism summaries were added in
`2af22db`.  The raw run is unchanged across those analyzer-only commits.

Change: added a crossed real-intervention controller and audit, fixed the
burn-in result gate, corrected asynchronous per-step coverage accounting, and
added reproducible step/chunk/prefix mechanism summaries.

Expected effect: distinguish descriptive telemetry from source-only action
predictions that survive paired real interventions.

Verification: local and remote action-aware test suites pass; all five sessions
completed; all stream/footer and request-accounting invariants pass; the final
analyzer was run twice and produced byte-identical output.

Result: Regime A passes; Regime B is invalid for the frozen effect-size test;
the global decision is `STOP_WORKLOAD_NOT_CROSSED`.

Remaining risk: one model, one TP, one trace family, three bands, two action
families, and deliberately constructed endpoints are development evidence
only.  The strong positive regime is too obvious to support a paper claim.

## Data sanity

- Measured runs: n=15; elapsed 300.610-317.350 seconds; 15 distinct.  Pass
  rate min/max 0.2620/1.0 with 11 distinct values; SLO-goodput min/max
  2.2267/8.5 requests/s with 11 distinct values.
- Telemetry intervals: n=15; records min/max 13,621/23,711; 15 distinct.
  Start gaps min/max 0.0412/0.1227 seconds; end gaps 0.00053/0.0649; uncovered
  internal gaps 0/0.3231.  One submit gap reached 1.1190 seconds but was fully
  covered by a 1.1269-second recorded execution; contiguous indices and zero
  drops were preserved.
- Sessions: n=5; cost min/max 1.1691/1.2730 H20-hours; 5 distinct.  V2 cost
  was 6.0862 H20-hours; V0/V1/V2 total was 6.7906, below the 8.0 cap.
- Regime-A split-prefill observations: n=6; min/max 0/41 requests; 6 distinct.
  Prefix-hit rates: n=6; min/max 0.12988/0.13851; 6 distinct.
- Checked invariants: non-negative counters and durations; ratios in `[0,1]`;
  exact request, arrival, and length hashes; 2550/2550 request accounting per
  measured run; uncensored outcomes; outcomes across configurations not all
  identical;
  five complete streams; monotonic timestamps; contiguous step indices; zero
  drops; footer/sidecar agreement; chunk-token accounting; bounded prefix hits;
  no OOM, controller error, or residual GPU allocation.  No unresolved red
  flag remains.  The three identical shared goodputs are reported as the
  offered-load ceiling, not treated as independent performance variation.
