261 lines
13 KiB
Markdown
261 lines
13 KiB
Markdown
# Action-aware constraint pilot v2 results
|
|
|
|
Date: 2026-07-14 (Asia/Singapore).
|
|
|
|
Decision: **`STOP_WORKLOAD_NOT_CROSSED`**.
|
|
|
|
The pilot produced one valid positive regime and one invalid-for-effect-size
|
|
regime. It supports continuing a narrower action-response investigation, but
|
|
it does not justify an end-to-end telemetry-guided tuner claim or new engine
|
|
instrumentation yet.
|
|
|
|
## Question tested
|
|
|
|
Given only a completed source run, can existing engine telemetry distinguish
|
|
which of two one-knob interventions will improve SLO-goodput more?
|
|
|
|
The frozen score counted scheduler steps with backlog where either MNS or MBBT
|
|
was exclusively at its configured limit. It made two pre-intervention
|
|
predictions on the same workload and offered load:
|
|
|
|
- Regime A: source `(MNS=16, MBBT=8192)` predicts increasing MNS to 64 over
|
|
increasing MBBT to 16384.
|
|
- Regime B: source `(MNS=64, MBBT=2048)` predicts increasing MBBT to 8192 over
|
|
increasing MNS to 128.
|
|
|
|
The primary outcome was 300-second SLO-goodput. A predicted action had to beat
|
|
the alternative on every paired request band by at least 10% of that band's
|
|
source goodput. Telemetry direction also had to remain stable at 25%, 50%,
|
|
75%, and 100% of the replay.
|
|
|
|
## Setup
|
|
|
|
- Host: `dash0`, GPU 0-3 used exclusively; four NVIDIA H20 GPUs; TP=4. GPU
|
|
4-7 remained idle to avoid co-location effects.
|
|
- Model: Qwen3-30B-A3B BF16 at
|
|
`/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`.
|
|
- Runtime: patched vLLM `0.24.1.dev3+g668cfb7e2`, source commit `4b253fd`, with
|
|
OpProf Layer-1 telemetry.
|
|
- Workload: `chat_w20260312_1000`, 2.125 requests/s/GPU, 300-second arrival
|
|
window, 128 output tokens, three disjoint request bands.
|
|
- Five fresh-server configurations, three measured runs each, counter-rotated
|
|
repetition order, 16-request warm-up, and a 510-request/60-second burn-in.
|
|
SLO early stopping was disabled.
|
|
- Exact request-id, arrival-order, and input-length hashes matched for every
|
|
paired comparison.
|
|
|
|
The authoritative run root is
|
|
`/home/admin/cpfs/wjh/action-aware-constraint-v2-20260714`. The final audit is
|
|
`pilot-audit-final.json`, SHA256
|
|
`7ebe080fcc4970bef423bc587253d157e75aed1ea8b410bd37770c17708135ab`.
|
|
|
|
## End-to-end result
|
|
|
|
### Regime A: strong MNS constraint
|
|
|
|
| Rep | Source 16/8192 | MBBT action 16/16384 | MNS action 64/8192 | MNS-only source steps | MBBT-only source steps | `(MNS action - MBBT action) / source` |
|
|
|---:|---:|---:|---:|---:|---:|---:|
|
|
| 1 | 4.710 | 7.687 | 8.500 | 79.85% | 0.038% | 17.27% |
|
|
| 2 | 2.803 | 4.150 | 8.500 | 94.61% | 0.005% | 155.17% |
|
|
| 3 | 2.227 | 3.943 | 8.500 | 79.87% | 0.005% | 204.64% |
|
|
|
|
Units are SLO-goodput requests/s except the percentage columns. The source
|
|
prediction was stable at every phase checkpoint and correct in all three
|
|
paired bands. The predicted MNS action cleared the frozen 10% material-margin
|
|
gate in all three bands.
|
|
|
|
This is a real but limited positive result. The source was an extreme case:
|
|
MNS was full on nearly every scheduler step that retained backlog, TTFT p50 was
|
|
1.64-5.61 seconds, KV usage remained below 2.45%, and no preemption occurred.
|
|
An expert or a simple rule could identify this case without a learned tuner.
|
|
|
|
### Regime B: MBBT direction at an outcome ceiling
|
|
|
|
| Rep | Source 64/2048 | MNS action 128/2048 | MBBT action 64/8192 | MBBT-only source steps | `(MBBT action - MNS action) / source` |
|
|
|---:|---:|---:|---:|---:|
|
|
| 1 | 8.423 | 8.420 | 8.500 | 10.90% | 0.950% |
|
|
| 2 | 8.447 | 8.500 | 8.500 | 12.03% | 0% |
|
|
| 3 | 8.500 | 8.497 | 8.500 | 8.79% | 0.039% |
|
|
|
|
The telemetry direction was stable in all phase checkpoints. The predicted
|
|
action won twice and tied once, while the wrong MNS action left the MBBT-only
|
|
state intact. However, the source already delivered 99.10-100% of the offered
|
|
8.5 requests/s. Even a perfect action could not reach the preregistered 10%
|
|
margin. Regime B therefore does not test material weak-signal value; it is a
|
|
workload-selection failure, not evidence that telemetry does or does not help
|
|
near a decision boundary.
|
|
|
|
The missing preflight condition is mathematical. For an effect threshold
|
|
`delta` and offered goodput ceiling `G`, the source must satisfy
|
|
`source <= G / (1 + delta)`. Here `G=8.5` and `delta=0.10`, so any source above
|
|
7.727 requests/s cannot possibly pass before either target is measured.
|
|
|
|
## Why the exclusive-limit rule is incomplete
|
|
|
|
The alternative MBBT action improved Regime A by 48.0%, 63.2%, and 77.1% over
|
|
the source even though MBBT was almost never the exclusive backlog constraint.
|
|
This rules out the binary interpretation "a knob that is not exclusively at
|
|
its cap cannot help."
|
|
|
|
Existing richer telemetry provides a plausible mechanism:
|
|
|
|
| Rep | Split-prefill requests, source -> MBBT action | Prefill steps, source -> MBBT action | Prefill requests/step, source -> MBBT action | Prefix-hit rate, source -> MBBT action |
|
|
|---:|---:|---:|---:|---:|
|
|
| 1 | 41 -> 2 | 2324 -> 2022 | 1.071 -> 1.249 | 13.851% -> 13.747% |
|
|
| 2 | 7 -> 0 | 2410 -> 2291 | 1.012 -> 1.076 | 13.078% -> 12.988% |
|
|
| 3 | 12 -> 1 | 2377 -> 2270 | 1.018 -> 1.096 | 13.613% -> 13.604% |
|
|
|
|
Increasing MBBT allows more prefill work to be packed into one iteration and
|
|
nearly eliminates split prefills. Under MNS=16, this can reduce the number of
|
|
iterations for which long prompts occupy scarce running slots. Prefix-cache
|
|
hit rates differ by at most 0.104 percentage points, and exact workload hashes
|
|
match, so neither explains the gain. Step-duration p99 also remains similar;
|
|
one 1.127-second decode-step outlier appears in `a_mbbt/rep2`, but the same
|
|
action direction occurs in all three bands.
|
|
|
|
This is a mechanism-consistent explanation, not a completed causal
|
|
decomposition. MBBT simultaneously changes total per-iteration token budget,
|
|
per-request chunk size, and multi-request packing. Instrumentation observes
|
|
their joint response but cannot separate those effects without another
|
|
intervention.
|
|
|
|
## Instrumentation decision
|
|
|
|
Do **not** add a new engine patch for this mechanism yet. The existing OpProf
|
|
stream already records submit/complete timestamps, prefill/decode composition,
|
|
chunked-prefill categories, prefix hits, queues, KV usage, and CUDA graph mode.
|
|
Those fields are sufficient to identify the interaction missed by the initial
|
|
exclusive-limit rule.
|
|
|
|
The next narrow mechanism ablation is available in the current vLLM runtime:
|
|
|
|
1. `(MNS=16, MBBT=8192, long-prefill-threshold=0)` is the current source.
|
|
2. `(MNS=16, MBBT=16384, long-prefill-threshold=8192)` keeps individual long
|
|
chunks at 8192 while increasing total packing budget.
|
|
3. `(MNS=16, MBBT=16384, long-prefill-threshold=0)` is the current MBBT action.
|
|
|
|
The runtime exposes `--long-prefill-token-threshold`; with a threshold of 8192,
|
|
the second arm separates total packing headroom from the larger per-request
|
|
chunk allowed by the third arm. A formal test must rerun all three arms fresh
|
|
with counter-rotated order rather than reuse today's endpoints.
|
|
|
|
## Correct tuning-research route
|
|
|
|
The pilot does not support turning the frozen equality checks into a larger
|
|
rule tree. The supported route is **intervention-calibrated, action-conditioned
|
|
system identification**:
|
|
|
|
```text
|
|
source event sequence + normalized config delta
|
|
-> predicted distribution of Delta SLO-goodput and evaluation cost
|
|
-> uncertainty-aware next-config selection
|
|
```
|
|
|
|
The policy input should retain continuous distributions and phase evolution:
|
|
queue/running residency, MNS and token slack, prefill/decode composition,
|
|
partial-prefill occupancy, step time, KV state, and graph behavior. Human
|
|
bottleneck labels and hand-authored `if queue then increase MNS` mappings are
|
|
not policy inputs. Mechanism summaries remain audit and interpretation tools;
|
|
the action response is learned from paired real interventions.
|
|
|
|
The harness has a narrower, non-heuristic role:
|
|
|
|
- define legal configurations and exact paired workloads;
|
|
- reject source points without outcome headroom before a full sweep;
|
|
- randomize/counter-rotate execution order and preserve failures/cost;
|
|
- validate stream coverage, hashes, request accounting, and censoring;
|
|
- expose target outcomes only after a source-only prediction is frozen;
|
|
- evaluate fixed-budget regret and H20-hours, not explanation quality alone.
|
|
|
|
The next tuning experiment should use non-extreme, non-ceiling source points
|
|
and a local two-dimensional MNS/MBBT neighborhood. A short run may screen load
|
|
only; every inferential telemetry and outcome result remains a 300-second run.
|
|
At least one held-out workload must be reserved before choosing features or
|
|
thresholds.
|
|
|
|
Primary evaluation is H20-hours/trials to reach 95% of the real local oracle
|
|
and cost-normalized regret AUC. Required baselines are random search,
|
|
config/outcome-only sequential search, the current rule heuristic, and the
|
|
same action-response model with telemetry removed. Action-ranking accuracy is
|
|
supporting evidence only.
|
|
|
|
## What this pilot establishes and does not establish
|
|
|
|
Established:
|
|
|
|
- Long-window engine state can make a correct, phase-stable action-family
|
|
prediction in an extreme MNS-constrained regime.
|
|
- A naive `queue > 0 -> increase MNS` rule would choose an ineffective action
|
|
in Regime B; action-conditioned state distinguishes the mechanism direction,
|
|
although the measured effect is immaterial at the selected load.
|
|
- Binary exclusive-cap attribution misses a substantial MNS/MBBT interaction;
|
|
existing chunk/step telemetry reveals a plausible explanation.
|
|
- Source outcome headroom must be an explicit experiment admission gate.
|
|
|
|
Not established:
|
|
|
|
- telemetry improves an end-to-end tuner over an outcome-only baseline;
|
|
- weak or mixed constraints can be ranked with material gain;
|
|
- the response transfers across workloads, models, TP, or hardware;
|
|
- new engine instrumentation is necessary;
|
|
- the chunking/packing breakdown is causal rather than mechanism-consistent.
|
|
|
|
## Change and verification
|
|
|
|
Reproduction:
|
|
|
|
```bash
|
|
python3 runs/action-aware-v0/test_pilot.py
|
|
|
|
python3 runs/action-aware-v0/analyze_pilot.py \
|
|
--run-root /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/runs/pilot \
|
|
--manifest runs/action-aware-v0/pilot-manifest-v2.json \
|
|
--output /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/pilot-audit-final.json
|
|
```
|
|
|
|
The fresh GPU run used AITuner commit `c5ab073`; asynchronous coverage was
|
|
corrected in `3facb18`; reproducible mechanism summaries were added in
|
|
`2af22db`. The raw run is unchanged across those analyzer-only commits.
|
|
|
|
Change: added a crossed real-intervention controller and audit, fixed the
|
|
burn-in result gate, corrected asynchronous per-step coverage accounting, and
|
|
added reproducible step/chunk/prefix mechanism summaries.
|
|
|
|
Expected effect: distinguish descriptive telemetry from source-only action
|
|
predictions that survive paired real interventions.
|
|
|
|
Verification: local and remote action-aware test suites pass; all five sessions
|
|
completed; all stream/footer and request-accounting invariants pass; the final
|
|
analyzer was run twice and produced byte-identical output.
|
|
|
|
Result: Regime A passes; Regime B is invalid for the frozen effect-size test;
|
|
the global decision is `STOP_WORKLOAD_NOT_CROSSED`.
|
|
|
|
Remaining risk: one model, one TP, one trace family, three bands, two action
|
|
families, and deliberately constructed endpoints are development evidence
|
|
only. The strong positive regime is too obvious to support a paper claim.
|
|
|
|
## Data sanity
|
|
|
|
- Measured runs: n=15; elapsed 300.610-317.350 seconds; 15 distinct. Pass
|
|
rate min/max 0.2620/1.0 with 11 distinct values; SLO-goodput min/max
|
|
2.2267/8.5 requests/s with 11 distinct values.
|
|
- Telemetry intervals: n=15; records min/max 13,621/23,711; 15 distinct.
|
|
Start gaps min/max 0.0412/0.1227 seconds; end gaps 0.00053/0.0649; uncovered
|
|
internal gaps 0/0.3231. One submit gap reached 1.1190 seconds but was fully
|
|
covered by a 1.1269-second recorded execution; contiguous indices and zero
|
|
drops were preserved.
|
|
- Sessions: n=5; cost min/max 1.1691/1.2730 H20-hours; 5 distinct. V2 cost
|
|
was 6.0862 H20-hours; V0/V1/V2 total was 6.7906, below the 8.0 cap.
|
|
- Regime-A split-prefill observations: n=6; min/max 0/41 requests; 6 distinct.
|
|
Prefix-hit rates: n=6; min/max 0.12988/0.13851; 6 distinct.
|
|
- Checked invariants: non-negative counters and durations; ratios in `[0,1]`;
|
|
exact request, arrival, and length hashes; 2550/2550 request accounting per
|
|
measured run; uncensored outcomes; outcomes across configurations not all
|
|
identical;
|
|
five complete streams; monotonic timestamps; contiguous step indices; zero
|
|
drops; footer/sidecar agreement; chunk-token accounting; bounded prefix hits;
|
|
no OOM, controller error, or residual GPU allocation. No unresolved red
|
|
flag remains. The three identical shared goodputs are reported as the
|
|
offered-load ceiling, not treated as independent performance variation.
|