Gahow Wang
029821c1b6
Single-GPU bench on dash1 GPU 0 (vanilla vLLM 0.18.1, chunked-prefill on,
no kv_connector). 3 decode batch sizes × 5 prefill sizes × 3 reps.
Method recap (driver: microbench/interference/driver.py, repurposed):
- Pin D streaming decode requests at constant max_tokens
- Inject one prefill-only request (max_tokens=1) of varying input length
- Bin decode-stream token timestamps into "during prefill" vs baseline
- Headline metric: effective per-stream TPOT during the prefill burst,
= prefill_ttft / (num_tokens_during_prefill / D). This is the average
rate at which each decode stream produces tokens during the burst.
p50 of inter-token intervals is deceptive (chunked-prefill makes most
intervals look normal); the burst-average gives the true cost.
Results (D=8 row, the most agentic-realistic case):
P (tokens) | prefill_ttft | per-stream TPOT during | penalty
2048 | 143 ms | 32 ms | 4×
8192 | 583 ms | 114 ms | 15×
32768 | 4520 ms | 388 ms | 52×
65536 | 15615 ms | 757 ms | 99×
131072 | 56991 ms | 1419 ms | 183×
Baseline TPOT at D=8: ~7.7 ms. So during a 131k-token prefill burst
each ongoing decode is running ~183× slower (i.e. essentially halted)
for ~57 seconds.
§3.2 implication: PD-disagg's promised phase-isolation benefit per
agentic request is bounded by the decode duration, which is 50–200 ms
for tool-call output. MB2 says the KV-transfer cost of PD-disagg
is 300 ms – 10 s for agentic-size requests. Cost > benefit for every
KV size above ~80 MiB (well below trace mean 192 MiB).
The new figs/pd_cost_vs_benefit.png overlays MB1 benefit ceiling
(50–200 ms band, capped by decode) onto MB2 transfer cost curve and
marks the agentic-distribution waypoints (trace mean, p90, p95, p99)
on the x-axis. Across the entire agentic distribution, the cost curve
sits above the benefit band.
Adds:
- microbench/fresh_setup/mb1_launch.sh: single-GPU vLLM launcher (no
kv_connector, default chunked_prefill=on, max_num_batched_tokens=8192)
- microbench/fresh_setup/mb1_driver.py: copy of the existing
microbench/interference/driver.py for cpfs deployment
- microbench/fresh_setup/analyze_mb1.py: aggregator emitting
per-(D, P) effective-TPOT-during + max PD-disagg-benefit table
- microbench/fresh_setup/plot_mb1.py: mb1 standalone +
pd_cost_vs_benefit headline figure
- analysis/mb1/summary.csv: 45 raw rows from the sweep
- analysis/mb1/breakdown.json: per-(D, P) aggregate
- analysis/mb1/README.md: persistent doc
- figs/mb1_interference.png: effective TPOT during prefill, one line per D
- figs/pd_cost_vs_benefit.png: §3.2 headline (cost > benefit everywhere)
Caveats noted in README:
- chunk_tokens=8192 only; Sarathi-Serve's smaller chunks would
interleave decode more aggressively. Chunk-size sensitivity is
flagged as next run.
- D ≤ 8; higher D may saturate or shrink the penalty further.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
178 lines
8.4 KiB
Markdown
178 lines
8.4 KiB
Markdown
# MB1 — Prefill–Decode Interference (chunked-prefill on, vLLM 0.18.1 default)
|
||
|
||
Persistent record of the phase-interference microbench used to put a
|
||
quantitative upper bound on **what PD-disaggregation can buy** under the
|
||
chunked-prefill-on baseline. Re-runs append a dated section at the
|
||
bottom; the **Summary** block is what gets cited.
|
||
|
||
---
|
||
|
||
## Summary (latest)
|
||
|
||
| Headline | Value |
|
||
|---|---|
|
||
| Baseline single-stream TPOT (D=1, idle GPU) | **4.8 ms** |
|
||
| Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** |
|
||
| Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** |
|
||
| Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** |
|
||
| Maximum PD-disagg benefit per agentic decode | **≤ 50–200 ms** (= decode duration) |
|
||
|
||
**§3.2 headline (cost vs benefit, this run + MB2)**:
|
||
|
||
> Under chunked-prefill, every ongoing decode stream is essentially
|
||
> **halted while a prefill chunk is in flight** — per-stream effective
|
||
> TPOT during the burst is 15× to 2000× baseline, scaling with prefill
|
||
> size. PD-disagg can recover this stall, but the recovery is bounded by
|
||
> the **decode duration** of the request being protected. For agentic,
|
||
> decode is 50–200 ms (tool-call output). MB2 shows PD-disagg pays
|
||
> 300 ms – 10 s of KV-transfer cost per request to do that recovery. The
|
||
> cost exceeds the benefit ceiling for any per-request KV ≥ ~80 MiB
|
||
> (~830 tokens) — well below all agentic operating points. The benefit
|
||
> never beats the cost in this workload.
|
||
|
||
## Setup
|
||
|
||
| Component | Value |
|
||
|---|---|
|
||
| Host | dash1, H20 96 GiB, driver 570.133.20 |
|
||
| Venv | `/home/admin/cpfs/wjh/agentic-kv-fresh/.venv` |
|
||
| vLLM | 0.18.1 official wheel (chunked-prefill default-on, V1 engine) |
|
||
| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct` |
|
||
| Launch flags | `--tensor-parallel-size 1 --enable-prefix-caching --gpu-memory-utilization 0.9 --max-model-len 200000 --max-num-batched-tokens 8192` |
|
||
| kv_connector | **none** (this measures pure single-GPU phase interference; PD-disagg cost lives in MB2) |
|
||
|
||
## Method
|
||
|
||
Adapted from `microbench/interference/driver.py`:
|
||
|
||
1. Start D streaming decode requests on `/v1/chat/completions` with a
|
||
long max_tokens cap. Discard the first 32 tokens as warmup.
|
||
2. After 1 s, inject one prefill-only request with `max_tokens=1` and
|
||
an input of `P` synthetic tokens (uuid-seeded for zero prefix-cache
|
||
reuse). Measure the prefill's TTFT.
|
||
3. Bin the *during-prefill* tokens from each decode stream by whether
|
||
their wall-clock falls inside `[prefill_inject_ts, prefill_inject_ts +
|
||
prefill_ttft]`. Report inter-token p50 / p90.
|
||
4. Bin a baseline run (D streams, no prefill injection) the same way.
|
||
|
||
We additionally compute the **effective per-stream TPOT during the
|
||
prefill burst** as the single most informative summary:
|
||
|
||
```
|
||
eff_TPOT_during = prefill_ttft_ms / (num_tokens_during_prefill / D)
|
||
```
|
||
|
||
This is the average rate at which each decode stream produces tokens
|
||
while a prefill is in flight. Compared to baseline TPOT it gives the
|
||
real per-stream throughput penalty (chunked-prefill p50 looks deceptively
|
||
fine because most decode-token intervals during the burst are at normal
|
||
speed; p90 sees the stall but is itself noisy; the effective TPOT is
|
||
the cleanest "average over the whole burst window" number).
|
||
|
||
## Results — 2026-05-27, dash1 GPU 0, chunk_tokens=8192
|
||
|
||
3 D × 5 P × 3 reps. Aggregated by `analyze_mb1.py`.
|
||
|
||
| D | P (tok) | base TPOT (ms) | prefill_ttft (ms) | per-stream tokens during | effective TPOT during (ms) | penalty | max PD-disagg benefit per stream (ms) |
|
||
|--:|--:|--:|--:|--:|--:|--:|--:|
|
||
| 1 | 2 048 | 4.79 | 163 | 4.0 | 41 | 8× | 144 |
|
||
| 1 | 8 192 | 4.78 | 584 | 5.0 | 117 | 24× | 560 |
|
||
| 1 | 32 768 | 4.78 | 4 515 | 5.0 | 903 | 189× | 4 491 |
|
||
| 1 | 65 536 | 4.78 | 15 568 | 5.3 | 2 919 | 610× | 15 542 |
|
||
| 1 | 131 072 | 4.78 | 56 765 | 5.7 | 10 017 | 2 094× | 56 738 |
|
||
| 4 | 2 048 | 5.62 | 138 | 3.9 | 36 | 6× | 117 |
|
||
| 4 | 8 192 | 6.08 | 574 | 4.5 | 128 | 21× | 547 |
|
||
| 4 | 32 768 | 6.09 | 4 529 | 11.9 | 381 | 63× | 4 457 |
|
||
| 4 | 65 536 | 5.85 | 15 587 | 19.8 | 789 | 135× | 15 471 |
|
||
| 4 | 131 072 | 6.27 | 56 697 | 37.4 | 1 517 | 242× | 56 463 |
|
||
| 8 | 2 048 | 7.71 | 143 | 4.5 | 32 | 4× | 109 |
|
||
| 8 | 8 192 | 7.69 | 583 | 5.1 | 114 | 15× | 544 |
|
||
| 8 | 32 768 | 7.42 | 4 520 | 11.7 | 387 | 52× | 4 434 |
|
||
| 8 | 65 536 | 7.67 | 15 615 | 20.6 | 757 | 99× | 15 457 |
|
||
| 8 | 131 072 | 7.74 | 56 991 | 40.2 | 1 419 | 183× | 56 680 |
|
||
|
||
**Reading the table**:
|
||
|
||
- *Baseline TPOT* grows mildly with D (4.8 ms → 7.7 ms as D goes 1 → 8).
|
||
Multi-stream decoding has small but nonzero contention even without
|
||
prefill.
|
||
- *Effective TPOT during* grows mostly with P: a single 8k prefill stalls
|
||
decode for ~580 ms regardless of D, so each stream emits only a handful
|
||
of tokens during that 580 ms window — effective per-stream TPOT
|
||
collapses to 100–130 ms. Larger prefill = more chunks = larger stall.
|
||
- *Penalty* is the eff/baseline ratio. Above 50× for P ≥ 32k. Above
|
||
500× for D=1 at P ≥ 65k.
|
||
- *Max PD-disagg benefit per stream* = `prefill_ttft − per_stream_tokens
|
||
× baseline_TPOT` ≈ `prefill_ttft` (since interference essentially
|
||
halts decode). This is the entire prefill duration's worth of decode
|
||
time that could in principle be recovered.
|
||
|
||
Two big caveats for **agentic** application:
|
||
|
||
1. **Decode is short** (~50–200 ms for tool-call output). The actual
|
||
recoverable benefit per request is bounded by the decode duration,
|
||
not by `prefill_ttft`. If a decode lasts 100 ms and a 5-second prefill
|
||
collides with it, PD-disagg can save at most 100 ms — not 5 s.
|
||
2. **PD-disagg pays KV-transfer cost** (MB2: 300 ms – 10 s per request
|
||
for agentic sizes). For any KV ≥ ~80 MiB the cost already exceeds the
|
||
~100 ms benefit ceiling. Cost > benefit across the whole agentic
|
||
distribution.
|
||
|
||
## §3.2 cost-vs-benefit figure
|
||
|
||
`figs/pd_cost_vs_benefit.png` overlays MB1 benefit ceiling (50–200 ms
|
||
band, capped by decode duration) on top of MB2 transfer cost curve. The
|
||
cost curve crosses the benefit ceiling somewhere around **80 MiB / 830
|
||
tokens** of KV — well below the trace mean (192 MiB / 2k tok ≈ trace
|
||
mean per request KV, and we know agentic averages 33k tokens, p99
|
||
125k). For anything bigger PD-disagg pays more than it can recover.
|
||
|
||
## Reproduction
|
||
|
||
```bash
|
||
# vllm pair-free single-instance launch
|
||
ssh dash1 'GPU=0 PORT=8000 CHUNK_TOKENS=8192 \
|
||
bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh start'
|
||
|
||
# sweep
|
||
ssh dash1 'source /home/admin/cpfs/wjh/agentic-kv-fresh/.venv/bin/activate && \
|
||
python /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_driver.py \
|
||
--host 127.0.0.1 --port 8000 \
|
||
--model /home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
|
||
--decode-batch-sizes 1,4,8 --prefill-tokens 2048,8192,32768,65536,131072 \
|
||
--reps 3 --output-dir /home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results'
|
||
|
||
# pull + analyze
|
||
scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/summary.csv \
|
||
analysis/mb1/summary.csv
|
||
.venv/bin/python microbench/fresh_setup/analyze_mb1.py \
|
||
--summary analysis/mb1/summary.csv --out analysis/mb1/breakdown.json
|
||
.venv/bin/python microbench/fresh_setup/plot_mb1.py \
|
||
--mb1 analysis/mb1/breakdown.json \
|
||
--mb2-intra analysis/mb2/intra_kvboth_breakdown.json \
|
||
--mb2-inter analysis/mb2/inter_kvboth_breakdown.json
|
||
|
||
# teardown
|
||
ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop'
|
||
```
|
||
|
||
## Open questions / next runs
|
||
|
||
- **Chunk size sensitivity**: this run uses `--max-num-batched-tokens
|
||
8192`. Sarathi-Serve goes smaller (e.g. 1024) and recovers more
|
||
decode interleaving inside each prefill burst. Worth running
|
||
chunk_tokens ∈ {1024, 2048, 4096, 16384} to map the chunk-size axis.
|
||
- **Higher D**: 12, 16 streams to see whether the penalty saturates or
|
||
keeps shrinking per-stream.
|
||
- **Cross-validate effective_TPOT_during with token-time-series plot**:
|
||
raw per-token timestamps could reveal whether the stall is a few big
|
||
spikes or many small ones (currently inferred from p50/p90 spread).
|
||
|
||
## Run log
|
||
|
||
### 2026-05-27 — dash1 GPU 0, chunk_tokens=8192
|
||
|
||
3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on
|
||
dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`.
|
||
Figures: `figs/mb1_interference.png`, `figs/pd_cost_vs_benefit.png`.
|