agentic-kvc/analysis/mb1/README.md

MB1 — Prefill–Decode Interference (chunked-prefill on, vLLM 0.18.1 default)

Persistent record of the phase-interference microbench used to put a quantitative upper bound on what PD-disaggregation can buy under the chunked-prefill-on baseline. Re-runs append a dated section at the bottom; the Summary block is what gets cited.

---

Summary (latest)

Headline	Value
Baseline single-stream TPOT (D=1, idle GPU)	4.8 ms
Effective per-stream TPOT during 8k-token prefill burst (D=8)	114 ms (≈15× baseline)
Effective per-stream TPOT during 32k-token prefill burst (D=8)	388 ms (≈52×)
Effective per-stream TPOT during 131k-token prefill burst (D=8)	1419 ms (≈183×)
Maximum PD-disagg benefit per agentic decode	≤ 50–200 ms (= decode duration)

§3.2 headline (cost vs benefit, this run + MB2):

Under chunked-prefill, every ongoing decode stream is essentially halted while a prefill chunk is in flight — per-stream effective TPOT during the burst is 15× to 2000× baseline, scaling with prefill size. PD-disagg can recover this stall, but the recovery is bounded by the decode duration of the request being protected. For agentic, decode is 50–200 ms (tool-call output). MB2 shows PD-disagg pays 300 ms – 10 s of KV-transfer cost per request to do that recovery. The cost exceeds the benefit ceiling for any per-request KV ≥ ~80 MiB (~830 tokens) — well below all agentic operating points. The benefit never beats the cost in this workload.

Setup

Component	Value
Host	dash1, H20 96 GiB, driver 570.133.20
Venv	/home/admin/cpfs/wjh/agentic-kv-fresh/.venv
vLLM	0.18.1 official wheel (chunked-prefill default-on, V1 engine)
Model	/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct
Launch flags	--tensor-parallel-size 1 --enable-prefix-caching --gpu-memory-utilization 0.9 --max-model-len 200000 --max-num-batched-tokens 8192
kv_connector	none (this measures pure single-GPU phase interference; PD-disagg cost lives in MB2)

Method

Adapted from microbench/interference/driver.py:

1. Start D streaming decode requests on /v1/chat/completions with a long max_tokens cap. Discard the first 32 tokens as warmup.
2. After 1 s, inject one prefill-only request with max_tokens=1 and an input of P synthetic tokens (uuid-seeded for zero prefix-cache reuse). Measure the prefill's TTFT.
3. Bin the during-prefill tokens from each decode stream by whether their wall-clock falls inside [prefill_inject_ts, prefill_inject_ts + prefill_ttft]. Report inter-token p50 / p90.
4. Bin a baseline run (D streams, no prefill injection) the same way.

We additionally compute the effective per-stream TPOT during the prefill burst as the single most informative summary:

eff_TPOT_during = prefill_ttft_ms / (num_tokens_during_prefill / D)

This is the average rate at which each decode stream produces tokens while a prefill is in flight. Compared to baseline TPOT it gives the real per-stream throughput penalty (chunked-prefill p50 looks deceptively fine because most decode-token intervals during the burst are at normal speed; p90 sees the stall but is itself noisy; the effective TPOT is the cleanest "average over the whole burst window" number).

Results — 2026-05-27, dash1 GPU 0, chunk_tokens=8192

3 D × 5 P × 3 reps. Aggregated by analyze_mb1.py.

D	P (tok)	base TPOT (ms)	prefill_ttft (ms)	per-stream tokens during	effective TPOT during (ms)	penalty	max PD-disagg benefit per stream (ms)
1	2 048	4.79	163	4.0	41	8×	144
1	8 192	4.78	584	5.0	117	24×	560
1	32 768	4.78	4 515	5.0	903	189×	4 491
1	65 536	4.78	15 568	5.3	2 919	610×	15 542
1	131 072	4.78	56 765	5.7	10 017	2 094×	56 738
4	2 048	5.62	138	3.9	36	6×	117
4	8 192	6.08	574	4.5	128	21×	547
4	32 768	6.09	4 529	11.9	381	63×	4 457
4	65 536	5.85	15 587	19.8	789	135×	15 471
4	131 072	6.27	56 697	37.4	1 517	242×	56 463
8	2 048	7.71	143	4.5	32	4×	109
8	8 192	7.69	583	5.1	114	15×	544
8	32 768	7.42	4 520	11.7	387	52×	4 434
8	65 536	7.67	15 615	20.6	757	99×	15 457
8	131 072	7.74	56 991	40.2	1 419	183×	56 680

Reading the table:

• Baseline TPOT grows mildly with D (4.8 ms → 7.7 ms as D goes 1 → 8). Multi-stream decoding has small but nonzero contention even without prefill.
• Effective TPOT during grows mostly with P: a single 8k prefill stalls decode for ~580 ms regardless of D, so each stream emits only a handful of tokens during that 580 ms window — effective per-stream TPOT collapses to 100–130 ms. Larger prefill = more chunks = larger stall.
• Penalty is the eff/baseline ratio. Above 50× for P ≥ 32k. Above 500× for D=1 at P ≥ 65k.
• Max PD-disagg benefit per stream = prefill_ttft − per_stream_tokens × baseline_TPOT ≈ prefill_ttft (since interference essentially halts decode). This is the entire prefill duration's worth of decode time that could in principle be recovered.

Two big caveats for agentic application:

1. Decode is short (~50–200 ms for tool-call output). The actual recoverable benefit per request is bounded by the decode duration, not by prefill_ttft. If a decode lasts 100 ms and a 5-second prefill collides with it, PD-disagg can save at most 100 ms — not 5 s.
2. PD-disagg pays KV-transfer cost (MB2: 300 ms – 10 s per request for agentic sizes). For any KV ≥ ~80 MiB the cost already exceeds the ~100 ms benefit ceiling. Cost > benefit across the whole agentic distribution.

§3.2 cost-vs-benefit figure

figs/pd_cost_vs_benefit.png overlays MB1 benefit ceiling (50–200 ms band, capped by decode duration) on top of MB2 transfer cost curve. The cost curve crosses the benefit ceiling somewhere around 80 MiB / 830 tokens of KV — well below the trace mean (192 MiB / 2k tok ≈ trace mean per request KV, and we know agentic averages 33k tokens, p99 125k). For anything bigger PD-disagg pays more than it can recover.

Reproduction

# vllm pair-free single-instance launch
ssh dash1 'GPU=0 PORT=8000 CHUNK_TOKENS=8192 \
  bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh start'

# sweep
ssh dash1 'source /home/admin/cpfs/wjh/agentic-kv-fresh/.venv/bin/activate && \
  python /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_driver.py \
    --host 127.0.0.1 --port 8000 \
    --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
    --decode-batch-sizes 1,4,8 --prefill-tokens 2048,8192,32768,65536,131072 \
    --reps 3 --output-dir /home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results'

# pull + analyze
scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/summary.csv \
    analysis/mb1/summary.csv
.venv/bin/python microbench/fresh_setup/analyze_mb1.py \
  --summary analysis/mb1/summary.csv --out analysis/mb1/breakdown.json
.venv/bin/python microbench/fresh_setup/plot_mb1.py \
  --mb1 analysis/mb1/breakdown.json \
  --mb2-intra analysis/mb2/intra_kvboth_breakdown.json \
  --mb2-inter analysis/mb2/inter_kvboth_breakdown.json

# teardown
ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop'

Open questions / next runs

• Chunk size sensitivity: this run uses --max-num-batched-tokens 8192. Sarathi-Serve goes smaller (e.g. 1024) and recovers more decode interleaving inside each prefill burst. Worth running chunk_tokens ∈ {1024, 2048, 4096, 16384} to map the chunk-size axis.
• Higher D: 12, 16 streams to see whether the penalty saturates or keeps shrinking per-stream.
• Cross-validate effective_TPOT_during with token-time-series plot: raw per-token timestamps could reveal whether the stall is a few big spikes or many small ones (currently inferred from p50/p90 spread).

Run log

2026-05-27 — dash1 GPU 0, chunk_tokens=8192

3 × 5 × 3 sweep. CSV: analysis/mb1/summary.csv. Per-config JSONs on dash1 at /home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/. Figures: figs/mb1_interference.png, figs/pd_cost_vs_benefit.png.