agentic-kvc/analysis/mb1/README.md

MB1 — Prefill–Decode Interference (chunked-prefill on, vLLM 0.18.1 default)

Persistent record of the phase-interference microbench used to put a quantitative upper bound on what PD-disaggregation can buy under the chunked-prefill-on baseline. Re-runs append a dated section at the bottom; the Summary block is what gets cited.

---

Summary (latest)

Headline	Value
Baseline single-stream TPOT (D=1, idle GPU)	4.8 ms
Effective per-stream TPOT during 8k-token prefill burst (D=8)	114 ms (≈15× baseline)
Effective per-stream TPOT during 32k-token prefill burst (D=8)	388 ms (≈52×)
Effective per-stream TPOT during 131k-token prefill burst (D=8)	1419 ms (≈183×)

What MB1 actually measures:

During a prefill burst, every ongoing decode stream is essentially halted (per-stream effective TPOT is 15×–2000× baseline, scaling with prefill size). The total decode time lost per prefill event is D × T_prefill (D concurrent decodes each lose ~T_prefill of useful work). For the trace mean (P ≈ 33k tokens, T_prefill ≈ 4.5 s) at D=8 that's ~36 seconds of decode-equivalent work lost per request. This is the upper bound on what PD-disaggregation's phase isolation could recover on the decode side.

⚠ Correction (2026-05-27): an earlier version of this README framed the §3.2 PD-disagg argument as "phase-isolation benefit is capped at the decode duration of the new request (~50–200 ms), so MB2 transfer cost dominates". That framing was wrong. The correct accounting is benefit-per-prefill-event = D × T_prefill (aggregate decode time saved across all stalled streams), which is much larger than per-request transfer cost. The actual reason static PD-disagg fails in agentic is D-side KV pool capacity (figs/f4b_pdsep_kv_wall.png), not a cost-vs-benefit imbalance on phase isolation. See RESULTS_SUMMARY.md section 4 for the corrected framing.

Setup

Component	Value
Host	dash1, H20 96 GiB, driver 570.133.20
Venv	/home/admin/cpfs/wjh/agentic-kv-fresh/.venv
vLLM	0.18.1 official wheel (chunked-prefill default-on, V1 engine)
Model	/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct
Launch flags	--tensor-parallel-size 1 --enable-prefix-caching --gpu-memory-utilization 0.9 --max-model-len 200000 --max-num-batched-tokens 8192
kv_connector	none (this measures pure single-GPU phase interference; PD-disagg cost lives in MB2)

Method

Adapted from microbench/interference/driver.py:

1. Start D streaming decode requests on /v1/chat/completions with a long max_tokens cap. Discard the first 32 tokens as warmup.
2. After 1 s, inject one prefill-only request with max_tokens=1 and an input of P synthetic tokens (uuid-seeded for zero prefix-cache reuse). Measure the prefill's TTFT.
3. Bin the during-prefill tokens from each decode stream by whether their wall-clock falls inside [prefill_inject_ts, prefill_inject_ts + prefill_ttft]. Report inter-token p50 / p90.
4. Bin a baseline run (D streams, no prefill injection) the same way.

We additionally compute the effective per-stream TPOT during the prefill burst as the single most informative summary:

eff_TPOT_during = prefill_ttft_ms / (num_tokens_during_prefill / D)

This is the average rate at which each decode stream produces tokens while a prefill is in flight. Compared to baseline TPOT it gives the real per-stream throughput penalty (chunked-prefill p50 looks deceptively fine because most decode-token intervals during the burst are at normal speed; p90 sees the stall but is itself noisy; the effective TPOT is the cleanest "average over the whole burst window" number).

Results — 2026-05-27, dash1 GPU 0, chunk_tokens=8192

3 D × 5 P × 3 reps. Aggregated by analyze_mb1.py.

D	P (tok)	base TPOT (ms)	prefill_ttft (ms)	per-stream tokens during	effective TPOT during (ms)	penalty	max PD-disagg benefit per stream (ms)
1	2 048	4.79	163	4.0	41	8×	144
1	8 192	4.78	584	5.0	117	24×	560
1	32 768	4.78	4 515	5.0	903	189×	4 491
1	65 536	4.78	15 568	5.3	2 919	610×	15 542
1	131 072	4.78	56 765	5.7	10 017	2 094×	56 738
4	2 048	5.62	138	3.9	36	6×	117
4	8 192	6.08	574	4.5	128	21×	547
4	32 768	6.09	4 529	11.9	381	63×	4 457
4	65 536	5.85	15 587	19.8	789	135×	15 471
4	131 072	6.27	56 697	37.4	1 517	242×	56 463
8	2 048	7.71	143	4.5	32	4×	109
8	8 192	7.69	583	5.1	114	15×	544
8	32 768	7.42	4 520	11.7	387	52×	4 434
8	65 536	7.67	15 615	20.6	757	99×	15 457
8	131 072	7.74	56 991	40.2	1 419	183×	56 680

Reading the table:

• Baseline TPOT grows mildly with D (4.8 ms → 7.7 ms as D goes 1 → 8). Multi-stream decoding has small but nonzero contention even without prefill.
• Effective TPOT during grows mostly with P: a single 8k prefill stalls decode for ~580 ms regardless of D, so each stream emits only a handful of tokens during that 580 ms window — effective per-stream TPOT collapses to 100–130 ms. Larger prefill = more chunks = larger stall.
• Penalty is the eff/baseline ratio. Above 50× for P ≥ 32k. Above 500× for D=1 at P ≥ 65k.
• Max PD-disagg benefit per stream = prefill_ttft − per_stream_tokens × baseline_TPOT ≈ prefill_ttft (since interference essentially halts decode). This is the entire prefill duration's worth of decode time that could in principle be recovered.

Connecting to the §3.2 PD-disagg argument (corrected):

PD-disagg's promised phase-isolation benefit is per prefill event, not per request. When a new prefill arrives, it stalls every concurrent decode stream on the same GPU. The aggregate decode time lost across those D streams is D × T_prefill. PD-disagg moving prefill off-decode-GPU recovers all of it.

Plugging numbers per prefill event:

Prefill size	T_prefill	PD-disagg cost (MB2 T_transfer)	PD-disagg benefit (D=8 × T_prefill)	Ratio
2k tok (trace lower)	0.14 s	8 ms	1.1 s	0.7 %
33k tok (trace mean)	4.5 s	320 ms	36 s	0.9 %
125k tok (~p99)	57 s	1.9 s	456 s	0.4 %

On the phase-isolation axis alone, PD-disagg wins by 100×–250×. The reason static PD-disagg nonetheless fails in agentic is a different failure mode: the D-side KV pool cannot fit p90+ requests (p99 = 11.5 GiB; D-instance pool ≈ 38 GiB; 4P+4D halves system-wide decode capacity → TTFT p50 62×, success rate 99.5% → 52% in colleague's 4P+4D experiment). The structural problem is capacity (see figs/f4b_pdsep_kv_wall.png), not transfer-cost vs phase-isolation trade-off.

Reproduction

# vllm pair-free single-instance launch
ssh dash1 'GPU=0 PORT=8000 CHUNK_TOKENS=8192 \
  bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh start'

# sweep
ssh dash1 'source /home/admin/cpfs/wjh/agentic-kv-fresh/.venv/bin/activate && \
  python /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_driver.py \
    --host 127.0.0.1 --port 8000 \
    --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
    --decode-batch-sizes 1,4,8 --prefill-tokens 2048,8192,32768,65536,131072 \
    --reps 3 --output-dir /home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results'

# pull + analyze
scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/summary.csv \
    analysis/mb1/summary.csv
.venv/bin/python microbench/fresh_setup/analyze_mb1.py \
  --summary analysis/mb1/summary.csv --out analysis/mb1/breakdown.json
.venv/bin/python microbench/fresh_setup/plot_mb1.py \
  --mb1 analysis/mb1/breakdown.json \
  --mb2-intra analysis/mb2/intra_kvboth_breakdown.json \
  --mb2-inter analysis/mb2/inter_kvboth_breakdown.json

# teardown
ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop'

Open questions / next runs

• Chunk size sensitivity: this run uses --max-num-batched-tokens 8192. Sarathi-Serve goes smaller (e.g. 1024) and recovers more decode interleaving inside each prefill burst. Worth running chunk_tokens ∈ {1024, 2048, 4096, 16384} to map the chunk-size axis.
• Higher D: 12, 16 streams to see whether the penalty saturates or keeps shrinking per-stream.
• Cross-validate effective_TPOT_during with token-time-series plot: raw per-token timestamps could reveal whether the stall is a few big spikes or many small ones (currently inferred from p50/p90 spread).

Run log

2026-05-27 — dash1 GPU 0, chunk_tokens=8192

3 × 5 × 3 sweep. CSV: analysis/mb1/summary.csv. Per-config JSONs on dash1 at /home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/. Figure: figs/mb1_interference.png. The figure figs/pd_cost_vs_benefit.png from the original commit 029821c was based on the wrong "benefit ≤ decode duration" accounting; deleted in the correction commit.