agentic-kvc/analysis/pd_separation_analysis.md

PD Disaggregation for Agentic LLM Workloads: A Systematic Study

TL;DR

We benchmarked PD separation (prefill-decode disaggregation) against PD co-location on a production agentic-coder trace (GLM-5.1, 2.1M requests, avg 33.6k input tokens). Under a fair comparison with the same cache-aware global scheduler:

PD separation is net negative for single-machine agentic workloads. The root cause is not what prior work (DistServe, Splitwise) targeted — it is a KV cache memory wall on decode instances.

Config (TP=1, 8×H20)	TTFT p50	TPOT p90	GPU util	KV cache pressure
Combined DP=8 (cache-aware)	0.731s	0.073s	30.5%	Low (spread across 8 inst)
PD-Sep 6P+2D (cache-aware)	1.481s	0.077s	16.9%	97.1% on decode

Per-request breakdown shows 87.7% of TTFT is spent waiting for KV cache memory on decode instances, not prefill compute or KV transfer.

---

1. Workload Characterization

Trace: GLM-5.1 Agentic Coder, production cluster, 2 hours

Metric	Value
Requests	2,114,220
Input tokens	71.1B (avg 33.6k, p50=20k, p90=88k)
Output tokens	940M (avg 445, p50=80)
I/O ratio	75.6x aggregate, 217.8x per-request median
Prefill token share	98%
Sessions	1.3M (90% single-turn)
>32k input	38% of requests, 79% of tokens

KV cache reuse:

Metric	Value
Theoretical prefix cache hit (infinite, single inst)	71%
Shared hash blocks (ref>1)	47% of unique blocks
Intra-session reuse	57%
Top blocks ref count	64,754 (system prompt)
Actual APC (Combined, cache-aware, 8 inst)	44.7%
Actual APC (Round-robin, 8 inst)	20.8%

Request profile after prefix cache:

Bucket	Count	Avg new tokens to prefill
>90% cache hit (warm)	22%	1,314
50-90% cache hit	14%	10,052
1-50% cache hit	8%	38,909
0% cache hit (cold)	55%	17,696

2. Experiment Setup

Hardware: 8× NVIDIA H20 (96GB HBM, NVLink, 4× ConnectX-7 200Gbps RDMA)

Software: vLLM 0.18.1 (source in third_party/vllm/, patched scheduler assert), Mooncake 0.3.10 (RDMA KV transfer), uv-managed Python venv

Model: Qwen3-Coder-30B-A3B-Instruct (MoE 128E top-8, 3B active params)

Configurations tested (all use same cache-aware + token-level LB global scheduler unless noted):

Config	Instances	GPU allocation	Scheduler
Combined TP=8 DP=1	1	8 GPU shared	N/A (single)
Combined TP=2 DP=4	4 independent	2 GPU each	RR (legacy)
Combined TP=1 DP=8	8 independent	1 GPU each	RR / cache-aware
PD-Sep TP=1 4P+4D	4P + 4D Mooncake	4 GPU P, 4 GPU D	cache-aware
PD-Sep TP=1 6P+2D	6P + 2D Mooncake	6 GPU P, 2 GPU D	cache-aware

Benchmark params: 1000 sampled requests (200 for ablations), --enforce-eager, --max-model-len 200000

Trace sampler: scripts/sample_trace.py — random session sampling preserving multi-turn structure + hash_ids

Global scheduler: scripts/cache_aware_proxy.py — supports both --combined (PD-colo) and --prefill/--decode (PD-sep) modes. Score = ongoing_tokens/avg_load - α·cache_hit_ratio, session affinity for multi-turn.

3. Results

3.1 Main Comparison (unified cache-aware scheduler)

Config	OK/N	TTFT p50	TPOT p90	E2E p50	APC
Combined TP=1 DP=8 (cache-aware)	997/999	0.731s	0.073s	4.48s	44.7%
PD-Sep TP=1 4P+4D (cache-aware)	509/564	1.261s	0.074s	5.61s	40.2%
Combined TP=1 DP=8 (RR)	997/999	1.836s	0.086s	6.67s	20.8%

3.2 GPU Utilization (200 req, time_scale=20)

Config	All GPU mean	Prefill GPU	Decode GPU	Decode KV cache
Combined 8colo	30.5% (active 64%)	—	—	Distributed
PD-Sep 4P+4D	12.4% (active 24%)	16.9% (active 17%)	7.8% (active 30%)	~97%
PD-Sep 6P+2D	16.9% (active 28%)	16.2% (active 16%)	19.0% (active 64%)	~97%

3.3 Per-Request Breakdown (6P+2D, await mode)

Stage	p50	% of TTFT
Prefill (queue + compute + KV push)	0.108s	12.3%
Proxy overhead	0.000s	0.0%
KV pull + decode wait	109.6s	87.7%
Total TTFT	110.2s	100%

Root cause of 109.6s kv+decode: vLLM decode log shows Running: 0 reqs, Waiting: 6 reqs, KV cache: 97.1%. GPU idle, requests queued for KV cache memory.

3.4 Ablations

Ablation	Change	TTFT	TPOT p90	Verdict
P/D ratio: 6P+2D vs 4P+4D	More prefill GPUs	-26%	~same	Helps TTFT (less prefill queue)
Fire-and-forget vs await	Async prefill dispatch	+260%	-44%	Hurts (decode KV cache contention)

4. Analysis

4.1 DistServe's Assumptions vs Agentic Reality

Assumption	Chatbot (DistServe)	Agentic (this work)
A. P is compute-bound, D is memory-bound	✅	✅ Even at 95% reuse, prefill AI >1000x vs decode AI <2
B. PD co-location causes interference	✅	❌ Cache-aware routing eliminates interference (TPOT 0.073 vs 0.074)
C. KV transfer cost negligible	✅ (short input)	❌ Avg 33.6k tokens, TTFT +72% from transfer
D. Dedicated prefill improves throughput	✅	❌ 71% cache hit → prefill already lightweight
E. Decode KV cache not a bottleneck	✅ (short context)	❌ THE bottleneck: 97% KV cache on decode

4.2 Roofline: Prefill Stays Compute-Bound Under High Cache Reuse

SeqLen=64k, Model=Qwen3-30B-A3B MoE, GPU=H20 (ridge point=37 FLOP/byte)

Reuse%   NewTokens   AI (FLOP/byte)   Bound        vs Decode
0%       64,000      40,758           COMPUTE      26,813x
70%      19,200      20,610           COMPUTE      13,559x
90%       6,400       8,544           COMPUTE       5,621x
95%       3,200       4,549           COMPUTE       2,993x
Decode        1         1.5           MEMORY            1x

Even at 95% reuse, prefill AI = 4549 >> ridge point 37. Prefill remains compute-bound because Q×K^T attention scales with new_tokens × seq_len (quadratic in context, not just new tokens).

But absolute FLOPs drop: 71% cache → only 29% of tokens need compute. This makes P-D interference negligible without physical separation.

4.3 The Real Bottleneck: Decode KV Cache Memory Wall

PD separation concentrates all decode onto fewer GPUs:

	Combined (8 inst)	PD-Sep 6P+2D
Decode KV cache total	8 × 28GB = 224GB	2 × 28GB = 56GB
Concurrent decode reqs	~1 per inst	~4 per inst
KV cache utilization	Low	97.1%

At 97.1% KV cache usage, a 49-token request (KV = few KB) waits 114 seconds for a 64k-token request to finish decode and release its ~8GB of KV cache.

This is memory-capacity head-of-line blocking: the GPU is idle (Running: 0), but cannot schedule new requests because KV cache is full.

4.4 Why Cache-Aware Routing Matters More Than PD Separation

Change	TTFT impact	TPOT p90 impact	APC impact
RR → cache-aware routing	-60%	-15%	+24pp
Combined → PD-Sep	+72%	+1%	-5pp

Cache-aware routing provides "soft PD isolation" by reducing per-instance prefill workload through better cache utilization, without the KV transfer overhead or decode memory wall of physical PD separation.

5. Conclusions

1. Single-machine PD separation is net negative for agentic workloads due to decode KV cache memory wall
2. Cache-aware routing is the dominant optimization — improves TTFT by 60%, TPOT by 15%, APC by 24pp
3. Prefill stays compute-bound even at 95% cache reuse, but absolute compute drops enough to eliminate P-D interference
4. PD separation may help in multi-machine settings where decode has dedicated memory pools (e.g., DRAM-backed Mooncake KV store) not limited by single-GPU HBM

6. Patches Applied to vLLM 0.18.1

File	Change	Reason
v1/core/sched/scheduler.py	assert req_id in self.requests → graceful skip	KV transfer callback races with request abort

---

Appendix: Experiment Artifacts

Data on dash0 (~/agentic-kv/outputs/)

Directory	Config	Requests	Notes
v18_combined_1000req	TP=8 DP=1, 16 sess, 120s TO	1000	Baseline with /metrics APC
exp1_combined_tp2_dp4	TP=2 DP=4, RR, 8 sess	999	No summary (killed)
exp2_combined_tp1_dp8	TP=1 DP=8, cache-aware, 8 sess	999	Unified scheduler baseline
exp3_pd_sep_tp1_mooncake	TP=1 4P+4D Mooncake, cache-aware	~560	Multiple iterations
gpu_ab_combined	TP=1 DP=8 cache-aware, 200 req	200	GPU util CSV + metrics
gpu_ab_pdsep	TP=1 4P+4D cache-aware, 200 req	200	GPU util CSV + metrics
gpu_ab_6p2d	TP=1 6P+2D cache-aware, 200 req	200	Ablation 1: P/D ratio
gpu_ab_6p2d_fnf	TP=1 6P+2D fire-and-forget, 200 req	67	Ablation 2: scheduling
breakdown_await	TP=1 6P+2D await, 50 req	50	Per-stage breakdown

Trace on dash0

Path	Description
~/ali-trace/trace-glm5.1/	Raw production logs (301GB, 4 files × 30min)
~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl	Formatted 2h trace (2.1M requests)
~/agentic-kv/traces/sampled_1000req_seed42.jsonl	Sampled 1000 requests for benchmarks

Key Scripts

Script	Purpose
scripts/cache_aware_proxy.py	Unified global scheduler (combined + PD-sep modes)
scripts/sample_trace.py	Trace sampler preserving sessions + hash_ids
replayer/	Async trace replayer with streaming metrics
scripts/compute_roofline.py	Prefill/decode roofline analysis
scripts/analyze_cache_hit.py	Theoretical vs actual KV cache hit ratio
scripts/analyze_breakdown.py	Per-request stage breakdown from proxy
scripts/gpu_monitor.sh	5s-interval GPU utilization sampling

Reproducing

# On dash0, activate env
cd ~/agentic-kv && source .venv/bin/activate

# Sample trace
python scripts/sample_trace.py --input ~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl \
    --output traces/sampled_1000req_seed42.jsonl --target-requests 1000 --seed 42

# Combined TP=1 DP=8 + cache-aware scheduler
for i in $(seq 0 7); do
    MASTER_PORT=$((29500+i)) CUDA_VISIBLE_DEVICES=$i vllm serve $MODEL \
        --port $((8000+i)) --tp 1 --enable-prefix-caching --enforce-eager &
done
python scripts/cache_aware_proxy.py --combined http://127.0.0.1:800{0..7} --port 9090
python -m replayer --trace traces/sampled_1000req_seed42.jsonl \
    --endpoint http://localhost:9090 --time-scale 10 --max-inflight-sessions 8

# Breakdown data
curl http://localhost:9090/breakdown | python scripts/analyze_breakdown.py /dev/stdin