agentic-kvc/analysis/pd_sep_paper_section/system_analysis.md

Why does PD separation fail on agentic workloads even though prefill stays compute-bound?

This is the central paradox of the section. The roofline in figures/fig_c6_roofline.pdf shows prefill is compute-bound at every realistic reuse level — at 95 % cache reuse, arithmetic intensity is still ≈ 4,500 FLOP/byte, more than two orders of magnitude above the H20 ridge of 37. The DistServe / Splitwise argument follows directly from that fact: "prefill is compute-heavy, decode is memory-bound, so isolate them onto different GPUs and specialize each."

Yet on this workload, single-machine PD separation regresses TTFT by 72 % (REPORT.md §3.1) and saturates the decode-side KV pool at 97 % occupancy. This document explains, layer by layer, why a true premise (compute-bound) does not imply the conclusion (PD separation pays). All five layers are backed by either figures in this directory or measurements in analysis/pd_separation_analysis.md.

The short answer: the roofline tells you about the per-kernel efficiency of prefill. PD separation is a decision about a whole serving system. The gap between those two scales is where DistServe's argument loses force — and where agentic workloads, with their very large per-request KV footprint, push the system past a memory-capacity wall that chatbot workloads never reach.

---

Layer 1: compute-bound ≠ "needs dedicated GPUs"

Roofline analysis classifies a kernel. It answers the question, "given that this kernel is running, is it bottlenecked by FLOP rate or by HBM bandwidth?" — it does not answer:

• how long the kernel takes in wall-clock terms,
• whether two kernels can profitably share a GPU,
• whether moving the kernel to a different GPU makes it faster.

PD separation needs the second and third answers, not the first. A 50 ms compute-bound prefill burst can perfectly well coexist with decode steps on the same GPU; you lose at most a fraction of a decode step's latency per chunk. Co-location only fails when prefill bursts grow long enough that decode requests starve.

The DistServe paper's roofline argument is a necessary condition ("prefill can be compute-bound, so dedicating GPUs is not wasted"). It is not a sufficient condition ("therefore dedicated GPUs pay").

Layer 2: in agentic, absolute prefill work after cache hit is small

The roofline is computed in figures/fig_c6_roofline.pdf at a full 64 k context. But the operating point on the trace is shifted by prefix cache hits:

reuse	new tokens	prefill time @ ~7,000 tok/s
0 % (turn 1 cold)	64,000	~9 s
71 % (trace average)	18,600	~2.6 s
95 % (deep multi-turn)	3,200	~0.5 s

Average-case prefill is ~2.6 s of compute. With 8 GPUs and peak QPS 1.6, each GPU sees ~0.3 s of prefill work per second of wall-clock. Chunked prefill in vLLM slices this into ~8 k-token chunks of ~50–100 ms each, then yields to decode. The decode-side disturbance per HEAVY request is on the order of "a few hundred ms of stretched decode," not "seconds of stalled decode."

PD separation, in its best case, eliminates this disturbance. The ceiling on the benefit is therefore: hundreds of ms per HEAVY request. This budget has to absorb everything PD separation costs.

Layer 3: PD separation relocates compute; it does not accelerate it

A prefill kernel does the same FLOPs no matter which GPU runs it. PD separation's potential acceleration vectors are only two:

1. Larger prefill batch → better SM utilization for the prefill MMA kernels.
2. No chunked-prefill yield to decode → no overhead per chunk handoff.

Both are quantitatively negligible in this regime:

1. At peak QPS 1.6 with ~2.6 s of prefill per request, system-wide prefill concurrency averages to ~4 active prefills. A 4P split sees ~1 prefill per GPU at any moment, so batching gains are zero. A 6P split makes it worse, not better. The roofline ceiling of 148 TFLOPS is already reachable at batch=1 for sequences this long.

2. Chunked prefill's per-chunk overhead is dominated by scheduler tick time (≈ 1–2 ms), not the chunk transition. Removing it saves single percentages of prefill time.

So the speedup side of PD separation is bounded by the few-hundred-ms budget from Layer 2 and contains no hidden upside.

Layer 4: the costs of PD separation are workload-scaled

PD separation adds two costs, both of which scale up with workload size:

1. KV transfer over the network. Mooncake transfers KV block-by-block after the full prefill completes (no layer-wise pipeline; see analysis/elastic_hypotheses.md H5). Empirically, transfer takes ~1.1 s p50 for HEAVY requests (~40 k tokens of KV ≈ 3.8 GB at 96 KB/token), with tail extending to 18–30 s. Transfer time grows with context length.

2. Decode-side KV concentration. All decode work is funneled onto a subset of GPUs (4 of 8 in 4P+4D, 2 of 8 in 6P+2D). Per-D-instance KV demand therefore scales by N_total / N_D. This is the killer cost; Layer 5 quantifies it.

Both costs scale linearly or worse with per-request KV footprint. KV footprint, in turn, scales linearly with input length. So PD separation gets worse exactly along the axis (long context) where the workload is moving.

Layer 5: the decode-side KV memory wall (the actual mechanism)

Visualized in figures/fig_kv_memory_wall.pdf. The model is simple and its constants are auditable in scripts/plot_kv_memory_wall.py:

per-D occupancy = (concurrent_decode × KV_per_req) / (N_D × KV_pool_per_GPU)

with:

• KV_per_req = seqlen × 96 KB/token for Qwen3-30B-A3B (2 × 4 kv-heads × 128 head-dim × 2 bytes × 48 layers = 96 KB/tok)
• KV_pool_per_GPU ≈ 28 GB (96 GB H20 HBM minus weights and activations)
• concurrent_decode ≈ 8 at steady state (peak QPS 1.6 × mean decode duration ~5 s under Combined)

Plug in the trace's input distribution from figures/fig_c1a_io_cdf.pdf:

operating point	KV/req	Combined (N_D=8)	4P+4D (N_D=4)	6P+2D (N_D=2)
chatbot avg (2 k)	197 MB	0.7 %	1.4 %	2.7 %
agentic avg (33.6 k)	3.3 GB	12 %	23 %	46 %
agentic p90 (101 k)	9.9 GB	35 %	69 %	138 % ⚠
agentic p99 (132 k)	13.0 GB	45 %	90 % ⚠	181 % ⚠

vLLM's scheduler stops admitting new requests at ~90 % KV pool occupancy, so anything above the wall translates directly to queueing.

Two consequences fall out of this table:

1. PD-sep with even a 4P+4D split breaches the wall at p99 context. p99 alone is ~1 % of requests but holds the GPU for tens of seconds of decode, so its KV stays resident; over a long enough window the wall gets hit even from the tail. With 6P+2D the wall is breached well before p90.

2. For chatbot, the entire table sits under 3 %. PD separation never approaches the wall because chatbot per-request KV is 15× smaller. This is the assumption DistServe inherited from its target workload, and the assumption that silently breaks under agentic.

The empirical KV occupancy on the 6P+2D run was 97 % (analysis/pd_separation_analysis.md §3.3) — the model and the measurement agree to within the resolution of the steady-state assumption.

Layer 5b: empirical refinement — the bottleneck side depends on the P:D split

The model above predicts D-side saturation. The 6P+2D run in analysis/pd_separation_analysis.md §3.3 is consistent with it. But the new 4P+4D run (outputs/pd_matrix/pdsep-4p4d_cudagraph_seed1/, captured during this section's experiment matrix) tells a richer story.

Empirical numbers (combined-ca vs both PD-sep splits, same trace, all cudagraph, figures/fig_c2_pdsep_vs_combined.pdf):

metric	combined-ca (N=3)	pdsep-4p4d (N=1)	pdsep-6p2d (N=1)
success	99.5 %	52 % (444/850)	68 % (574/850)
TTFT p50	0.91 s	62.8 s (69×)	51.1 s (56×)
TTFT p90	12.7 s	491 s (39×)	400 s (31×)
TPOT p90	0.027 s	0.013 s (-52 %)	0.020 s (-26 %)
E2E p50	2.5 s	65.1 s (26×)	53.4 s (21×)
wall clock	944 s	7558 s (8×)	3693 s (3.9×)

The per-stage TTFT decomposition (figures/fig_c4_ttft_stacked.pdf) shows that for both PD-sep splits >97 % of TTFT is P-side prefill compute (65.6 s / 66.2 s in 4p4d; 43.1 s / 44.3 s in 6p2d). D-side wait + first token is at most 1.2 s in either config.

The KV-utilization time-series (figures/fig_c3_kv_timeseries.pdf) tells the full story:

• combined-ca: 8 GPUs oscillate 0–98 %, peaks bursty and short
• pdsep-4p4d: P-instances (orange) pinned at 85–100 % the entire 2-hour run; D-instances (red) bounce between 10 % and 50 % — only P side hits the wall
• pdsep-6p2d: both sides pinned near 100 % the entire run (per-instance peaks 99–100 % across all 8). P-side fills because D back-pressures (D can't free KV slots fast enough → P can't hand off → P-side KV accumulates).

This refines Layer 5: PD separation hits a memory wall on whichever side has fewer GPUs, and at extreme splits it co-saturates both sides through D-back-pressure.

Why P-side fills in 4P+4D

Two effects combine on P:

1. Compute concentration. Combined spreads prefill across 8 GPUs; 4P+4D over 4. Per-P-GPU prefill load is 2× the per-Combined-GPU load. With chunked prefill, multiple in-flight prefills compete for the scheduler.

2. KV residency on P. Mooncake does block-by-block transfer after the full prefill completes. Until D pulls and acknowledges every block, the completed-but-not-yet-transferred KV sits in P's pool — on top of all the partially-prefilled in-flight KV. Many concurrent 33–132 k contexts overwhelm a single 28 GB pool.

This is the same memory-wall mechanism Layer 5 described, but on the prefill side. The Layer 5 analytical model in figures/fig_kv_memory_wall.pdf accounted only for decode-side KV demand. The full model is:

P-side occupancy = (in_flight_prefills × KV_per_req) / (N_P × pool)
D-side occupancy = (concurrent_decode × KV_per_req) / (N_D × pool)

Whichever side hits the wall first becomes the back-pressure source. 4P+4D's P-side fills first because 4 GPUs are doing 8 GPUs' worth of prefill. 6P+2D's D-side hits the wall (4× concentration), which then back-pressures P (no slots to hand off into) until P also fills. In either case, some side blocks — which is why PD separation regresses across the P:D ratios we tested.

Updated falsifiable condition

The condition for PD separation to not regress is now two-sided:

max(
  in_flight_prefills × KV_per_req / (N_P × pool),
  concurrent_decode  × KV_per_req / (N_D × pool)
) < 1

For chatbot workloads (KV/req ≈ 200 MB), this holds easily on either side. For agentic with KV/req ≈ 3.3 GB on average and 10–13 GB at the tail, both terms cross 1 well below the chosen N_P or N_D.

Followups: re-render fig_kv_memory_wall.pdf to show both P and D curves once the 6P+2D run lands, with the empirical P-side and D-side peaks marked.

Layer 6: the DistServe / Splitwise assumption that silently breaks

To formalize: the regime in which PD separation pays is bounded by both a roofline condition and a memory-capacity condition:

Condition	Form	Chatbot	Agentic
Prefill is compute-bound	AI_prefill ≫ ridge	✓	✓
Decode is memory-bound	AI_decode ≪ ridge	✓	✓
Per-D-instance KV demand fits	concurrent × KV/req / (N_D × pool) < 1	✓ (≪ 1)	✗ (>1 at p90+)
KV transfer time ≪ saved interference	transfer_s ≪ saved_decode_stall_s	✓ (KV is MB)	✗ (KV is GB)

DistServe and Splitwise hold all four conditions implicitly in their short-context regime. Agentic violates the bottom two. Both violations have the same root cause: per-request KV footprint is 15–60× larger.

This is the falsifiable claim of the section: PD separation pays iff per-request KV footprint × decode concurrency stays well below the per-D-instance HBM pool. When that condition fails — and it fails unavoidably for long-context agentic workloads — PD separation is net negative regardless of how compute-bound prefill is.

The roofline doesn't tell you whether you're inside this regime; only the memory budget does.

---

What this means for the paper section

The figures we already have support this argument:

• fig_c1a_io_cdf.pdf — establishes the input-length distribution responsible for the large KV footprint (p50 33.5 k, p90 101 k, p99 132 k).
• fig_c1b_reuse.pdf — establishes that 79 % of reuse is intra-session, i.e. the request set has long-lived sessions whose KV must sit in the pool through many decode steps.
• fig_c6_roofline.pdf — establishes the prefill compute-bound fact. This is the apparent contradiction the section resolves.
• fig_kv_memory_wall.pdf — establishes the resolution: the memory budget is what governs PD separation's viability, not the roofline.
• fig_c7_routing_lever.pdf — establishes that cache-aware routing recovers most of the benefit PD separation promises, without paying the memory-wall cost.

Missing for a rigorous re-grounding (deferred until the cudagraph re-run matrix lands):

• Per-step decode KV utilization time-series from a live PD-sep run (currently inferred from a single log snapshot of "Running: 0, Waiting: 6, KV cache: 97.1 %"). This would directly show the memory wall being hit instead of relying on the steady-state model.
• Per-request TTFT stacked breakdown (prefill, KV transfer, decode-side wait) on the new trace; currently analysis/pd_separation_analysis.md §3.3 has it on the old methodology.
• CUDA-graph ablation: with --enforce-eager removed, PD-sep's D-node could in principle close some of the per-step decode latency gap. The Layer 5 model is gate-independent — wall demand grows with concurrency, not per-step latency — so this should not change the conclusion. But the section needs the measurement to say so honestly.

The 4 h cudagraph experiment matrix (Combined / PD-sep × eager / cudagraph × 3 seeds) on traces/w600_r0.0015_st30.jsonl would settle those three items.