# Why does PD separation fail on agentic workloads even though prefill stays compute-bound? This is the central paradox of the section. The roofline in `figures/fig_c6_roofline.pdf` shows prefill is compute-bound at every realistic reuse level — at 95 % cache reuse, arithmetic intensity is still ≈ 4,500 FLOP/byte, more than two orders of magnitude above the H20 ridge of 37. The DistServe / Splitwise argument follows directly from that fact: "prefill is compute-heavy, decode is memory-bound, so isolate them onto different GPUs and specialize each." Yet on this workload, single-machine PD separation regresses TTFT by 72 % (REPORT.md §3.1) and saturates the decode-side KV pool at 97 % occupancy. This document explains, layer by layer, why a true premise (compute-bound) does not imply the conclusion (PD separation pays). All five layers are backed by either figures in this directory or measurements in `analysis/pd_separation_analysis.md`. The short answer: **the roofline tells you about the per-kernel efficiency of prefill. PD separation is a decision about a whole serving system. The gap between those two scales is where DistServe's argument loses force — and where agentic workloads, with their very large per-request KV footprint, push the system past a memory-capacity wall that chatbot workloads never reach.** --- ## Layer 1: compute-bound ≠ "needs dedicated GPUs" Roofline analysis classifies a *kernel*. It answers the question, "given that this kernel is running, is it bottlenecked by FLOP rate or by HBM bandwidth?" — it does **not** answer: - how long the kernel takes in wall-clock terms, - whether two kernels can profitably share a GPU, - whether moving the kernel to a different GPU makes it faster. PD separation needs the second and third answers, not the first. A 50 ms compute-bound prefill burst can perfectly well coexist with decode steps on the same GPU; you lose at most a fraction of a decode step's latency per chunk. Co-location only fails when prefill bursts grow long enough that decode requests starve. The DistServe paper's roofline argument is a *necessary* condition ("prefill *can* be compute-bound, so dedicating GPUs is *not wasted*"). It is **not** a *sufficient* condition ("therefore dedicated GPUs pay"). ## Layer 2: in agentic, absolute prefill work after cache hit is small The roofline is computed in `figures/fig_c6_roofline.pdf` at a full 64 k context. But the operating point on the trace is shifted by prefix cache hits: | reuse | new tokens | prefill time @ ~7,000 tok/s | |---|---|---| | 0 % (turn 1 cold) | 64,000 | ~9 s | | 71 % (trace average) | 18,600 | ~2.6 s | | 95 % (deep multi-turn) | 3,200 | ~0.5 s | Average-case prefill is ~2.6 s of compute. With 8 GPUs and peak QPS 1.6, each GPU sees ~0.3 s of prefill work per second of wall-clock. Chunked prefill in vLLM slices this into ~8 k-token chunks of ~50–100 ms each, then yields to decode. The decode-side disturbance per HEAVY request is on the order of "a few hundred ms of stretched decode," not "seconds of stalled decode." PD separation, in its best case, eliminates this disturbance. The *ceiling* on the benefit is therefore: hundreds of ms per HEAVY request. This budget has to absorb everything PD separation costs. ## Layer 3: PD separation relocates compute; it does not accelerate it A prefill kernel does the same FLOPs no matter which GPU runs it. PD separation's potential acceleration vectors are only two: 1. **Larger prefill batch** → better SM utilization for the prefill MMA kernels. 2. **No chunked-prefill yield to decode** → no overhead per chunk handoff. Both are quantitatively negligible in this regime: 1. At peak QPS 1.6 with ~2.6 s of prefill per request, *system-wide* prefill concurrency averages to ~4 active prefills. A 4P split sees ~1 prefill per GPU at any moment, so batching gains are zero. A 6P split makes it worse, not better. The roofline ceiling of 148 TFLOPS is already reachable at batch=1 for sequences this long. 2. Chunked prefill's per-chunk overhead is dominated by scheduler tick time (≈ 1–2 ms), not the chunk transition. Removing it saves single percentages of prefill time. So the *speedup side* of PD separation is bounded by the few-hundred-ms budget from Layer 2 and contains no hidden upside. ## Layer 4: the costs of PD separation are workload-scaled PD separation adds two costs, both of which scale *up* with workload size: 1. **KV transfer over the network.** Mooncake transfers KV block-by-block after the full prefill completes (no layer-wise pipeline; see `analysis/elastic_hypotheses.md` H5). Empirically, transfer takes ~1.1 s p50 for HEAVY requests (~40 k tokens of KV ≈ 3.8 GB at 96 KB/token), with tail extending to 18–30 s. Transfer time grows with context length. 2. **Decode-side KV concentration.** All decode work is funneled onto a subset of GPUs (4 of 8 in 4P+4D, 2 of 8 in 6P+2D). Per-D-instance KV *demand* therefore scales by N_total / N_D. This is the killer cost; Layer 5 quantifies it. Both costs scale linearly or worse with per-request KV footprint. KV footprint, in turn, scales linearly with input length. So PD separation gets *worse* exactly along the axis (long context) where the workload is moving. ## Layer 5: the decode-side KV memory wall (the actual mechanism) Visualized in `figures/fig_kv_memory_wall.pdf`. The model is simple and its constants are auditable in `scripts/plot_kv_memory_wall.py`: ``` per-D occupancy = (concurrent_decode × KV_per_req) / (N_D × KV_pool_per_GPU) ``` with: - `KV_per_req` = `seqlen × 96 KB/token` for Qwen3-30B-A3B (2 × 4 kv-heads × 128 head-dim × 2 bytes × 48 layers = 96 KB/tok) - `KV_pool_per_GPU` ≈ 28 GB (96 GB H20 HBM minus weights and activations) - `concurrent_decode` ≈ 8 at steady state (peak QPS 1.6 × mean decode duration ~5 s under Combined) Plug in the trace's input distribution from `figures/fig_c1a_io_cdf.pdf`: | operating point | KV/req | Combined (N_D=8) | 4P+4D (N_D=4) | 6P+2D (N_D=2) | |---|---|---|---|---| | chatbot avg (2 k) | 197 MB | 0.7 % | 1.4 % | 2.7 % | | **agentic avg (33.6 k)** | **3.3 GB** | **12 %** | **23 %** | **46 %** | | **agentic p90 (101 k)** | **9.9 GB** | **35 %** | **69 %** | **138 %** ⚠ | | **agentic p99 (132 k)** | **13.0 GB** | **45 %** | **90 %** ⚠ | **181 %** ⚠ | vLLM's scheduler stops admitting new requests at ~90 % KV pool occupancy, so anything above the wall translates directly to queueing. Two consequences fall out of this table: 1. **PD-sep with even a 4P+4D split breaches the wall at p99 context.** p99 alone is ~1 % of requests but holds the GPU for tens of seconds of decode, so its KV stays resident; over a long enough window the wall gets hit even from the tail. With 6P+2D the wall is breached well before p90. 2. **For chatbot, the entire table sits under 3 %.** PD separation never approaches the wall because chatbot per-request KV is 15× smaller. This is the assumption DistServe inherited from its target workload, and the assumption that silently breaks under agentic. The empirical KV occupancy on the 6P+2D run was 97 % (`analysis/pd_separation_analysis.md` §3.3) — the model and the measurement agree to within the resolution of the steady-state assumption. ## Layer 6: the DistServe / Splitwise assumption that silently breaks To formalize: the regime in which PD separation pays is bounded by both a roofline condition *and* a memory-capacity condition: | Condition | Form | Chatbot | Agentic | |---|---|---|---| | Prefill is compute-bound | AI_prefill ≫ ridge | ✓ | ✓ | | Decode is memory-bound | AI_decode ≪ ridge | ✓ | ✓ | | Per-D-instance KV demand fits | concurrent × KV/req / (N_D × pool) < 1 | ✓ (≪ 1) | ✗ (>1 at p90+) | | KV transfer time ≪ saved interference | transfer_s ≪ saved_decode_stall_s | ✓ (KV is MB) | ✗ (KV is GB) | DistServe and Splitwise hold all four conditions implicitly in their short-context regime. Agentic violates the bottom two. Both violations have the same root cause: per-request KV footprint is 15–60× larger. This is the falsifiable claim of the section: **PD separation pays iff per-request KV footprint × decode concurrency stays well below the per-D-instance HBM pool. When that condition fails — and it fails unavoidably for long-context agentic workloads — PD separation is net negative regardless of how compute-bound prefill is.** The roofline doesn't tell you whether you're inside this regime; only the memory budget does. --- ## What this means for the paper section The figures we already have support this argument: - `fig_c1a_io_cdf.pdf` — establishes the input-length distribution responsible for the large KV footprint (p50 33.5 k, p90 101 k, p99 132 k). - `fig_c1b_reuse.pdf` — establishes that 79 % of reuse is intra-session, i.e. the request set has long-lived sessions whose KV must sit in the pool through many decode steps. - `fig_c6_roofline.pdf` — establishes the prefill compute-bound fact. This is the apparent contradiction the section resolves. - `fig_kv_memory_wall.pdf` — establishes the resolution: the memory budget is what governs PD separation's viability, not the roofline. - `fig_c7_routing_lever.pdf` — establishes that cache-aware routing recovers most of the benefit PD separation promises, without paying the memory-wall cost. Missing for a rigorous re-grounding (deferred until the cudagraph re-run matrix lands): - Per-step decode KV utilization time-series from a live PD-sep run (currently inferred from a single log snapshot of "Running: 0, Waiting: 6, KV cache: 97.1 %"). This would *directly* show the memory wall being hit instead of relying on the steady-state model. - Per-request TTFT stacked breakdown (prefill, KV transfer, decode-side wait) on the new trace; currently `analysis/pd_separation_analysis.md` §3.3 has it on the old methodology. - CUDA-graph ablation: with `--enforce-eager` removed, PD-sep's D-node could in principle close some of the per-step decode latency gap. The Layer 5 model is gate-independent — wall demand grows with concurrency, not per-step latency — so this should not change the conclusion. But the section needs the measurement to say so honestly. The 4 h cudagraph experiment matrix (Combined / PD-sep × eager / cudagraph × 3 seeds) on `traces/w600_r0.0015_st30.jsonl` would settle those three items.