# Why does PD separation fail on agentic workloads even though prefill stays compute-bound? This is the central paradox of the section. The roofline in `figures/fig_c6_roofline.pdf` shows prefill is compute-bound at every realistic reuse level — at 95 % cache reuse, arithmetic intensity is still ≈ 4,500 FLOP/byte, more than two orders of magnitude above the H20 ridge of 37. The DistServe / Splitwise argument follows directly from that fact: "prefill is compute-heavy, decode is memory-bound, so isolate them onto different GPUs and specialize each." Yet on this workload, single-machine PD separation regresses TTFT by 72 % (REPORT.md §3.1) and saturates the decode-side KV pool at 97 % occupancy. This document explains, layer by layer, why a true premise (compute-bound) does not imply the conclusion (PD separation pays). All five layers are backed by either figures in this directory or measurements in `analysis/pd_separation_analysis.md`. The short answer: **the roofline tells you about the per-kernel efficiency of prefill. PD separation is a decision about a whole serving system. The gap between those two scales is where DistServe's argument loses force — and where agentic workloads, with their very large per-request KV footprint, push the system past a memory-capacity wall that chatbot workloads never reach.** --- ## Layer 1: compute-bound ≠ "needs dedicated GPUs" Roofline analysis classifies a *kernel*. It answers the question, "given that this kernel is running, is it bottlenecked by FLOP rate or by HBM bandwidth?" — it does **not** answer: - how long the kernel takes in wall-clock terms, - whether two kernels can profitably share a GPU, - whether moving the kernel to a different GPU makes it faster. PD separation needs the second and third answers, not the first. A 50 ms compute-bound prefill burst can perfectly well coexist with decode steps on the same GPU; you lose at most a fraction of a decode step's latency per chunk. Co-location only fails when prefill bursts grow long enough that decode requests starve. The DistServe paper's roofline argument is a *necessary* condition ("prefill *can* be compute-bound, so dedicating GPUs is *not wasted*"). It is **not** a *sufficient* condition ("therefore dedicated GPUs pay"). ## Layer 2: in agentic, absolute prefill work after cache hit is small The roofline is computed in `figures/fig_c6_roofline.pdf` at a full 64 k context. But the operating point on the trace is shifted by prefix cache hits: | reuse | new tokens | prefill time @ ~7,000 tok/s | |---|---|---| | 0 % (turn 1 cold) | 64,000 | ~9 s | | 71 % (trace average) | 18,600 | ~2.6 s | | 95 % (deep multi-turn) | 3,200 | ~0.5 s | Average-case prefill is ~2.6 s of compute. With 8 GPUs and peak QPS 1.6, each GPU sees ~0.3 s of prefill work per second of wall-clock. Chunked prefill in vLLM slices this into ~8 k-token chunks of ~50–100 ms each, then yields to decode. The decode-side disturbance per HEAVY request is on the order of "a few hundred ms of stretched decode," not "seconds of stalled decode." PD separation, in its best case, eliminates this disturbance. The *ceiling* on the benefit is therefore: hundreds of ms per HEAVY request. This budget has to absorb everything PD separation costs. ## Layer 3: PD separation relocates compute; it does not accelerate it A prefill kernel does the same FLOPs no matter which GPU runs it. PD separation's potential acceleration vectors are only two: 1. **Larger prefill batch** → better SM utilization for the prefill MMA kernels. 2. **No chunked-prefill yield to decode** → no overhead per chunk handoff. Both are quantitatively negligible in this regime: 1. At peak QPS 1.6 with ~2.6 s of prefill per request, *system-wide* prefill concurrency averages to ~4 active prefills. A 4P split sees ~1 prefill per GPU at any moment, so batching gains are zero. A 6P split makes it worse, not better. The roofline ceiling of 148 TFLOPS is already reachable at batch=1 for sequences this long. 2. Chunked prefill's per-chunk overhead is dominated by scheduler tick time (≈ 1–2 ms), not the chunk transition. Removing it saves single percentages of prefill time. So the *speedup side* of PD separation is bounded by the few-hundred-ms budget from Layer 2 and contains no hidden upside. ## Layer 4: the costs of PD separation are workload-scaled PD separation adds two costs, both of which scale *up* with workload size: 1. **KV transfer over the network.** Mooncake transfers KV block-by-block after the full prefill completes (no layer-wise pipeline; see `analysis/elastic_hypotheses.md` H5). Empirically, transfer takes ~1.1 s p50 for HEAVY requests (~40 k tokens of KV ≈ 3.8 GB at 96 KB/token), with tail extending to 18–30 s. Transfer time grows with context length. 2. **Decode-side KV concentration.** All decode work is funneled onto a subset of GPUs (4 of 8 in 4P+4D, 2 of 8 in 6P+2D). Per-D-instance KV *demand* therefore scales by N_total / N_D. This is the killer cost; Layer 5 quantifies it. Both costs scale linearly or worse with per-request KV footprint. KV footprint, in turn, scales linearly with input length. So PD separation gets *worse* exactly along the axis (long context) where the workload is moving. ## Layer 5: the decode-side KV memory wall (the actual mechanism) Visualized in `figures/fig_kv_memory_wall.pdf`. The model is simple and its constants are auditable in `scripts/plot_kv_memory_wall.py`: ``` per-D occupancy = (concurrent_decode × KV_per_req) / (N_D × KV_pool_per_GPU) ``` with: - `KV_per_req` = `seqlen × 96 KB/token` for Qwen3-30B-A3B (2 × 4 kv-heads × 128 head-dim × 2 bytes × 48 layers = 96 KB/tok) - `KV_pool_per_GPU` ≈ 28 GB (96 GB H20 HBM minus weights and activations) - `concurrent_decode` ≈ 8 at steady state (peak QPS 1.6 × mean decode duration ~5 s under Combined) Plug in the trace's input distribution from `figures/fig_c1a_io_cdf.pdf`: | operating point | KV/req | Combined (N_D=8) | 4P+4D (N_D=4) | 6P+2D (N_D=2) | |---|---|---|---|---| | chatbot avg (2 k) | 197 MB | 0.7 % | 1.4 % | 2.7 % | | **agentic avg (33.6 k)** | **3.3 GB** | **12 %** | **23 %** | **46 %** | | **agentic p90 (101 k)** | **9.9 GB** | **35 %** | **69 %** | **138 %** ⚠ | | **agentic p99 (132 k)** | **13.0 GB** | **45 %** | **90 %** ⚠ | **181 %** ⚠ | vLLM's scheduler stops admitting new requests at ~90 % KV pool occupancy, so anything above the wall translates directly to queueing. Two consequences fall out of this table: 1. **PD-sep with even a 4P+4D split breaches the wall at p99 context.** p99 alone is ~1 % of requests but holds the GPU for tens of seconds of decode, so its KV stays resident; over a long enough window the wall gets hit even from the tail. With 6P+2D the wall is breached well before p90. 2. **For chatbot, the entire table sits under 3 %.** PD separation never approaches the wall because chatbot per-request KV is 15× smaller. This is the assumption DistServe inherited from its target workload, and the assumption that silently breaks under agentic. The empirical KV occupancy on the 6P+2D run was 97 % (`analysis/pd_separation_analysis.md` §3.3) — the model and the measurement agree to within the resolution of the steady-state assumption. ## Layer 5b: empirical refinement — the bottleneck side depends on the P:D split The model above predicts D-side saturation. The 6P+2D run in `analysis/pd_separation_analysis.md` §3.3 is consistent with it. But the new 4P+4D run (`outputs/pd_matrix/pdsep-4p4d_cudagraph_seed1/`, captured during this section's experiment matrix) tells a richer story. Empirical numbers (combined-ca vs both PD-sep splits, same trace, all cudagraph, `figures/fig_c2_pdsep_vs_combined.pdf`): | metric | combined-ca (N=3) | pdsep-4p4d (N=1) | pdsep-6p2d (N=1) | |---|---|---|---| | success | 99.5 % | **52 %** (444/850) | **68 %** (574/850) | | TTFT p50 | 0.91 s | **62.8 s** (69×) | **51.1 s** (56×) | | TTFT p90 | 12.7 s | **491 s** (39×) | **400 s** (31×) | | TPOT p90 | 0.027 s | 0.013 s (-52 %) | 0.020 s (-26 %) | | E2E p50 | 2.5 s | **65.1 s** (26×) | **53.4 s** (21×) | | wall clock | 944 s | 7558 s (8×) | 3693 s (3.9×) | The per-stage TTFT decomposition (`figures/fig_c4_ttft_stacked.pdf`) shows that for both PD-sep splits **>97 % of TTFT is P-side prefill compute** (65.6 s / 66.2 s in 4p4d; 43.1 s / 44.3 s in 6p2d). D-side wait + first token is at most 1.2 s in either config. The KV-utilization time-series (`figures/fig_c3_kv_timeseries.pdf`) tells the full story: - **combined-ca**: 8 GPUs oscillate 0–98 %, peaks bursty and short - **pdsep-4p4d**: P-instances (orange) pinned at 85–100 % the entire 2-hour run; D-instances (red) bounce between 10 % and 50 % — only P side hits the wall - **pdsep-6p2d**: **both** sides pinned near 100 % the entire run (per-instance peaks 99–100 % across all 8). P-side fills because D back-pressures (D can't free KV slots fast enough → P can't hand off → P-side KV accumulates). This refines Layer 5: PD separation hits a memory wall on whichever side has fewer GPUs, and at extreme splits it co-saturates both sides through D-back-pressure. ### Why P-side fills in 4P+4D Two effects combine on P: 1. **Compute concentration.** Combined spreads prefill across 8 GPUs; 4P+4D over 4. Per-P-GPU prefill load is 2× the per-Combined-GPU load. With chunked prefill, multiple in-flight prefills compete for the scheduler. 2. **KV residency on P.** Mooncake does block-by-block transfer *after* the full prefill completes. Until D pulls and acknowledges every block, the completed-but-not-yet-transferred KV sits in P's pool — on top of all the partially-prefilled in-flight KV. Many concurrent 33–132 k contexts overwhelm a single 28 GB pool. This is the same memory-wall mechanism Layer 5 described, but on the *prefill* side. The Layer 5 analytical model in `figures/fig_kv_memory_wall.pdf` accounted only for *decode-side* KV demand. The full model is: ``` P-side occupancy = (in_flight_prefills × KV_per_req) / (N_P × pool) D-side occupancy = (concurrent_decode × KV_per_req) / (N_D × pool) ``` Whichever side hits the wall first becomes the back-pressure source. 4P+4D's P-side fills first because 4 GPUs are doing 8 GPUs' worth of prefill. 6P+2D's D-side hits the wall (4× concentration), which then back-pressures P (no slots to hand off into) until P also fills. In either case, *some* side blocks — which is why PD separation regresses across the P:D ratios we tested. ### Updated falsifiable condition The condition for PD separation to *not* regress is now two-sided: ``` max( in_flight_prefills × KV_per_req / (N_P × pool), concurrent_decode × KV_per_req / (N_D × pool) ) < 1 ``` For chatbot workloads (KV/req ≈ 200 MB), this holds easily on either side. For agentic with KV/req ≈ 3.3 GB on average and 10–13 GB at the tail, both terms cross 1 well below the chosen N_P or N_D. Followups: re-render `fig_kv_memory_wall.pdf` to show both P and D curves once the 6P+2D run lands, with the empirical P-side and D-side peaks marked. ## Layer 6: the DistServe / Splitwise assumption that silently breaks To formalize: the regime in which PD separation pays is bounded by both a roofline condition *and* a memory-capacity condition: | Condition | Form | Chatbot | Agentic | |---|---|---|---| | Prefill is compute-bound | AI_prefill ≫ ridge | ✓ | ✓ | | Decode is memory-bound | AI_decode ≪ ridge | ✓ | ✓ | | Per-D-instance KV demand fits | concurrent × KV/req / (N_D × pool) < 1 | ✓ (≪ 1) | ✗ (>1 at p90+) | | KV transfer time ≪ saved interference | transfer_s ≪ saved_decode_stall_s | ✓ (KV is MB) | ✗ (KV is GB) | DistServe and Splitwise hold all four conditions implicitly in their short-context regime. Agentic violates the bottom two. Both violations have the same root cause: per-request KV footprint is 15–60× larger. This is the falsifiable claim of the section: **PD separation pays iff per-request KV footprint × decode concurrency stays well below the per-D-instance HBM pool. When that condition fails — and it fails unavoidably for long-context agentic workloads — PD separation is net negative regardless of how compute-bound prefill is.** The roofline doesn't tell you whether you're inside this regime; only the memory budget does. --- ## What this means for the paper section The figures we already have support this argument: - `fig_c1a_io_cdf.pdf` — establishes the input-length distribution responsible for the large KV footprint (p50 33.5 k, p90 101 k, p99 132 k). - `fig_c1b_reuse.pdf` — establishes that 79 % of reuse is intra-session, i.e. the request set has long-lived sessions whose KV must sit in the pool through many decode steps. - `fig_c6_roofline.pdf` — establishes the prefill compute-bound fact. This is the apparent contradiction the section resolves. - `fig_kv_memory_wall.pdf` — establishes the resolution: the memory budget is what governs PD separation's viability, not the roofline. - `fig_c7_routing_lever.pdf` — establishes that cache-aware routing recovers most of the benefit PD separation promises, without paying the memory-wall cost. Missing for a rigorous re-grounding (deferred until the cudagraph re-run matrix lands): - Per-step decode KV utilization time-series from a live PD-sep run (currently inferred from a single log snapshot of "Running: 0, Waiting: 6, KV cache: 97.1 %"). This would *directly* show the memory wall being hit instead of relying on the steady-state model. - Per-request TTFT stacked breakdown (prefill, KV transfer, decode-side wait) on the new trace; currently `analysis/pd_separation_analysis.md` §3.3 has it on the old methodology. - CUDA-graph ablation: with `--enforce-eager` removed, PD-sep's D-node could in principle close some of the per-step decode latency gap. The Layer 5 model is gate-independent — wall demand grows with concurrency, not per-step latency — so this should not change the conclusion. But the section needs the measurement to say so honestly. The 4 h cudagraph experiment matrix (Combined / PD-sep × eager / cudagraph × 3 seeds) on `traces/w600_r0.0015_st30.jsonl` would settle those three items.