Gahow Wang
4028c587b1
Adds the system-level argument resolving the roofline/PD-sep paradox. Even at 95% cache reuse prefill stays compute-bound (the C6 roofline fact), yet PD separation regresses TTFT 72%. The new system_analysis.md walks through six layers showing why the roofline claim is necessary but not sufficient, with the falsifiable condition being decode-side KV memory budget: concurrent_decode * KV_per_req / (N_D * HBM_pool). For chatbot this ratio is << 1 at any layout; for agentic at p90+ context it goes >> 1 under 4P+4D and 6P+2D, predicting the empirical 97% decode KV occupancy. fig_kv_memory_wall.pdf visualizes the model with audit-able constants; fig_c1a/b ground the per-request KV-size inputs in the actual sampled trace (input p50=33.5k, p90=101k, intra-session reuse 79.2%). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
227 lines
10 KiB
Markdown
227 lines
10 KiB
Markdown
# Why does PD separation fail on agentic workloads even though prefill stays compute-bound?
|
||
|
||
This is the central paradox of the section. The roofline in
|
||
`figures/fig_c6_roofline.pdf` shows prefill is compute-bound at every
|
||
realistic reuse level — at 95 % cache reuse, arithmetic intensity is still
|
||
≈ 4,500 FLOP/byte, more than two orders of magnitude above the H20 ridge of
|
||
37. The DistServe / Splitwise argument follows directly from that fact:
|
||
"prefill is compute-heavy, decode is memory-bound, so isolate them onto
|
||
different GPUs and specialize each."
|
||
|
||
Yet on this workload, single-machine PD separation regresses TTFT by 72 %
|
||
(REPORT.md §3.1) and saturates the decode-side KV pool at 97 % occupancy.
|
||
This document explains, layer by layer, why a true premise (compute-bound)
|
||
does not imply the conclusion (PD separation pays). All five layers are
|
||
backed by either figures in this directory or measurements in
|
||
`analysis/pd_separation_analysis.md`.
|
||
|
||
The short answer: **the roofline tells you about the per-kernel efficiency
|
||
of prefill. PD separation is a decision about a whole serving system. The
|
||
gap between those two scales is where DistServe's argument loses force —
|
||
and where agentic workloads, with their very large per-request KV
|
||
footprint, push the system past a memory-capacity wall that chatbot
|
||
workloads never reach.**
|
||
|
||
---
|
||
|
||
## Layer 1: compute-bound ≠ "needs dedicated GPUs"
|
||
|
||
Roofline analysis classifies a *kernel*. It answers the question, "given
|
||
that this kernel is running, is it bottlenecked by FLOP rate or by HBM
|
||
bandwidth?" — it does **not** answer:
|
||
|
||
- how long the kernel takes in wall-clock terms,
|
||
- whether two kernels can profitably share a GPU,
|
||
- whether moving the kernel to a different GPU makes it faster.
|
||
|
||
PD separation needs the second and third answers, not the first. A 50 ms
|
||
compute-bound prefill burst can perfectly well coexist with decode steps
|
||
on the same GPU; you lose at most a fraction of a decode step's latency
|
||
per chunk. Co-location only fails when prefill bursts grow long enough
|
||
that decode requests starve.
|
||
|
||
The DistServe paper's roofline argument is a *necessary* condition
|
||
("prefill *can* be compute-bound, so dedicating GPUs is *not wasted*"). It
|
||
is **not** a *sufficient* condition ("therefore dedicated GPUs pay").
|
||
|
||
## Layer 2: in agentic, absolute prefill work after cache hit is small
|
||
|
||
The roofline is computed in `figures/fig_c6_roofline.pdf` at a full 64 k
|
||
context. But the operating point on the trace is shifted by prefix cache
|
||
hits:
|
||
|
||
| reuse | new tokens | prefill time @ ~7,000 tok/s |
|
||
|---|---|---|
|
||
| 0 % (turn 1 cold) | 64,000 | ~9 s |
|
||
| 71 % (trace average) | 18,600 | ~2.6 s |
|
||
| 95 % (deep multi-turn) | 3,200 | ~0.5 s |
|
||
|
||
Average-case prefill is ~2.6 s of compute. With 8 GPUs and peak QPS 1.6,
|
||
each GPU sees ~0.3 s of prefill work per second of wall-clock. Chunked
|
||
prefill in vLLM slices this into ~8 k-token chunks of ~50–100 ms each, then
|
||
yields to decode. The decode-side disturbance per HEAVY request is on the
|
||
order of "a few hundred ms of stretched decode," not "seconds of stalled
|
||
decode."
|
||
|
||
PD separation, in its best case, eliminates this disturbance. The
|
||
*ceiling* on the benefit is therefore: hundreds of ms per HEAVY request.
|
||
This budget has to absorb everything PD separation costs.
|
||
|
||
## Layer 3: PD separation relocates compute; it does not accelerate it
|
||
|
||
A prefill kernel does the same FLOPs no matter which GPU runs it. PD
|
||
separation's potential acceleration vectors are only two:
|
||
|
||
1. **Larger prefill batch** → better SM utilization for the prefill MMA
|
||
kernels.
|
||
2. **No chunked-prefill yield to decode** → no overhead per chunk handoff.
|
||
|
||
Both are quantitatively negligible in this regime:
|
||
|
||
1. At peak QPS 1.6 with ~2.6 s of prefill per request, *system-wide*
|
||
prefill concurrency averages to ~4 active prefills. A 4P split sees ~1
|
||
prefill per GPU at any moment, so batching gains are zero. A 6P split
|
||
makes it worse, not better. The roofline ceiling of 148 TFLOPS is
|
||
already reachable at batch=1 for sequences this long.
|
||
|
||
2. Chunked prefill's per-chunk overhead is dominated by scheduler tick
|
||
time (≈ 1–2 ms), not the chunk transition. Removing it saves single
|
||
percentages of prefill time.
|
||
|
||
So the *speedup side* of PD separation is bounded by the few-hundred-ms
|
||
budget from Layer 2 and contains no hidden upside.
|
||
|
||
## Layer 4: the costs of PD separation are workload-scaled
|
||
|
||
PD separation adds two costs, both of which scale *up* with workload size:
|
||
|
||
1. **KV transfer over the network.** Mooncake transfers KV block-by-block
|
||
after the full prefill completes (no layer-wise pipeline; see
|
||
`analysis/elastic_hypotheses.md` H5). Empirically, transfer takes
|
||
~1.1 s p50 for HEAVY requests (~40 k tokens of KV ≈ 3.8 GB at
|
||
96 KB/token), with tail extending to 18–30 s. Transfer time grows with
|
||
context length.
|
||
|
||
2. **Decode-side KV concentration.** All decode work is funneled onto a
|
||
subset of GPUs (4 of 8 in 4P+4D, 2 of 8 in 6P+2D). Per-D-instance KV
|
||
*demand* therefore scales by N_total / N_D. This is the killer cost;
|
||
Layer 5 quantifies it.
|
||
|
||
Both costs scale linearly or worse with per-request KV footprint. KV
|
||
footprint, in turn, scales linearly with input length. So PD separation
|
||
gets *worse* exactly along the axis (long context) where the workload is
|
||
moving.
|
||
|
||
## Layer 5: the decode-side KV memory wall (the actual mechanism)
|
||
|
||
Visualized in `figures/fig_kv_memory_wall.pdf`. The model is simple and
|
||
its constants are auditable in
|
||
`scripts/plot_kv_memory_wall.py`:
|
||
|
||
```
|
||
per-D occupancy = (concurrent_decode × KV_per_req) / (N_D × KV_pool_per_GPU)
|
||
```
|
||
|
||
with:
|
||
|
||
- `KV_per_req` = `seqlen × 96 KB/token` for Qwen3-30B-A3B
|
||
(2 × 4 kv-heads × 128 head-dim × 2 bytes × 48 layers = 96 KB/tok)
|
||
- `KV_pool_per_GPU` ≈ 28 GB (96 GB H20 HBM minus weights and activations)
|
||
- `concurrent_decode` ≈ 8 at steady state (peak QPS 1.6 × mean decode
|
||
duration ~5 s under Combined)
|
||
|
||
Plug in the trace's input distribution from
|
||
`figures/fig_c1a_io_cdf.pdf`:
|
||
|
||
| operating point | KV/req | Combined (N_D=8) | 4P+4D (N_D=4) | 6P+2D (N_D=2) |
|
||
|---|---|---|---|---|
|
||
| chatbot avg (2 k) | 197 MB | 0.7 % | 1.4 % | 2.7 % |
|
||
| **agentic avg (33.6 k)** | **3.3 GB** | **12 %** | **23 %** | **46 %** |
|
||
| **agentic p90 (101 k)** | **9.9 GB** | **35 %** | **69 %** | **138 %** ⚠ |
|
||
| **agentic p99 (132 k)** | **13.0 GB** | **45 %** | **90 %** ⚠ | **181 %** ⚠ |
|
||
|
||
vLLM's scheduler stops admitting new requests at ~90 % KV pool occupancy,
|
||
so anything above the wall translates directly to queueing.
|
||
|
||
Two consequences fall out of this table:
|
||
|
||
1. **PD-sep with even a 4P+4D split breaches the wall at p99 context.**
|
||
p99 alone is ~1 % of requests but holds the GPU for tens of seconds of
|
||
decode, so its KV stays resident; over a long enough window the wall
|
||
gets hit even from the tail. With 6P+2D the wall is breached well
|
||
before p90.
|
||
|
||
2. **For chatbot, the entire table sits under 3 %.** PD separation never
|
||
approaches the wall because chatbot per-request KV is 15× smaller. This
|
||
is the assumption DistServe inherited from its target workload, and
|
||
the assumption that silently breaks under agentic.
|
||
|
||
The empirical KV occupancy on the 6P+2D run was 97 %
|
||
(`analysis/pd_separation_analysis.md` §3.3) — the model and the
|
||
measurement agree to within the resolution of the steady-state assumption.
|
||
|
||
## Layer 6: the DistServe / Splitwise assumption that silently breaks
|
||
|
||
To formalize: the regime in which PD separation pays is bounded by both a
|
||
roofline condition *and* a memory-capacity condition:
|
||
|
||
| Condition | Form | Chatbot | Agentic |
|
||
|---|---|---|---|
|
||
| Prefill is compute-bound | AI_prefill ≫ ridge | ✓ | ✓ |
|
||
| Decode is memory-bound | AI_decode ≪ ridge | ✓ | ✓ |
|
||
| Per-D-instance KV demand fits | concurrent × KV/req / (N_D × pool) < 1 | ✓ (≪ 1) | ✗ (>1 at p90+) |
|
||
| KV transfer time ≪ saved interference | transfer_s ≪ saved_decode_stall_s | ✓ (KV is MB) | ✗ (KV is GB) |
|
||
|
||
DistServe and Splitwise hold all four conditions implicitly in their
|
||
short-context regime. Agentic violates the bottom two. Both violations
|
||
have the same root cause: per-request KV footprint is 15–60× larger.
|
||
|
||
This is the falsifiable claim of the section: **PD separation pays iff
|
||
per-request KV footprint × decode concurrency stays well below the
|
||
per-D-instance HBM pool. When that condition fails — and it fails
|
||
unavoidably for long-context agentic workloads — PD separation is net
|
||
negative regardless of how compute-bound prefill is.**
|
||
|
||
The roofline doesn't tell you whether you're inside this regime; only the
|
||
memory budget does.
|
||
|
||
---
|
||
|
||
## What this means for the paper section
|
||
|
||
The figures we already have support this argument:
|
||
|
||
- `fig_c1a_io_cdf.pdf` — establishes the input-length distribution
|
||
responsible for the large KV footprint (p50 33.5 k, p90 101 k, p99
|
||
132 k).
|
||
- `fig_c1b_reuse.pdf` — establishes that 79 % of reuse is intra-session,
|
||
i.e. the request set has long-lived sessions whose KV must sit in the
|
||
pool through many decode steps.
|
||
- `fig_c6_roofline.pdf` — establishes the prefill compute-bound fact.
|
||
This is the apparent contradiction the section resolves.
|
||
- `fig_kv_memory_wall.pdf` — establishes the resolution: the memory budget
|
||
is what governs PD separation's viability, not the roofline.
|
||
- `fig_c7_routing_lever.pdf` — establishes that cache-aware routing
|
||
recovers most of the benefit PD separation promises, without paying the
|
||
memory-wall cost.
|
||
|
||
Missing for a rigorous re-grounding (deferred until the cudagraph re-run
|
||
matrix lands):
|
||
|
||
- Per-step decode KV utilization time-series from a live PD-sep run
|
||
(currently inferred from a single log snapshot of "Running: 0, Waiting:
|
||
6, KV cache: 97.1 %"). This would *directly* show the memory wall being
|
||
hit instead of relying on the steady-state model.
|
||
- Per-request TTFT stacked breakdown (prefill, KV transfer, decode-side
|
||
wait) on the new trace; currently `analysis/pd_separation_analysis.md`
|
||
§3.3 has it on the old methodology.
|
||
- CUDA-graph ablation: with `--enforce-eager` removed, PD-sep's D-node
|
||
could in principle close some of the per-step decode latency gap. The
|
||
Layer 5 model is gate-independent — wall demand grows with concurrency,
|
||
not per-step latency — so this should not change the conclusion. But
|
||
the section needs the measurement to say so honestly.
|
||
|
||
The 4 h cudagraph experiment matrix (Combined / PD-sep × eager /
|
||
cudagraph × 3 seeds) on `traces/w600_r0.0015_st30.jsonl` would settle
|
||
those three items.
|