agentic-kvc/analysis/pd_sep_paper_section/system_analysis.md

# Why does PD separation fail on agentic workloads even though prefill stays compute-bound?

This is the central paradox of the section. The roofline in
`figures/fig_c6_roofline.pdf` shows prefill is compute-bound at every
realistic reuse level — at 95 % cache reuse, arithmetic intensity is still
≈ 4,500 FLOP/byte, more than two orders of magnitude above the H20 ridge of
37. The DistServe / Splitwise argument follows directly from that fact:
"prefill is compute-heavy, decode is memory-bound, so isolate them onto
different GPUs and specialize each."

Yet on this workload, single-machine PD separation regresses TTFT by 72 %
(REPORT.md §3.1) and saturates the decode-side KV pool at 97 % occupancy.
This document explains, layer by layer, why a true premise (compute-bound)
does not imply the conclusion (PD separation pays). All five layers are
backed by either figures in this directory or measurements in
`analysis/pd_separation_analysis.md`.

The short answer: **the roofline tells you about the per-kernel efficiency
of prefill. PD separation is a decision about a whole serving system. The
gap between those two scales is where DistServe's argument loses force —
and where agentic workloads, with their very large per-request KV
footprint, push the system past a memory-capacity wall that chatbot
workloads never reach.**

---

## Layer 1: compute-bound ≠ "needs dedicated GPUs"

Roofline analysis classifies a *kernel*. It answers the question, "given
that this kernel is running, is it bottlenecked by FLOP rate or by HBM
bandwidth?" — it does **not** answer:

- how long the kernel takes in wall-clock terms,
- whether two kernels can profitably share a GPU,
- whether moving the kernel to a different GPU makes it faster.

PD separation needs the second and third answers, not the first. A 50 ms
compute-bound prefill burst can perfectly well coexist with decode steps
on the same GPU; you lose at most a fraction of a decode step's latency
per chunk. Co-location only fails when prefill bursts grow long enough
that decode requests starve.

The DistServe paper's roofline argument is a *necessary* condition
("prefill *can* be compute-bound, so dedicating GPUs is *not wasted*"). It
is **not** a *sufficient* condition ("therefore dedicated GPUs pay").

## Layer 2: in agentic, absolute prefill work after cache hit is small

The roofline is computed in `figures/fig_c6_roofline.pdf` at a full 64 k
context. But the operating point on the trace is shifted by prefix cache
hits:

| reuse | new tokens | prefill time @ ~7,000 tok/s |
|---|---|---|
| 0 % (turn 1 cold) | 64,000 | ~9 s |
| 71 % (trace average) | 18,600 | ~2.6 s |
| 95 % (deep multi-turn) | 3,200 | ~0.5 s |

Average-case prefill is ~2.6 s of compute. With 8 GPUs and peak QPS 1.6,
each GPU sees ~0.3 s of prefill work per second of wall-clock. Chunked
prefill in vLLM slices this into ~8 k-token chunks of ~50–100 ms each, then
yields to decode. The decode-side disturbance per HEAVY request is on the
order of "a few hundred ms of stretched decode," not "seconds of stalled
decode."

PD separation, in its best case, eliminates this disturbance. The
*ceiling* on the benefit is therefore: hundreds of ms per HEAVY request.
This budget has to absorb everything PD separation costs.

## Layer 3: PD separation relocates compute; it does not accelerate it

A prefill kernel does the same FLOPs no matter which GPU runs it. PD
separation's potential acceleration vectors are only two:

1. **Larger prefill batch** → better SM utilization for the prefill MMA
   kernels.
2. **No chunked-prefill yield to decode** → no overhead per chunk handoff.

Both are quantitatively negligible in this regime:

1. At peak QPS 1.6 with ~2.6 s of prefill per request, *system-wide*
   prefill concurrency averages to ~4 active prefills. A 4P split sees ~1
   prefill per GPU at any moment, so batching gains are zero. A 6P split
   makes it worse, not better. The roofline ceiling of 148 TFLOPS is
   already reachable at batch=1 for sequences this long.

2. Chunked prefill's per-chunk overhead is dominated by scheduler tick
   time (≈ 1–2 ms), not the chunk transition. Removing it saves single
   percentages of prefill time.

So the *speedup side* of PD separation is bounded by the few-hundred-ms
budget from Layer 2 and contains no hidden upside.

## Layer 4: the costs of PD separation are workload-scaled

PD separation adds two costs, both of which scale *up* with workload size:

1. **KV transfer over the network.** Mooncake transfers KV block-by-block
   after the full prefill completes (no layer-wise pipeline; see
   `analysis/elastic_hypotheses.md` H5). Empirically, transfer takes
   ~1.1 s p50 for HEAVY requests (~40 k tokens of KV ≈ 3.8 GB at
   96 KB/token), with tail extending to 18–30 s. Transfer time grows with
   context length.

2. **Decode-side KV concentration.** All decode work is funneled onto a
   subset of GPUs (4 of 8 in 4P+4D, 2 of 8 in 6P+2D). Per-D-instance KV
   *demand* therefore scales by N_total / N_D. This is the killer cost;
   Layer 5 quantifies it.

Both costs scale linearly or worse with per-request KV footprint. KV
footprint, in turn, scales linearly with input length. So PD separation
gets *worse* exactly along the axis (long context) where the workload is
moving.

## Layer 5: the decode-side KV memory wall (the actual mechanism)

Visualized in `figures/fig_kv_memory_wall.pdf`. The model is simple and
its constants are auditable in
`scripts/plot_kv_memory_wall.py`:

```
per-D occupancy = (concurrent_decode × KV_per_req) / (N_D × KV_pool_per_GPU)
```

with:

- `KV_per_req` = `seqlen × 96 KB/token` for Qwen3-30B-A3B
  (2 × 4 kv-heads × 128 head-dim × 2 bytes × 48 layers = 96 KB/tok)
- `KV_pool_per_GPU` ≈ 28 GB (96 GB H20 HBM minus weights and activations)
- `concurrent_decode` ≈ 8 at steady state (peak QPS 1.6 × mean decode
  duration ~5 s under Combined)

Plug in the trace's input distribution from
`figures/fig_c1a_io_cdf.pdf`:

| operating point | KV/req | Combined (N_D=8) | 4P+4D (N_D=4) | 6P+2D (N_D=2) |
|---|---|---|---|---|
| chatbot avg (2 k) | 197 MB | 0.7 % | 1.4 % | 2.7 % |
| **agentic avg (33.6 k)** | **3.3 GB** | **12 %** | **23 %** | **46 %** |
| **agentic p90 (101 k)** | **9.9 GB** | **35 %** | **69 %** | **138 %** ⚠ |
| **agentic p99 (132 k)** | **13.0 GB** | **45 %** | **90 %** ⚠ | **181 %** ⚠ |

vLLM's scheduler stops admitting new requests at ~90 % KV pool occupancy,
so anything above the wall translates directly to queueing.

Two consequences fall out of this table:

1. **PD-sep with even a 4P+4D split breaches the wall at p99 context.**
   p99 alone is ~1 % of requests but holds the GPU for tens of seconds of
   decode, so its KV stays resident; over a long enough window the wall
   gets hit even from the tail. With 6P+2D the wall is breached well
   before p90.

2. **For chatbot, the entire table sits under 3 %.** PD separation never
   approaches the wall because chatbot per-request KV is 15× smaller. This
   is the assumption DistServe inherited from its target workload, and
   the assumption that silently breaks under agentic.

The empirical KV occupancy on the 6P+2D run was 97 %
(`analysis/pd_separation_analysis.md` §3.3) — the model and the
measurement agree to within the resolution of the steady-state assumption.

## Layer 5b: empirical refinement — the bottleneck side depends on the P:D split

The model above predicts D-side saturation. The 6P+2D run in
`analysis/pd_separation_analysis.md` §3.3 is consistent with it. But the
new 4P+4D run (`outputs/pd_matrix/pdsep-4p4d_cudagraph_seed1/`, captured
during this section's experiment matrix) tells a richer story.

Empirical numbers (combined-ca vs both PD-sep splits, same trace, all
cudagraph, `figures/fig_c2_pdsep_vs_combined.pdf`):

| metric | combined-ca (N=3) | pdsep-4p4d (N=1) | pdsep-6p2d (N=1) |
|---|---|---|---|
| success | 99.5 % | **52 %** (444/850) | **68 %** (574/850) |
| TTFT p50 | 0.91 s | **62.8 s** (69×) | **51.1 s** (56×) |
| TTFT p90 | 12.7 s | **491 s** (39×) | **400 s** (31×) |
| TPOT p90 | 0.027 s | 0.013 s (-52 %) | 0.020 s (-26 %) |
| E2E p50 | 2.5 s | **65.1 s** (26×) | **53.4 s** (21×) |
| wall clock | 944 s | 7558 s (8×) | 3693 s (3.9×) |

The per-stage TTFT decomposition (`figures/fig_c4_ttft_stacked.pdf`)
shows that for both PD-sep splits **>97 % of TTFT is P-side prefill
compute** (65.6 s / 66.2 s in 4p4d; 43.1 s / 44.3 s in 6p2d). D-side
wait + first token is at most 1.2 s in either config.

The KV-utilization time-series (`figures/fig_c3_kv_timeseries.pdf`)
tells the full story:

- **combined-ca**: 8 GPUs oscillate 0–98 %, peaks bursty and short
- **pdsep-4p4d**: P-instances (orange) pinned at 85–100 % the entire
  2-hour run; D-instances (red) bounce between 10 % and 50 % — only
  P side hits the wall
- **pdsep-6p2d**: **both** sides pinned near 100 % the entire run
  (per-instance peaks 99–100 % across all 8). P-side fills because D
  back-pressures (D can't free KV slots fast enough → P can't
  hand off → P-side KV accumulates).

This refines Layer 5: PD separation hits a memory wall on whichever
side has fewer GPUs, and at extreme splits it co-saturates both sides
through D-back-pressure.

### Why P-side fills in 4P+4D

Two effects combine on P:

1. **Compute concentration.** Combined spreads prefill across 8 GPUs;
   4P+4D over 4. Per-P-GPU prefill load is 2× the per-Combined-GPU load.
   With chunked prefill, multiple in-flight prefills compete for the
   scheduler.

2. **KV residency on P.** Mooncake does block-by-block transfer *after*
   the full prefill completes. Until D pulls and acknowledges every
   block, the completed-but-not-yet-transferred KV sits in P's pool —
   on top of all the partially-prefilled in-flight KV. Many concurrent
   33–132 k contexts overwhelm a single 28 GB pool.

This is the same memory-wall mechanism Layer 5 described, but on the
*prefill* side. The Layer 5 analytical model in
`figures/fig_kv_memory_wall.pdf` accounted only for *decode-side* KV
demand. The full model is:

```
P-side occupancy = (in_flight_prefills × KV_per_req) / (N_P × pool)
D-side occupancy = (concurrent_decode × KV_per_req) / (N_D × pool)
```

Whichever side hits the wall first becomes the back-pressure source.
4P+4D's P-side fills first because 4 GPUs are doing 8 GPUs' worth of
prefill. 6P+2D's D-side hits the wall (4× concentration), which then
back-pressures P (no slots to hand off into) until P also fills. In
either case, *some* side blocks — which is why PD separation regresses
across the P:D ratios we tested.

### Updated falsifiable condition

The condition for PD separation to *not* regress is now two-sided:

```
max(
  in_flight_prefills × KV_per_req / (N_P × pool),
  concurrent_decode  × KV_per_req / (N_D × pool)
) < 1
```

For chatbot workloads (KV/req ≈ 200 MB), this holds easily on either
side. For agentic with KV/req ≈ 3.3 GB on average and 10–13 GB at the
tail, both terms cross 1 well below the chosen N_P or N_D.

Followups: re-render `fig_kv_memory_wall.pdf` to show both P and D
curves once the 6P+2D run lands, with the empirical P-side and D-side
peaks marked.

## Layer 6: the DistServe / Splitwise assumption that silently breaks

To formalize: the regime in which PD separation pays is bounded by both a
roofline condition *and* a memory-capacity condition:

| Condition | Form | Chatbot | Agentic |
|---|---|---|---|
| Prefill is compute-bound | AI_prefill ≫ ridge | ✓ | ✓ |
| Decode is memory-bound | AI_decode ≪ ridge | ✓ | ✓ |
| Per-D-instance KV demand fits | concurrent × KV/req / (N_D × pool) < 1 | ✓ (≪ 1) | ✗ (>1 at p90+) |
| KV transfer time ≪ saved interference | transfer_s ≪ saved_decode_stall_s | ✓ (KV is MB) | ✗ (KV is GB) |

DistServe and Splitwise hold all four conditions implicitly in their
short-context regime. Agentic violates the bottom two. Both violations
have the same root cause: per-request KV footprint is 15–60× larger.

This is the falsifiable claim of the section: **PD separation pays iff
per-request KV footprint × decode concurrency stays well below the
per-D-instance HBM pool. When that condition fails — and it fails
unavoidably for long-context agentic workloads — PD separation is net
negative regardless of how compute-bound prefill is.**

The roofline doesn't tell you whether you're inside this regime; only the
memory budget does.

---

## What this means for the paper section

The figures we already have support this argument:

- `fig_c1a_io_cdf.pdf` — establishes the input-length distribution
  responsible for the large KV footprint (p50 33.5 k, p90 101 k, p99
  132 k).
- `fig_c1b_reuse.pdf` — establishes that 79 % of reuse is intra-session,
  i.e. the request set has long-lived sessions whose KV must sit in the
  pool through many decode steps.
- `fig_c6_roofline.pdf` — establishes the prefill compute-bound fact.
  This is the apparent contradiction the section resolves.
- `fig_kv_memory_wall.pdf` — establishes the resolution: the memory budget
  is what governs PD separation's viability, not the roofline.
- `fig_c7_routing_lever.pdf` — establishes that cache-aware routing
  recovers most of the benefit PD separation promises, without paying the
  memory-wall cost.

Missing for a rigorous re-grounding (deferred until the cudagraph re-run
matrix lands):

- Per-step decode KV utilization time-series from a live PD-sep run
  (currently inferred from a single log snapshot of "Running: 0, Waiting:
  6, KV cache: 97.1 %"). This would *directly* show the memory wall being
  hit instead of relying on the steady-state model.
- Per-request TTFT stacked breakdown (prefill, KV transfer, decode-side
  wait) on the new trace; currently `analysis/pd_separation_analysis.md`
  §3.3 has it on the old methodology.
- CUDA-graph ablation: with `--enforce-eager` removed, PD-sep's D-node
  could in principle close some of the per-step decode latency gap. The
  Layer 5 model is gate-independent — wall demand grows with concurrency,
  not per-step latency — so this should not change the conclusion. But
  the section needs the measurement to say so honestly.

The 4 h cudagraph experiment matrix (Combined / PD-sep × eager /
cudagraph × 3 seeds) on `traces/w600_r0.0015_st30.jsonl` would settle
those three items.