Gahow Wang
2e6a369046
Document the consumer EngineCore crash chain (D-pool 97% -> 112k-token KV transfer fails -> negative prompt-token counter -> prometheus ValueError -> engine dead -> cliff failure). Explains the round-robin 6P+2D rep variance (100/56/80%) as intermittent consumer death, and notes the counter-clamp patch needed to compare routing arms fairly. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
261 lines
12 KiB
Markdown
261 lines
12 KiB
Markdown
# PD-disaggregation under an agentic workload — does it work?
|
||
|
||
**Consolidated results doc.** Self-contained writeup of every PD-disagg
|
||
argument and experiment, with figures inline. For the live experiment TODO
|
||
list see [PD_DISAGG_INVESTIGATION.md](PD_DISAGG_INVESTIGATION.md).
|
||
|
||
Date: 2026-05-28 · Hardware: dash1, 8×GPU · Model: Qwen3-Coder-30B-A3B-Instruct
|
||
· vLLM 0.18.1 (V1, chunked-prefill on) · Mooncake 0.3.11 · Trace:
|
||
`w600_r0.0015_st30.jsonl` (1214 requests, agentic multi-turn).
|
||
|
||
---
|
||
|
||
## TL;DR (verdict)
|
||
|
||
**No static prefill/decode split beats 8-way colocation (8C) on this agentic
|
||
workload.** Every disaggregated ratio we tried is dominated by 8C on the
|
||
metric the user actually feels (TTFT, end-to-end latency, request
|
||
completion), and the failure *moves* with the ratio:
|
||
|
||
- **D-heavy bottleneck** (6P+2D, 4P+4D): the decode pool saturates (peak
|
||
**99.6% / 97.5%**) while the prefill pool sits at **~30%** — half the
|
||
cluster's KV is stranded on the wrong side.
|
||
- **P-heavy bottleneck** (2P+6D): the 2 prefill instances can't keep up,
|
||
the prefill pool jams at **99.7%**, **872 requests** pile up in the queue
|
||
and **91% of requests never complete**.
|
||
- **8C** keeps a single elastic pool that absorbs whichever phase is hot at
|
||
the moment → steady utilization **34%**, **100% completion**, fastest
|
||
wall-clock, best p50/p90 latency.
|
||
|
||
PD-disagg *does* deliver the phase-isolation win we predicted in MB1 — its
|
||
**TPOT is 10–35× cleaner** — but that win is swamped by TTFT inflation,
|
||
request loss, and a total collapse of prefix-cache reuse under the stock
|
||
round-robin router.
|
||
|
||
This is the empirical backing for the paper's claim: **agentic workloads
|
||
have time-varying P:D demand that no static partition can track; colocation
|
||
wins because its pool is elastic.** (H1 *and* H2 from the investigation doc,
|
||
unified by one mechanism.)
|
||
|
||
---
|
||
|
||
## 1. Why this experiment exists
|
||
|
||
Earlier cost accounting (MB1 phase-interference, MB2 KV-transfer cost) showed
|
||
that on the **phase-isolation axis alone**, PD-disagg actually *wins*: it
|
||
removes prefill→decode interference, and the transfer cost is small relative
|
||
to the interference it avoids. So "PD-disagg is bad for agentic" could not be
|
||
argued from phase isolation — we needed a system-level experiment that
|
||
measures the whole picture (queueing, pool capacity, cache reuse), not just
|
||
the isolated phase cost.
|
||
|
||
See [analysis/mb1](../../analysis/mb1) and [analysis/mb2](../../analysis/mb2)
|
||
for that accounting. This doc is the system-level answer.
|
||
|
||
---
|
||
|
||
## 2. Setup
|
||
|
||
| | |
|
||
|---|---|
|
||
| Configs | `8C` (8× kv_both colo), `6P+2D`, `4P+4D`, `2P+6D` (prefill+decode split) |
|
||
| PD routing | stock **round-robin** on both P and D (vLLM official `mooncake_connector_proxy`) |
|
||
| Trace | `w600_r0.0015_st30.jsonl`, 1214 requests, agentic multi-turn |
|
||
| Reps | 1 (rep1) for this analysis; the 3-rep sweep confirmed run-to-run consistency before we converged on rep1 for iteration speed |
|
||
| KV instrumentation | V1 scheduler patched to dump per-request KV block allocation every 100 ms per EngineCore (see `instrument_kv_snapshot.py`) |
|
||
|
||
8C is the fair baseline: 8 colocated instances, replayer round-robins across
|
||
them directly (no proxy). PD configs route through the proxy.
|
||
|
||
---
|
||
|
||
## 3. Headline result — no PD ratio beats 8C
|
||
|
||
All numbers are rep1.
|
||
|
||
| Metric | **8C** | 6P+2D | 4P+4D | 2P+6D |
|
||
|---|---|---|---|---|
|
||
| **completion** | **100%** | 100% | 100% | **9%** 💀 |
|
||
| wall-clock (drain trace) | **2994 s** | 3419 s | 4171 s | 5762 s |
|
||
| prefix-cache hit | **19.4%** | 0% | 0% | 0% |
|
||
| TTFT mean | **18.0 s** | 44.8 s | 70.0 s | 106.8 s |
|
||
| TTFT p50 | **7.0 s** | 41.0 s | 56.4 s | 23.6 s |
|
||
| TTFT p90 | **53.1 s** | 86.7 s | 153.1 s | 498 s |
|
||
| E2E p50 | **10.8 s** | 44.5 s | 59.5 s | 26.3 s |
|
||
| E2E p90 | **83.3 s** | 91.8 s | 157.1 s | 499 s |
|
||
|
||
|
||
|
||
> ⚠️ **Read the percentiles with the completion rate.** Latency percentiles
|
||
> are computed over *successful* requests only. 2P+6D's "p99 = 577 s" covers
|
||
> just the 9% that finished — the other 91% never returned, so its real
|
||
> experience is far worse than any latency bar suggests.
|
||
|
||
8C wins p50 by **4×** and p90 decisively. The only metric where a PD config
|
||
edges 8C is E2E **p99** (6P+2D 148 s vs 8C 194 s) — and that is the flip side
|
||
of the next result.
|
||
|
||
---
|
||
|
||
## 4. The duality — PD wins TPOT, loses TTFT
|
||
|
||
PD-disagg delivers exactly the phase-isolation benefit MB1 predicted: with no
|
||
prefill stealing decode steps, **inter-token latency is dramatically cleaner.**
|
||
|
||
| TPOT | **8C** | 6P+2D | 4P+4D | 2P+6D |
|
||
|---|---|---|---|---|
|
||
| mean | 87 ms | 11 ms | 9 ms | 6 ms |
|
||
| p90 | 230 ms | 18 ms | 14 ms | 8 ms |
|
||
| p99 | **1129 ms** | **26 ms** | **20 ms** | **12 ms** |
|
||
|
||
PD's TPOT p99 is **10–35× lower** — once a request reaches a dedicated decode
|
||
instance it streams without interruption. 8C's 1.1 s TPOT p99 *is* the
|
||
chunked-prefill interference tax (decode steps occasionally stalled behind an
|
||
8k-token prefill chunk), consistent with MB1.
|
||
|
||
**But the win is local.** TTFT inflates 2.5–6× because every request now pays
|
||
P→D handoff + admission into a smaller, saturated decode pool. For this
|
||
workload's modest output lengths, TTFT dominates total time, so the TPOT win
|
||
never pays for itself. This is the cost/benefit imbalance made concrete:
|
||
phase isolation is real, but it is the wrong thing to optimize when the pool
|
||
is the binding constraint.
|
||
|
||
---
|
||
|
||
## 5. Root cause — per-role KV pool occupancy (the kill shot)
|
||
|
||
The cluster-average KV utilization is *misleading* and nearly hid the result:
|
||
|
||
|
||
|
||
6P+2D and 4P+4D look only ~42–46% utilized on cluster average — yet they have
|
||
128–152 requests queued. The average hides that **one pool is pegged while
|
||
the other idles.** Splitting the KV pool by role exposes it:
|
||
|
||
|
||
|
||
| Config | P-pool steady | D-pool steady | D-pool **peak** | binding side |
|
||
|---|---|---|---|---|
|
||
| 8C | — single shared pool — | 34% | 72% | none (elastic) |
|
||
| 6P+2D | 31% | **74%** | **99.6%** | **decode** |
|
||
| 4P+4D | 29% | **60%** | **97.5%** | **decode** |
|
||
| 2P+6D | **92%** | 95% | 96% | **prefill** (P jams first) |
|
||
|
||
|
||
|
||
**The mechanism, unified:**
|
||
|
||
- A static P:D split fixes the KV capacity on each side at deploy time.
|
||
- The agentic workload's instantaneous P:D demand *drifts* (bursts of new
|
||
sessions = prefill-heavy; long tool-call-driven turns = decode-heavy).
|
||
- Whichever side is undersized *for the current phase* saturates and
|
||
back-pressures the whole pipeline, while the other side's KV sits stranded.
|
||
- 6P+2D / 4P+4D → decode side too small → D-pool hits ~100%, prefilled
|
||
requests queue for a decode slot → TTFT explodes (this is **H1**).
|
||
- 2P+6D → prefill side too small → P-pool hits ~100%, requests can't even
|
||
start → 872 queued, 91% dropped.
|
||
- **8C colocation has no partition**: prefill and decode share one pool, so
|
||
the pool elastically reallocates to whichever phase is hot. Steady
|
||
utilization stays at 34% with 100% completion.
|
||
|
||
This is **H1 (D-pool capacity ceiling)** and **H2 (static-partition
|
||
mismatch)** turning out to be the *same* phenomenon seen from two ratios.
|
||
|
||
### 5.1 The same pressure crashes consumers (a vLLM 0.18.1 fragility)
|
||
|
||
D-pool saturation doesn't just slow things down — under this workload it
|
||
**crashes the decode instances**. The exact chain, from the 6P+2D consumer
|
||
logs:
|
||
|
||
1. D-pool fills to **97.2%** (the capacity ceiling above).
|
||
2. A large request needs its KV pulled to the consumer, but the transfer
|
||
fails: `Mooncake transfer engine returned -1` (observed on a **112,793-token**
|
||
request — agentic sessions have very long multi-turn contexts, and the
|
||
pool had no room).
|
||
3. `kv_load_failure_policy=fail` fails that request — by itself recoverable.
|
||
4. **But** the failure path computes `PromptTokenStats.local_cache_hit =
|
||
num_cached + recomputed − num_external_computed`, which goes **negative**
|
||
when the external transfer exceeded the scheduler's cached count.
|
||
5. `loggers.record()` calls `Counter.inc(negative)` → prometheus_client raises
|
||
*"Counters can only be incremented by non-negative amounts"* → the
|
||
**EngineCore dies**.
|
||
6. Once the consumer's engine is dead, **every** subsequent request fails.
|
||
|
||
The signature is a cliff, not a slope: in the session-routing 6P+2D run, all
|
||
80 successes landed in the first ~110 s, then **zero** of the next ~2,800 s.
|
||
This same intermittent consumer death is almost certainly why the
|
||
round-robin 6P+2D reps varied so wildly (100% / 56% / 80%) — the consumer
|
||
crashed at different points in each rep.
|
||
|
||
**Two takeaways:** (a) PD-disagg under agentic context lengths hits KV-transfer
|
||
failures that colocation never does (8C never transfers — it prefills and
|
||
decodes in the same pool); (b) vLLM 0.18.1's failure handling amplifies one
|
||
failed request into a total collapse. We patched the counter underflow
|
||
(`instrument_kv_snapshot.py`, clamp to ≥ 0) so a transfer failure stays a
|
||
single failed request, which is required to compare routing arms fairly in §6.
|
||
|
||
---
|
||
|
||
## 6. The routing handicap — and whether smarter routing rescues PD
|
||
|
||
Every PD config above shows **prefix-cache hit = 0%**, versus 8C's 19%. That
|
||
is not fundamental to disaggregation — it is the stock proxy round-robining
|
||
the **prefill** side: consecutive turns of one agentic session land on
|
||
*different* producers, so each turn re-prefills the whole conversation from
|
||
scratch. That both inflates TTFT and piles extra load on the prefill pool
|
||
(directly worsening the 2P+6D collapse).
|
||
|
||
The correct PD scheduling policy (as the design argues): **P should be chosen
|
||
by session affinity** (reuse the producer's prefix cache) while **D is chosen
|
||
by load balance** (decode KV is freshly transferred per turn, so D gains
|
||
nothing from affinity). We added this as an env-gated mode in the proxy
|
||
(`MB5_P_ROUTING=session`, consistent hash on `X-Session-Id`; D stays
|
||
round-robin) and re-ran the best-performing disaggregated config, **6P+2D**.
|
||
|
||
> **Status: session-affinity 6P+2D run in progress.** Results below will be
|
||
> filled in when it completes; the question it answers is *how much of the
|
||
> gap to 8C does restoring prefix-cache reuse close.*
|
||
|
||
<!-- SESSION_AFFINITY_RESULTS -->
|
||
*(pending)*
|
||
|
||
---
|
||
|
||
## 7. Caveats / honesty
|
||
|
||
- **Single rep** for this analysis. The earlier 3-rep sweep showed 8C and
|
||
4P+4D are tight run-to-run, but 6P+2D completion varied (rep1 100% vs rep2
|
||
56% vs rep3 80%) — i.e. the D-pool sits right at the cliff edge, so 6P+2D's
|
||
"100% rep1" is optimistic. The qualitative ranking is robust; exact numbers
|
||
on the marginal configs are not.
|
||
- **Latency percentiles count successes only** (see §3 warning). For failing
|
||
configs the latency bars *understate* the damage.
|
||
- **Round-robin baseline.** §6 addresses the routing fairness concern head-on
|
||
with a session-affinity re-run.
|
||
- Trace is a single agentic workload; conclusions are about *this* class of
|
||
workload (sub-second tool-call cadence, multi-turn sessions), not all LLM
|
||
serving.
|
||
|
||
---
|
||
|
||
## 8. Reproduce
|
||
|
||
```bash
|
||
# from repo root, after microbench/fresh_setup/deploy.sh dash1
|
||
# 1. round-robin baseline sweep (1 rep)
|
||
ssh dash1 'CONFIGS="8C 6P+2D 4P+4D 2P+6D" REPS=1 RUN_TAG=<tag> \
|
||
bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb5_run.sh'
|
||
|
||
# 2. reduce on dash1 (numpy-only; handles the multi-GB snapshot dirs)
|
||
ssh dash1 '.venv/bin/python scripts/aggregate_mb5.py --sweep-root mb5_runs \
|
||
--tag <tag> --configs "8C 6P+2D 4P+4D 2P+6D" --reps 1 \
|
||
--reduce-to mb5_runs/reduced_<tag>.json'
|
||
|
||
# 3. pull the compact JSON, render figures locally
|
||
scp dash1:.../mb5_runs/reduced_<tag>.json analysis/mb5/
|
||
.venv/bin/python microbench/fresh_setup/aggregate_mb5.py \
|
||
--from-reduced analysis/mb5/reduced_<tag>.json --out-dir figs/mb5
|
||
|
||
# session-affinity arm: prefix the run with MB5_P_ROUTING=session
|
||
```
|