Correct PD-disagg cost/benefit framing across repo
The §3.2 cost-vs-benefit math in commits029821c(MB1 plot + pd_cost_vs_benefit.png) andabde010(RESULTS_SUMMARY.md) was wrong. What was wrong: I framed PD-disagg's max phase-isolation benefit as "≤ decode duration of the new request (~50–200 ms)" — implicitly treating the benefit as per-request and bounded by that request's own decode. The correct accounting is per-prefill-event across all stalled streams: benefit_per_prefill = D × T_prefill × (1 − TPOT_baseline/TPOT_during) ≈ D × T_prefill which follows from the chunked-prefill math (each of L/N chunks slows D ongoing decode steps from ~10 ms to t ms, summing to D × T_prefill). Plug MB1 + MB2 numbers in: prefill size | T_prefill | T_transfer | D=8 benefit | cost/benefit 2k tok | 0.14 s | 8 ms | 1.1 s | 0.7 % 33k tok | 4.5 s | 320 ms | 36 s | 0.9 % 125k tok | 57 s | 1.9 s | 456 s | 0.4 % On the phase-isolation axis alone, PD-disagg WINS by 100×–250× — the opposite of what the deleted figure showed. The actual dominant reason static PD-disagg fails in agentic is the D-side KV pool capacity wall (figs/f4b_pdsep_kv_wall.png) — p99 single-request KV is 11.5 GiB, per-D-instance pool is 38 GiB, so 4P+4D halves system decode capacity. Colleague's 4P+4D experiment showed TTFT p50 62× worse and success rate 99.5% → 52%, driven by pool overflow + queueing, not by transfer latency. Changes (all touched files explicitly listed; no `git add -u`): - figs/pd_cost_vs_benefit.png : DELETED (figure built on wrong math) - microbench/fresh_setup/plot_mb1.py : drop the pd_cost_vs_benefit function; keep mb1_interference.png and update its title to note per-prefill aggregate stall = D × T_prefill (not capped by decode) - figs/mb1_interference.png : regenerated, no misleading band annotation - analysis/mb1/README.md : Summary block rewritten ("what MB1 measures"; no more "max benefit = decode duration" claim); §3.2 implications section replaced with the corrected per-prefill-event table; explicit ⚠ Correction note documents what was wrong - analysis/mb2/README.md : Summary block + §3.2 implications section rewritten the same way; ⚠ Correction note links to RESULTS_SUMMARY §4 - RESULTS_SUMMARY.md §4 + §6 : §4 reordered to lead with the D-side capacity argument (the real failure mode), MB1/MB2 demoted from "kill-shot for PD-disagg" to "supporting context inputs to a cost-benefit table that actually favors PD-disagg on this axis"; §6 paper-claims list reordered to remove the wrong "PD-disagg loses on cost-vs-benefit" claim and replace with the corrected ones PAPER_OUTLINE.md and MEETING.md were checked and never picked up this specific wrong claim — they already (correctly) frame §3.2 around the D-side KV memory wall. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -24,12 +24,25 @@ get cheaper by co-locating P and D on the same node — the ~9.7 GB/s
|
||||
ceiling applies regardless. Halving the transfer cost cannot be bought
|
||||
back by topology.
|
||||
|
||||
**Headline for the paper §3.2**: at the agentic tail, **pure KV transfer
|
||||
takes 1.5 – 10 s**. A median agentic decode is **50 – 200 ms** of tool-call
|
||||
output. So **PD-disaggregation adds 8 – 100 × decode-time of transfer on
|
||||
top of every routed request**. Phase isolation (the thing PD-disagg
|
||||
trades transfer cost for) can only win back at most one decode duration
|
||||
— for agentic that's negligible. The arithmetic is one-sided.
|
||||
**What MB2 actually measures**: the **per-request charge** that
|
||||
PD-disagg pays for every routed request — `T_transfer ≈ KV_size / 9.7
|
||||
GB/s`. For agentic this is **8 ms (192 MiB / trace lower) – 1.9 s
|
||||
(11.5 GiB / p99)**.
|
||||
|
||||
**⚠ Correction (2026-05-27)**: an earlier version of this README
|
||||
framed §3.2 as "transfer cost (1.5–10 s) >> decode duration (50–200 ms),
|
||||
so PD-disagg loses on cost-vs-benefit." That accounting was wrong:
|
||||
PD-disagg's phase-isolation benefit is **per-prefill-event** and equals
|
||||
`D × T_prefill` (aggregate across stalled decode streams), not the
|
||||
single-request decode duration. With trace-mean `T_prefill = 4.5 s` and
|
||||
D = 8, the benefit is ~36 s — far larger than the ~0.32 s transfer
|
||||
cost. PD-disagg's phase-isolation axis is a *win*, not a loss.
|
||||
|
||||
The actual reason static PD-disagg fails in agentic is **D-side KV
|
||||
capacity** (`figs/f4b_pdsep_kv_wall.png`), not a cost-vs-benefit
|
||||
imbalance. See `RESULTS_SUMMARY.md` section 4 for the corrected
|
||||
framing. MB2 still serves as the source of the per-request transfer
|
||||
cost number used in that analysis.
|
||||
|
||||
---
|
||||
|
||||
@@ -137,43 +150,44 @@ analysis; not done yet.
|
||||
treats them as additional samples (same sizes); the per-size
|
||||
aggregates use all of them.
|
||||
|
||||
## Implications for §3.2 PD-disagg cost argument
|
||||
## Implications for §3.2 PD-disagg argument
|
||||
|
||||
For each PD-disagg-routed request, transfer wall-time is:
|
||||
|
||||
```
|
||||
T_transfer(KV_size) = max( pure_transfer(KV_size), rx_overhead )
|
||||
≈ KV_size / 9.7 GB/s for KV_size <= 3 GiB
|
||||
T_transfer(KV_size) ≈ KV_size / 9.7 GB/s for KV_size ≤ 3 GiB
|
||||
≈ 0.3 – 10 s for KV_size in [3, 12] GiB
|
||||
```
|
||||
|
||||
Agentic decode wall-time is typically 50 – 200 ms (tool-call output of
|
||||
a few tens of tokens at ~50 tok/s). So the **transfer/decode ratio**
|
||||
under intra-node best-case Mooncake is:
|
||||
This is the **per-request transfer charge** of PD-disagg. It's a
|
||||
real cost, but in the context of phase-isolation accounting it is
|
||||
*small* compared to the benefit:
|
||||
|
||||
| KV size | T_transfer @9.7 GB/s | typical decode | T_transfer / T_decode |
|
||||
|---|---:|---:|---:|
|
||||
| 192 MiB (2k tok) | 20 ms | 100 ms | 0.2× |
|
||||
| 768 MiB (8k tok) | 84 ms | 100 ms | 0.8× |
|
||||
| 3 GiB (33k tok ≈ trace mean) | 321 ms | 100 ms | **3.2×** |
|
||||
| 6 GiB (~p90) | 1900 ms | 100 ms | **19×** |
|
||||
| 12 GiB (~p99) | 2800 ms | 100 ms | **28×** (median) – **100×** (p99 variance) |
|
||||
| Prefill | T_prefill (MB1) | T_transfer (MB2) | Phase-isolation benefit at D=8 = D × T_prefill |
|
||||
|---:|---:|---:|---:|
|
||||
| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s |
|
||||
| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s |
|
||||
| 125k tok (~p99) | 57 s | 1.9 s | 456 s |
|
||||
|
||||
PD-disagg's promised payoff is *eliminating prefill–decode interference
|
||||
on the decode instance*. The maximum benefit it can buy is bounded
|
||||
above by the **decode duration itself** (you cannot recover more time
|
||||
than the decode existed). For agentic that's 50 – 200 ms. The cost is
|
||||
the table column above — 0.3 – 10 s of transfer per routed request.
|
||||
On the phase-isolation axis alone, PD-disagg recovers two orders of
|
||||
magnitude more decode time than it pays in transfer. **It is NOT this
|
||||
axis that defeats static PD-disagg in agentic** — see colleague's
|
||||
4P+4D experiment (TTFT p50 62×, success rate 99.5% → 52%) which is
|
||||
driven by **D-side KV-pool overflow** on long-context requests
|
||||
(`figs/f4b_pdsep_kv_wall.png`), not by transfer latency.
|
||||
|
||||
**Cost > Benefit by 5× to 100× across the agentic distribution.** Below
|
||||
~3 GiB the ratio is small (≤1×); above 3 GiB the ratio explodes; above
|
||||
6 GiB even individual draws can take 10 s for a single transfer.
|
||||
What MB2 contributes to the paper is therefore:
|
||||
- The **per-request transfer cost number** (used as the cost input
|
||||
to the cost-benefit accounting above).
|
||||
- The empirical observation that **Mooncake's transfer cost is
|
||||
topology-independent** — intra-node and inter-node both go through
|
||||
the RDMA NIC and hit the same 9.7 GB/s ceiling. PD-disagg's
|
||||
transfer cost does not get cheaper by co-locating P and D.
|
||||
|
||||
This data alone is not the whole §3.2 argument — we still need to
|
||||
account for D-side KV capacity (`f4b`, separate axis), cache reuse loss,
|
||||
and static-partition mismatch (MB3 / MB4 / MB5). But it nails one
|
||||
of the two key cost axes with measured numbers from vanilla mooncake,
|
||||
not the dash0 patched build.
|
||||
The dominant §3.2 failure mode of static PD-disagg in agentic is
|
||||
**capacity**, not transfer cost. MB3 / MB4 / MB5 will quantify the
|
||||
remaining axes (D-pool occupancy, cache reuse degradation under PD
|
||||
routing, static-partition mismatch).
|
||||
|
||||
## Open questions / next runs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user