Correct PD-disagg cost/benefit framing across repo
The §3.2 cost-vs-benefit math in commits029821c(MB1 plot + pd_cost_vs_benefit.png) andabde010(RESULTS_SUMMARY.md) was wrong. What was wrong: I framed PD-disagg's max phase-isolation benefit as "≤ decode duration of the new request (~50–200 ms)" — implicitly treating the benefit as per-request and bounded by that request's own decode. The correct accounting is per-prefill-event across all stalled streams: benefit_per_prefill = D × T_prefill × (1 − TPOT_baseline/TPOT_during) ≈ D × T_prefill which follows from the chunked-prefill math (each of L/N chunks slows D ongoing decode steps from ~10 ms to t ms, summing to D × T_prefill). Plug MB1 + MB2 numbers in: prefill size | T_prefill | T_transfer | D=8 benefit | cost/benefit 2k tok | 0.14 s | 8 ms | 1.1 s | 0.7 % 33k tok | 4.5 s | 320 ms | 36 s | 0.9 % 125k tok | 57 s | 1.9 s | 456 s | 0.4 % On the phase-isolation axis alone, PD-disagg WINS by 100×–250× — the opposite of what the deleted figure showed. The actual dominant reason static PD-disagg fails in agentic is the D-side KV pool capacity wall (figs/f4b_pdsep_kv_wall.png) — p99 single-request KV is 11.5 GiB, per-D-instance pool is 38 GiB, so 4P+4D halves system decode capacity. Colleague's 4P+4D experiment showed TTFT p50 62× worse and success rate 99.5% → 52%, driven by pool overflow + queueing, not by transfer latency. Changes (all touched files explicitly listed; no `git add -u`): - figs/pd_cost_vs_benefit.png : DELETED (figure built on wrong math) - microbench/fresh_setup/plot_mb1.py : drop the pd_cost_vs_benefit function; keep mb1_interference.png and update its title to note per-prefill aggregate stall = D × T_prefill (not capped by decode) - figs/mb1_interference.png : regenerated, no misleading band annotation - analysis/mb1/README.md : Summary block rewritten ("what MB1 measures"; no more "max benefit = decode duration" claim); §3.2 implications section replaced with the corrected per-prefill-event table; explicit ⚠ Correction note documents what was wrong - analysis/mb2/README.md : Summary block + §3.2 implications section rewritten the same way; ⚠ Correction note links to RESULTS_SUMMARY §4 - RESULTS_SUMMARY.md §4 + §6 : §4 reordered to lead with the D-side capacity argument (the real failure mode), MB1/MB2 demoted from "kill-shot for PD-disagg" to "supporting context inputs to a cost-benefit table that actually favors PD-disagg on this axis"; §6 paper-claims list reordered to remove the wrong "PD-disagg loses on cost-vs-benefit" claim and replace with the corrected ones PAPER_OUTLINE.md and MEETING.md were checked and never picked up this specific wrong claim — they already (correctly) frame §3.2 around the D-side KV memory wall. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -15,20 +15,28 @@ bottom; the **Summary** block is what gets cited.
|
||||
| Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** |
|
||||
| Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** |
|
||||
| Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** |
|
||||
| Maximum PD-disagg benefit per agentic decode | **≤ 50–200 ms** (= decode duration) |
|
||||
|
||||
**§3.2 headline (cost vs benefit, this run + MB2)**:
|
||||
**What MB1 actually measures**:
|
||||
|
||||
> Under chunked-prefill, every ongoing decode stream is essentially
|
||||
> **halted while a prefill chunk is in flight** — per-stream effective
|
||||
> TPOT during the burst is 15× to 2000× baseline, scaling with prefill
|
||||
> size. PD-disagg can recover this stall, but the recovery is bounded by
|
||||
> the **decode duration** of the request being protected. For agentic,
|
||||
> decode is 50–200 ms (tool-call output). MB2 shows PD-disagg pays
|
||||
> 300 ms – 10 s of KV-transfer cost per request to do that recovery. The
|
||||
> cost exceeds the benefit ceiling for any per-request KV ≥ ~80 MiB
|
||||
> (~830 tokens) — well below all agentic operating points. The benefit
|
||||
> never beats the cost in this workload.
|
||||
> During a prefill burst, every ongoing decode stream is essentially
|
||||
> halted (per-stream effective TPOT is 15×–2000× baseline, scaling with
|
||||
> prefill size). The **total decode time lost per prefill event is
|
||||
> `D × T_prefill`** (D concurrent decodes each lose ~T_prefill of useful
|
||||
> work). For the trace mean (P ≈ 33k tokens, T_prefill ≈ 4.5 s) at D=8
|
||||
> that's **~36 seconds of decode-equivalent work lost per request**.
|
||||
> This is the **upper bound on what PD-disaggregation's phase isolation
|
||||
> could recover** on the decode side.
|
||||
|
||||
**⚠ Correction (2026-05-27)**: an earlier version of this README framed
|
||||
the §3.2 PD-disagg argument as "phase-isolation benefit is capped at
|
||||
the decode duration of the new request (~50–200 ms), so MB2 transfer
|
||||
cost dominates". That framing was wrong. The correct accounting is
|
||||
benefit-per-prefill-event = D × T_prefill (aggregate decode time saved
|
||||
across all stalled streams), which is **much larger than per-request
|
||||
transfer cost**. The actual reason static PD-disagg fails in agentic
|
||||
is **D-side KV pool capacity** (`figs/f4b_pdsep_kv_wall.png`), not a
|
||||
cost-vs-benefit imbalance on phase isolation. See `RESULTS_SUMMARY.md`
|
||||
section 4 for the corrected framing.
|
||||
|
||||
## Setup
|
||||
|
||||
@@ -107,25 +115,30 @@ the cleanest "average over the whole burst window" number).
|
||||
halts decode). This is the entire prefill duration's worth of decode
|
||||
time that could in principle be recovered.
|
||||
|
||||
Two big caveats for **agentic** application:
|
||||
**Connecting to the §3.2 PD-disagg argument** (corrected):
|
||||
|
||||
1. **Decode is short** (~50–200 ms for tool-call output). The actual
|
||||
recoverable benefit per request is bounded by the decode duration,
|
||||
not by `prefill_ttft`. If a decode lasts 100 ms and a 5-second prefill
|
||||
collides with it, PD-disagg can save at most 100 ms — not 5 s.
|
||||
2. **PD-disagg pays KV-transfer cost** (MB2: 300 ms – 10 s per request
|
||||
for agentic sizes). For any KV ≥ ~80 MiB the cost already exceeds the
|
||||
~100 ms benefit ceiling. Cost > benefit across the whole agentic
|
||||
distribution.
|
||||
PD-disagg's promised phase-isolation benefit is **per prefill event**,
|
||||
not per request. When a new prefill arrives, it stalls every concurrent
|
||||
decode stream on the same GPU. The aggregate decode time lost across
|
||||
those D streams is `D × T_prefill`. PD-disagg moving prefill off-decode-GPU
|
||||
recovers all of it.
|
||||
|
||||
## §3.2 cost-vs-benefit figure
|
||||
Plugging numbers per prefill event:
|
||||
|
||||
`figs/pd_cost_vs_benefit.png` overlays MB1 benefit ceiling (50–200 ms
|
||||
band, capped by decode duration) on top of MB2 transfer cost curve. The
|
||||
cost curve crosses the benefit ceiling somewhere around **80 MiB / 830
|
||||
tokens** of KV — well below the trace mean (192 MiB / 2k tok ≈ trace
|
||||
mean per request KV, and we know agentic averages 33k tokens, p99
|
||||
125k). For anything bigger PD-disagg pays more than it can recover.
|
||||
| Prefill size | T_prefill | PD-disagg cost (MB2 T_transfer) | PD-disagg benefit (D=8 × T_prefill) | Ratio |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s | 0.7 % |
|
||||
| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s | 0.9 % |
|
||||
| 125k tok (~p99) | 57 s | 1.9 s | 456 s | 0.4 % |
|
||||
|
||||
On the **phase-isolation axis alone**, PD-disagg wins by 100×–250×.
|
||||
The reason static PD-disagg nonetheless **fails in agentic** is a
|
||||
*different* failure mode: the D-side KV pool cannot fit p90+ requests
|
||||
(p99 = 11.5 GiB; D-instance pool ≈ 38 GiB; 4P+4D halves system-wide
|
||||
decode capacity → TTFT p50 62×, success rate 99.5% → 52% in colleague's
|
||||
4P+4D experiment). The structural problem is **capacity** (see
|
||||
`figs/f4b_pdsep_kv_wall.png`), not transfer-cost vs phase-isolation
|
||||
trade-off.
|
||||
|
||||
## Reproduction
|
||||
|
||||
@@ -174,4 +187,7 @@ ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop
|
||||
|
||||
3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on
|
||||
dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`.
|
||||
Figures: `figs/mb1_interference.png`, `figs/pd_cost_vs_benefit.png`.
|
||||
Figure: `figs/mb1_interference.png`. The figure
|
||||
`figs/pd_cost_vs_benefit.png` from the original commit `029821c` was
|
||||
based on the wrong "benefit ≤ decode duration" accounting; **deleted in
|
||||
the correction commit**.
|
||||
|
||||
Reference in New Issue
Block a user