gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Correct PD-disagg cost/benefit framing across repo

Browse Source 
The §3.2 cost-vs-benefit math in commits 029821c (MB1 plot +
pd_cost_vs_benefit.png) and abde010 (RESULTS_SUMMARY.md) was wrong.

What was wrong:
  I framed PD-disagg's max phase-isolation benefit as "≤ decode duration
  of the new request (~50–200 ms)" — implicitly treating the benefit as
  per-request and bounded by that request's own decode. The correct
  accounting is per-prefill-event across all stalled streams:

      benefit_per_prefill = D × T_prefill × (1 − TPOT_baseline/TPOT_during)
                          ≈ D × T_prefill

  which follows from the chunked-prefill math (each of L/N chunks slows
  D ongoing decode steps from ~10 ms to t ms, summing to D × T_prefill).

Plug MB1 + MB2 numbers in:

  prefill size | T_prefill | T_transfer | D=8 benefit | cost/benefit
   2k tok      | 0.14 s    |     8 ms   |   1.1 s     |    0.7 %
  33k tok      | 4.5  s    |  320 ms    |  36   s     |    0.9 %
 125k tok      | 57   s    |  1.9 s     | 456   s     |    0.4 %

On the phase-isolation axis alone, PD-disagg WINS by 100×–250× — the
opposite of what the deleted figure showed.

The actual dominant reason static PD-disagg fails in agentic is the
D-side KV pool capacity wall (figs/f4b_pdsep_kv_wall.png) — p99
single-request KV is 11.5 GiB, per-D-instance pool is 38 GiB, so 4P+4D
halves system decode capacity. Colleague's 4P+4D experiment showed
TTFT p50 62× worse and success rate 99.5% → 52%, driven by pool
overflow + queueing, not by transfer latency.

Changes (all touched files explicitly listed; no `git add -u`):
- figs/pd_cost_vs_benefit.png : DELETED (figure built on wrong math)
- microbench/fresh_setup/plot_mb1.py : drop the pd_cost_vs_benefit
  function; keep mb1_interference.png and update its title to note
  per-prefill aggregate stall = D × T_prefill (not capped by decode)
- figs/mb1_interference.png : regenerated, no misleading band annotation
- analysis/mb1/README.md : Summary block rewritten ("what MB1 measures";
  no more "max benefit = decode duration" claim); §3.2 implications
  section replaced with the corrected per-prefill-event table; explicit
  ⚠ Correction note documents what was wrong
- analysis/mb2/README.md : Summary block + §3.2 implications section
  rewritten the same way; ⚠ Correction note links to RESULTS_SUMMARY §4
- RESULTS_SUMMARY.md §4 + §6 : §4 reordered to lead with the D-side
  capacity argument (the real failure mode), MB1/MB2 demoted from
  "kill-shot for PD-disagg" to "supporting context inputs to a
  cost-benefit table that actually favors PD-disagg on this axis";
  §6 paper-claims list reordered to remove the wrong "PD-disagg loses
  on cost-vs-benefit" claim and replace with the corrected ones

PAPER_OUTLINE.md and MEETING.md were checked and never picked up this
specific wrong claim — they already (correctly) frame §3.2 around the
D-side KV memory wall.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-27 22:04:49 +08:00
parent abde010b64
commit da39ab6804
6 changed files with 155 additions and 178 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
analysis/mb1/README.md
View File

@@ -15,20 +15,28 @@ bottom; the **Summary** block is what gets cited.
| Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** |
| Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** |
| Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** |
| Maximum PD-disagg benefit per agentic decode | **≤ 50–200 ms** (= decode duration) |
**§3.2 headline (cost vs benefit, this run + MB2)**:
**What MB1 actually measures**:
> Under chunked-prefill, every ongoing decode stream is essentially
> **halted while a prefill chunk is in flight** — per-stream effective
> TPOT during the burst is 15× to 2000× baseline, scaling with prefill
> size. PD-disagg can recover this stall, but the recovery is bounded by
> the **decode duration** of the request being protected. For agentic,
> decode is 50–200 ms (tool-call output). MB2 shows PD-disagg pays
> 300 ms – 10 s of KV-transfer cost per request to do that recovery. The
> cost exceeds the benefit ceiling for any per-request KV ≥ ~80 MiB
> (~830 tokens) — well below all agentic operating points. The benefit
> never beats the cost in this workload.
> During a prefill burst, every ongoing decode stream is essentially
> halted (per-stream effective TPOT is 15×–2000× baseline, scaling with
> prefill size). The **total decode time lost per prefill event is
> `D × T_prefill`** (D concurrent decodes each lose ~T_prefill of useful
> work). For the trace mean (P ≈ 33k tokens, T_prefill ≈ 4.5 s) at D=8
> that's **~36 seconds of decode-equivalent work lost per request**.
> This is the **upper bound on what PD-disaggregation's phase isolation
> could recover** on the decode side.
**⚠ Correction (2026-05-27)**: an earlier version of this README framed
the §3.2 PD-disagg argument as "phase-isolation benefit is capped at
the decode duration of the new request (~50–200 ms), so MB2 transfer
cost dominates". That framing was wrong. The correct accounting is
benefit-per-prefill-event = D × T_prefill (aggregate decode time saved
across all stalled streams), which is **much larger than per-request
transfer cost**. The actual reason static PD-disagg fails in agentic
is **D-side KV pool capacity** (`figs/f4b_pdsep_kv_wall.png`), not a
cost-vs-benefit imbalance on phase isolation. See `RESULTS_SUMMARY.md`
section 4 for the corrected framing.
## Setup
@@ -107,25 +115,30 @@ the cleanest "average over the whole burst window" number).
halts decode). This is the entire prefill duration's worth of decode
time that could in principle be recovered.
Two big caveats for **agentic** application:
**Connecting to the §3.2 PD-disagg argument** (corrected):
1. **Decode is short** (~50–200 ms for tool-call output). The actual
recoverable benefit per request is bounded by the decode duration,
not by `prefill_ttft`. If a decode lasts 100 ms and a 5-second prefill
collides with it, PD-disagg can save at most 100 ms — not 5 s.
2. **PD-disagg pays KV-transfer cost** (MB2: 300 ms – 10 s per request
for agentic sizes). For any KV ≥ ~80 MiB the cost already exceeds the
~100 ms benefit ceiling. Cost > benefit across the whole agentic
distribution.
PD-disagg's promised phase-isolation benefit is **per prefill event**,
not per request. When a new prefill arrives, it stalls every concurrent
decode stream on the same GPU. The aggregate decode time lost across
those D streams is `D × T_prefill`. PD-disagg moving prefill off-decode-GPU
recovers all of it.
## §3.2 cost-vs-benefit figure
Plugging numbers per prefill event:
`figs/pd_cost_vs_benefit.png` overlays MB1 benefit ceiling (50–200 ms
band, capped by decode duration) on top of MB2 transfer cost curve. The
cost curve crosses the benefit ceiling somewhere around **80 MiB / 830
tokens** of KV — well below the trace mean (192 MiB / 2k tok ≈ trace
mean per request KV, and we know agentic averages 33k tokens, p99
125k). For anything bigger PD-disagg pays more than it can recover.
| Prefill size | T_prefill | PD-disagg cost (MB2 T_transfer) | PD-disagg benefit (D=8 × T_prefill) | Ratio |
|---:|---:|---:|---:|---:|
| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s | 0.7 % |
| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s | 0.9 % |
| 125k tok (~p99) | 57 s | 1.9 s | 456 s | 0.4 % |
On the **phase-isolation axis alone**, PD-disagg wins by 100×–250×.
The reason static PD-disagg nonetheless **fails in agentic** is a
*different* failure mode: the D-side KV pool cannot fit p90+ requests
(p99 = 11.5 GiB; D-instance pool ≈ 38 GiB; 4P+4D halves system-wide
decode capacity → TTFT p50 62×, success rate 99.5% → 52% in colleague's
4P+4D experiment). The structural problem is **capacity** (see
`figs/f4b_pdsep_kv_wall.png`), not transfer-cost vs phase-isolation
trade-off.
## Reproduction
@@ -174,4 +187,7 @@ ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop
3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on
dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`.
Figures: `figs/mb1_interference.png`, `figs/pd_cost_vs_benefit.png`.
Figure: `figs/mb1_interference.png`. The figure
`figs/pd_cost_vs_benefit.png` from the original commit `029821c` was
based on the wrong "benefit ≤ decode duration" accounting; **deleted in
the correction commit**.

78
analysis/mb2/README.md
View File

@@ -24,12 +24,25 @@ get cheaper by co-locating P and D on the same node — the ~9.7 GB/s
ceiling applies regardless. Halving the transfer cost cannot be bought
back by topology.
**Headline for the paper §3.2**: at the agentic tail, **pure KV transfer
takes 1.5 – 10 s**. A median agentic decode is **50 – 200 ms** of tool-call
output. So **PD-disaggregation adds 8 – 100 × decode-time of transfer on
top of every routed request**. Phase isolation (the thing PD-disagg
trades transfer cost for) can only win back at most one decode duration
— for agentic that's negligible. The arithmetic is one-sided.
**What MB2 actually measures**: the **per-request charge** that
PD-disagg pays for every routed request — `T_transfer ≈ KV_size / 9.7
GB/s`. For agentic this is **8 ms (192 MiB / trace lower) – 1.9 s
(11.5 GiB / p99)**.
**⚠ Correction (2026-05-27)**: an earlier version of this README
framed §3.2 as "transfer cost (1.5–10 s) >> decode duration (50–200 ms),
so PD-disagg loses on cost-vs-benefit." That accounting was wrong:
PD-disagg's phase-isolation benefit is **per-prefill-event** and equals
`D × T_prefill` (aggregate across stalled decode streams), not the
single-request decode duration. With trace-mean `T_prefill = 4.5 s` and
D = 8, the benefit is ~36 s — far larger than the ~0.32 s transfer
cost. PD-disagg's phase-isolation axis is a *win*, not a loss.
The actual reason static PD-disagg fails in agentic is **D-side KV
capacity** (`figs/f4b_pdsep_kv_wall.png`), not a cost-vs-benefit
imbalance. See `RESULTS_SUMMARY.md` section 4 for the corrected
framing. MB2 still serves as the source of the per-request transfer
cost number used in that analysis.
---
@@ -137,43 +150,44 @@ analysis; not done yet.
treats them as additional samples (same sizes); the per-size
aggregates use all of them.
## Implications for §3.2 PD-disagg cost argument
## Implications for §3.2 PD-disagg argument
For each PD-disagg-routed request, transfer wall-time is:
```
T_transfer(KV_size) = max( pure_transfer(KV_size), rx_overhead )
≈ KV_size / 9.7 GB/s for KV_size <= 3 GiB
T_transfer(KV_size) ≈ KV_size / 9.7 GB/s for KV_size ≤ 3 GiB
≈ 0.3 – 10 s for KV_size in [3, 12] GiB
```
Agentic decode wall-time is typically 50 – 200 ms (tool-call output of
a few tens of tokens at ~50 tok/s). So the **transfer/decode ratio**
under intra-node best-case Mooncake is:
This is the **per-request transfer charge** of PD-disagg. It's a
real cost, but in the context of phase-isolation accounting it is
*small* compared to the benefit:
| KV size | T_transfer @9.7 GB/s | typical decode | T_transfer / T_decode |
|---|---:|---:|---:|
| 192 MiB (2k tok) | 20 ms | 100 ms | 0.2× |
| 768 MiB (8k tok) | 84 ms | 100 ms | 0.8× |
| 3 GiB (33k tok ≈ trace mean) | 321 ms | 100 ms | **3.2×** |
| 6 GiB (~p90) | 1900 ms | 100 ms | **19×** |
| 12 GiB (~p99) | 2800 ms | 100 ms | **28×** (median) – **100×** (p99 variance) |
| Prefill | T_prefill (MB1) | T_transfer (MB2) | Phase-isolation benefit at D=8 = D × T_prefill |
|---:|---:|---:|---:|
| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s |
| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s |
| 125k tok (~p99) | 57 s | 1.9 s | 456 s |
PD-disagg's promised payoff is *eliminating prefill–decode interference
on the decode instance*. The maximum benefit it can buy is bounded
above by the **decode duration itself** (you cannot recover more time
than the decode existed). For agentic that's 50 – 200 ms. The cost is
the table column above — 0.3 – 10 s of transfer per routed request.
On the phase-isolation axis alone, PD-disagg recovers two orders of
magnitude more decode time than it pays in transfer. **It is NOT this
axis that defeats static PD-disagg in agentic** — see colleague's
4P+4D experiment (TTFT p50 62×, success rate 99.5% → 52%) which is
driven by **D-side KV-pool overflow** on long-context requests
(`figs/f4b_pdsep_kv_wall.png`), not by transfer latency.
**Cost > Benefit by 5× to 100× across the agentic distribution.** Below
~3 GiB the ratio is small (≤1×); above 3 GiB the ratio explodes; above
6 GiB even individual draws can take 10 s for a single transfer.
What MB2 contributes to the paper is therefore:
- The **per-request transfer cost number** (used as the cost input
to the cost-benefit accounting above).
- The empirical observation that **Mooncake's transfer cost is
topology-independent** — intra-node and inter-node both go through
the RDMA NIC and hit the same 9.7 GB/s ceiling. PD-disagg's
transfer cost does not get cheaper by co-locating P and D.
This data alone is not the whole §3.2 argument — we still need to
account for D-side KV capacity (`f4b`, separate axis), cache reuse loss,
and static-partition mismatch (MB3 / MB4 / MB5). But it nails one
of the two key cost axes with measured numbers from vanilla mooncake,
not the dash0 patched build.
The dominant §3.2 failure mode of static PD-disagg in agentic is
**capacity**, not transfer cost. MB3 / MB4 / MB5 will quantify the
remaining axes (D-pool occupancy, cache reuse degradation under PD
routing, static-partition mismatch).
## Open questions / next runs