Correct PD-disagg cost/benefit framing across repo
The §3.2 cost-vs-benefit math in commits029821c(MB1 plot + pd_cost_vs_benefit.png) andabde010(RESULTS_SUMMARY.md) was wrong. What was wrong: I framed PD-disagg's max phase-isolation benefit as "≤ decode duration of the new request (~50–200 ms)" — implicitly treating the benefit as per-request and bounded by that request's own decode. The correct accounting is per-prefill-event across all stalled streams: benefit_per_prefill = D × T_prefill × (1 − TPOT_baseline/TPOT_during) ≈ D × T_prefill which the user pointed out follows from the chunked-prefill math (each of L/N chunks slows D ongoing decode steps from ~10 ms to t ms, summing to D × T_prefill). Plug MB1 + MB2 numbers in: prefill size | T_prefill | T_transfer | D=8 benefit | cost/benefit 2k tok | 0.14 s | 8 ms | 1.1 s | 0.7 % 33k tok | 4.5 s | 320 ms | 36 s | 0.9 % 125k tok | 57 s | 1.9 s | 456 s | 0.4 % On the phase-isolation axis alone, PD-disagg WINS by 100×–250× — the opposite of what the deleted figure showed. The actual dominant reason static PD-disagg fails in agentic is the D-side KV pool capacity wall (figs/f4b_pdsep_kv_wall.png) — p99 single-request KV is 11.5 GiB, per-D-instance pool is 38 GiB, so 4P+4D halves system decode capacity. Colleague's 4P+4D experiment showed TTFT p50 62× worse and success rate 99.5% → 52%, driven by pool overflow + queueing, not by transfer latency. Changes: - figs/pd_cost_vs_benefit.png : DELETED (figure built on wrong math) - microbench/fresh_setup/plot_mb1.py : drop the pd_cost_vs_benefit function; keep mb1_interference.png and update its title to note per-prefill aggregate stall = D × T_prefill (not capped by decode) - figs/mb1_interference.png : regenerated, no misleading band annotation - analysis/mb1/README.md : Summary block rewritten ("what MB1 measures"; no more "max benefit = decode duration" claim); §3.2 implications section replaced with the corrected per-prefill-event table; explicit ⚠ Correction note documents what was wrong - analysis/mb2/README.md : Summary block + §3.2 implications section rewritten the same way; ⚠ Correction note links to RESULTS_SUMMARY §4 - RESULTS_SUMMARY.md §4 + §6 : §4 reordered to lead with the D-side capacity argument (the real failure mode), MB1/MB2 demoted from "kill-shot for PD-disagg" to "supporting context inputs to a cost-benefit table that actually favors PD-disagg on this axis"; §6 paper-claims list reordered to remove the wrong "PD-disagg loses on cost-vs-benefit" claim and replace with the corrected ones PAPER_OUTLINE.md and MEETING.md were checked and never picked up this specific wrong claim — they already (correctly) frame §3.2 around the D-side KV memory wall. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -39,52 +39,63 @@ Production trace = Qwen3-Coder agentic，1.3 M sessions / 2.1 M reqs / 7200 s。
|
||||
|
||||
参考图：`figs/f4a_apc_loss.png`、`figs/f4b_pdsep_kv_wall.png`、`figs/f4c_per_worker_ttft.png`、`figs/f6_e2e_latency_bars.png`、`figs/f6_e2e_latency_full_grid.png`。
|
||||
|
||||
## 4. PD-disagg 在 agentic 下输——cost vs benefit（§3.2）
|
||||
## 4. Static PD-disagg 为什么失败（§3.2）—— 容量问题，不是 cost-benefit 问题
|
||||
|
||||
由两个独立 microbench 钉死（**全用 vanilla vLLM 0.18.1 + Mooncake 0.3.11，fresh venv，无 patch**）。
|
||||
⚠ **2026-05-27 纠正**：本节前一版本论证"PD-disagg 因为 transfer cost > phase isolation benefit 而失败"。**这个论证算错了**。正确的 phase-isolation benefit 是**每个 prefill 事件 × D 个 concurrent stream** 的总和（≈ `D × T_prefill`），不是单个 request 的 decode 时长。用正确公式，PD-disagg 在 phase-isolation 这一维上**赢 colo 一两个数量级**。Static PD-disagg 在 agentic 上失败的**真正根因是 D 侧 KV pool 容量**，不是这一维。
|
||||
|
||||
### 4.1 MB2 — KV transfer cost
|
||||
### 4.1 真正的失败模式：D 侧 KV 容量天花板
|
||||
|
||||
dash1 GPU 0+1（intra-node）和 dash1 ↔ dash2（inter-node, 200 Gbps RoCE）扫 9 个 size × 5 reps。
|
||||
| | 8C colo | 4P+4D PD-disagg |
|
||||
|---|---:|---:|
|
||||
| Per-D-instance KV pool（0.4 × 96 GiB） | 38 GiB | 38 GiB |
|
||||
| 系统总 decode 容量（D 实例数 × 单池） | 8 × 38 = **304 GiB** | 4 × 38 = **152 GiB** |
|
||||
| p99 单请求 KV = 11.5 GiB → 最多并发 decode | 24 | **12（减半）** |
|
||||
|
||||
| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求（11.5 GiB）transfer 时间 |
|
||||
Colleague 4P+4D 实测：TTFT p50 0.91 s → **62.8 s（62×）**、success rate **99.5% → 52%**。失败模式：**D 池溢出 + 排队**，不是 transfer 延迟。
|
||||
|
||||
参考图：`figs/f4b_pdsep_kv_wall.png`（pdf 版本是高质量 paper figure）。
|
||||
|
||||
### 4.2 MB2 — KV transfer cost（per-request 一次性成本，**不**是 dominant cost）
|
||||
|
||||
dash1 GPU 0+1（intra）和 dash1 ↔ dash2（inter, 200 Gbps RoCE）扫 9 个 size × 5 reps。
|
||||
|
||||
| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求 11.5 GiB transfer |
|
||||
|---|---|---|
|
||||
| Intra-node | **9.7 GB/s** | p50 **1.9 s** · min 1.5 s · max 10 s |
|
||||
| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · min 1.3 s · max 9.2 s |
|
||||
| Intra-node | **9.7 GB/s** | p50 **1.9 s** · max 10 s |
|
||||
| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · max 9.2 s |
|
||||
|
||||
**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC，包括 intra-node loopback**，不走 NVLink。200 Gbps NIC 是天花板，**PD-disagg 的 transfer cost 与拓扑无关**。
|
||||
**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC**，不走 NVLink。200 Gbps NIC 是天花板。**PD-disagg transfer cost 与拓扑无关**。
|
||||
|
||||
参考图：`figs/mb2_transfer_time_compare.png`、`figs/mb2_transfer_bw_compare.png`、doc `analysis/mb2/README.md`。
|
||||
参考图：`figs/mb2_transfer_time_compare.png`、doc `analysis/mb2/README.md`。
|
||||
|
||||
### 4.2 MB1 — Phase interference（chunked-prefill on, 默认 baseline）
|
||||
### 4.3 MB1 — Phase interference（PD-disagg 的潜在 benefit 上界）
|
||||
|
||||
dash1 GPU 0 单 instance，D（concurrent decodes）× P（prefill size）扫描。
|
||||
dash1 GPU 0 单 instance（无 kv_connector），chunked-prefill 默认开启，D × P 扫描。D=8 结果：
|
||||
|
||||
D=8（最 agentic-realistic）的结果：
|
||||
|
||||
| Prefill | prefill_ttft | per-stream TPOT during | penalty |
|
||||
| Prefill | T_prefill | per-stream TPOT during | penalty |
|
||||
|---|---:|---:|---:|
|
||||
| 2k tok | 143 ms | 32 ms | 4× |
|
||||
| 8k | 583 ms | 114 ms | 15× |
|
||||
| 32k | 4.5 s | 388 ms | **52×** |
|
||||
| 65k | 15.6 s | 757 ms | **99×** |
|
||||
| 131k | 57 s | 1419 ms | **183×** |
|
||||
| 32k tok | 4.5 s | 388 ms | **52×** |
|
||||
| 131k tok | 57 s | 1419 ms | **183×** |
|
||||
|
||||
baseline TPOT 7.7 ms。**Decode 在大 prefill 期间基本被 halted**。chunked-prefill 已经默认开启，PD-disagg 在它之上能额外提供的 phase isolation = **decode 在 prefill 期间被 halted 的那部分时间**。
|
||||
**Decode 在 prefill 期间被几乎完全 halted**，单 stream 损失 ≈ `T_prefill` 的时间。**每个 prefill event 总 decode 损失 ≈ `D × T_prefill`**。
|
||||
|
||||
参考图：`figs/mb1_interference.png`、doc `analysis/mb1/README.md`。
|
||||
|
||||
### 4.3 联合结论
|
||||
### 4.4 联合 cost-benefit（per-prefill event）
|
||||
|
||||
| | Per-request |
|
||||
|---|---|
|
||||
| **Max PD-disagg benefit**（救回来的 decode 时间）| ≤ **decode 时长 = 50–200 ms**（agentic tool-call output）|
|
||||
| **PD-disagg cost**（MB2 transfer p50）| 80 MiB ≈ 8 ms · 3 GiB ≈ 320 ms · 11.5 GiB ≈ **1.9 s**（p99 实测最差 10 s）|
|
||||
| Cost / Benefit | **每个 KV ≥ 80 MiB 的请求都输**；trace 平均 KV 192 MiB → 已经输 |
|
||||
| Prefill (KV size) | T_prefill | Cost = T_transfer | Benefit = D × T_prefill (D=8) | Cost / Benefit |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 2k tok (192 MiB) | 0.14 s | 8 ms | 1.1 s | **0.7%** |
|
||||
| 33k tok (3 GiB, trace mean) | 4.5 s | 0.32 s | 36 s | **0.9%** |
|
||||
| 125k tok (12 GiB, ~p99) | 57 s | 1.9 s | 456 s | **0.4%** |
|
||||
|
||||
**结论**：在 agentic 上 **PD-disaggregation 是结构性失败的**。Chunked-prefill 默认已经在 colocation 内做了 first-order phase isolation；PD-disagg 在此之上能额外补的（decode 短时段没被 prefill 挤）小于它新带来的（每个 routed 请求都付 KV transfer）。这个结论与拓扑无关（intra-node 和 inter-node 一样）。
|
||||
**PD-disagg 在 phase-isolation 这一维赢 100×–250×**。但**这不是 §3.2 该用的论证**，因为 §3.2 真正的 dominant failure 是 §4.1 的 D 池容量天花板（颠覆了上面的全部数学）。
|
||||
|
||||
参考图：`figs/pd_cost_vs_benefit.png`（§3.2 headline）。
|
||||
**总结**：
|
||||
- D 侧 KV 容量天花板（§4.1）→ PD-disagg 在 agentic 上**结构性失败**。
|
||||
- MB1 + MB2 的合计 cost-benefit 在 phase isolation 维度上 PD-disagg 是赢的，**但这件事被容量天花板压倒**。
|
||||
- Paper §3.2 论证应该聚焦"D 池装不下"，MB1/MB2 数据用作 supporting context（per-request transfer charge 量化、phase isolation benefit 量化）而不是 main argument。
|
||||
|
||||
## 5. EAR 设计的实证状态（§4）
|
||||
|
||||
@@ -96,10 +107,12 @@ baseline TPOT 7.7 ms。**Decode 在大 prefill 期间基本被 halted**。chunke
|
||||
## 6. 已经能写的 paper 主张（按 confidence 排序）
|
||||
|
||||
1. **Agentic vs chatbot 在调度上是不同 regime**（dispatch coupling + sub-second tool-call mass）—— 实证完整
|
||||
2. **PD-disaggregation 在 agentic 下输**（cost > benefit，跨拓扑）—— **MB1 + MB2 实证完整**
|
||||
3. **三类现有调度 baseline 各自的失败模式** —— 实证完整
|
||||
4. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整
|
||||
5. **Hot-triggered migration 修复 sticky 的 hot pin** —— **design 完整、e2e 待验证**
|
||||
2. **三类现有调度 baseline 各自的失败模式**（load-balance / static PD-disagg / pure sticky）—— 实证完整
|
||||
3. **Static PD-disagg 在 agentic 下失败的 dominant 根因是 D 侧 KV 容量**（不是 phase-isolation cost-benefit）—— 实证完整（`f4b` + colleague 4P+4D 数据）
|
||||
4. **Mooncake transfer cost 拓扑无关**（intra ≈ inter，~9.7 GB/s NIC 上限）—— 实证完整（MB2）
|
||||
5. **Phase isolation interference 在 chunked-prefill on 下仍然显著**（per-stream TPOT during prefill 15×–2000× baseline）—— 实证完整（MB1）。**注意**：这条数据本身不直接论证 "PD-disagg 失败"，因为算正确账后 PD-disagg 反而在这一维上赢；它的用途是给 §3.2 提供 phase-isolation benefit 上界的量化。
|
||||
6. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整
|
||||
7. **Hot-triggered migration 修复 sticky 的 hot pin** —— design 完整、e2e 待验证
|
||||
|
||||
## 7. 待做
|
||||
|
||||
|
||||
Reference in New Issue
Block a user