gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Correct PD-disagg cost/benefit framing across repo

Browse Source 
The §3.2 cost-vs-benefit math in commits 029821c (MB1 plot +
pd_cost_vs_benefit.png) and abde010 (RESULTS_SUMMARY.md) was wrong.

What was wrong:
  I framed PD-disagg's max phase-isolation benefit as "≤ decode duration
  of the new request (~50–200 ms)" — implicitly treating the benefit as
  per-request and bounded by that request's own decode. The correct
  accounting is per-prefill-event across all stalled streams:

      benefit_per_prefill = D × T_prefill × (1 − TPOT_baseline/TPOT_during)
                          ≈ D × T_prefill

  which the user pointed out follows from the chunked-prefill math
  (each of L/N chunks slows D ongoing decode steps from ~10 ms to t ms,
  summing to D × T_prefill).

Plug MB1 + MB2 numbers in:

  prefill size | T_prefill | T_transfer | D=8 benefit | cost/benefit
   2k tok      | 0.14 s    |     8 ms   |   1.1 s     |    0.7 %
  33k tok      | 4.5  s    |  320 ms    |  36   s     |    0.9 %
 125k tok      | 57   s    |  1.9 s     | 456   s     |    0.4 %

On the phase-isolation axis alone, PD-disagg WINS by 100×–250× — the
opposite of what the deleted figure showed.

The actual dominant reason static PD-disagg fails in agentic is the
D-side KV pool capacity wall (figs/f4b_pdsep_kv_wall.png) — p99
single-request KV is 11.5 GiB, per-D-instance pool is 38 GiB, so 4P+4D
halves system decode capacity. Colleague's 4P+4D experiment showed
TTFT p50 62× worse and success rate 99.5% → 52%, driven by pool
overflow + queueing, not by transfer latency.

Changes:
- figs/pd_cost_vs_benefit.png : DELETED (figure built on wrong math)
- microbench/fresh_setup/plot_mb1.py : drop the pd_cost_vs_benefit
  function; keep mb1_interference.png and update its title to note
  per-prefill aggregate stall = D × T_prefill (not capped by decode)
- figs/mb1_interference.png : regenerated, no misleading band annotation
- analysis/mb1/README.md : Summary block rewritten ("what MB1 measures";
  no more "max benefit = decode duration" claim); §3.2 implications
  section replaced with the corrected per-prefill-event table; explicit
  ⚠ Correction note documents what was wrong
- analysis/mb2/README.md : Summary block + §3.2 implications section
  rewritten the same way; ⚠ Correction note links to RESULTS_SUMMARY §4
- RESULTS_SUMMARY.md §4 + §6 : §4 reordered to lead with the D-side
  capacity argument (the real failure mode), MB1/MB2 demoted from
  "kill-shot for PD-disagg" to "supporting context inputs to a
  cost-benefit table that actually favors PD-disagg on this axis";
  §6 paper-claims list reordered to remove the wrong "PD-disagg loses
  on cost-vs-benefit" claim and replace with the corrected ones

PAPER_OUTLINE.md and MEETING.md were checked and never picked up this
specific wrong claim — they already (correctly) frame §3.2 around the
D-side KV memory wall.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-27 22:02:52 +08:00
parent abde010b64
commit 4936c6b861
8 changed files with 438 additions and 215 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
RESULTS_SUMMARY.md
View File

@@ -39,52 +39,63 @@ Production trace = Qwen3-Coder agentic，1.3 M sessions / 2.1 M reqs / 7200 s。
参考图：`figs/f4a_apc_loss.png`、`figs/f4b_pdsep_kv_wall.png`、`figs/f4c_per_worker_ttft.png`、`figs/f6_e2e_latency_bars.png`、`figs/f6_e2e_latency_full_grid.png`。
## 4. PD-disagg 在 agentic 下输——cost vs benefit（§3.2）
## 4. Static PD-disagg 为什么失败（§3.2）—— 容量问题，不是 cost-benefit 问题
由两个独立 microbench 钉死（**全用 vanilla vLLM 0.18.1 + Mooncake 0.3.11，fresh venv，无 patch**）。
⚠ **2026-05-27 纠正**：本节前一版本论证"PD-disagg 因为 transfer cost > phase isolation benefit 而失败"。**这个论证算错了**。正确的 phase-isolation benefit 是**每个 prefill 事件 × D 个 concurrent stream** 的总和（≈ `D × T_prefill`），不是单个 request 的 decode 时长。用正确公式，PD-disagg 在 phase-isolation 这一维上**赢 colo 一两个数量级**。Static PD-disagg 在 agentic 上失败的**真正根因是 D 侧 KV pool 容量**，不是这一维。
### 4.1 MB2 — KV transfer cost
### 4.1 真正的失败模式：D 侧 KV 容量天花板
dash1 GPU 0+1（intra-node）和 dash1 ↔ dash2（inter-node, 200 Gbps RoCE）扫 9 个 size × 5 reps。
| | 8C colo | 4P+4D PD-disagg |
|---|---:|---:|
| Per-D-instance KV pool（0.4 × 96 GiB） | 38 GiB | 38 GiB |
| 系统总 decode 容量（D 实例数 × 单池） | 8 × 38 = **304 GiB** | 4 × 38 = **152 GiB** |
| p99 单请求 KV = 11.5 GiB → 最多并发 decode | 24 | **12（减半）** |
| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求（11.5 GiB）transfer 时间 |
Colleague 4P+4D 实测：TTFT p50 0.91 s → **62.8 s（62×）**、success rate **99.5% → 52%**。失败模式：**D 池溢出 + 排队**，不是 transfer 延迟。
参考图：`figs/f4b_pdsep_kv_wall.png`（pdf 版本是高质量 paper figure）。
### 4.2 MB2 — KV transfer cost（per-request 一次性成本，**不**是 dominant cost）
dash1 GPU 0+1（intra）和 dash1 ↔ dash2（inter, 200 Gbps RoCE）扫 9 个 size × 5 reps。
| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求 11.5 GiB transfer |
|---|---|---|
| Intra-node | **9.7 GB/s** | p50 **1.9 s** · min 1.5 s · max 10 s |
| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · min 1.3 s · max 9.2 s |
| Intra-node | **9.7 GB/s** | p50 **1.9 s** · max 10 s |
| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · max 9.2 s |
**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC，包括 intra-node loopback**，不走 NVLink。200 Gbps NIC 是天花板，**PD-disagg 的 transfer cost 与拓扑无关**。
**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC**，不走 NVLink。200 Gbps NIC 是天花板。**PD-disagg transfer cost 与拓扑无关**。
参考图：`figs/mb2_transfer_time_compare.png`、`figs/mb2_transfer_bw_compare.png`、doc `analysis/mb2/README.md`。
参考图：`figs/mb2_transfer_time_compare.png`、doc `analysis/mb2/README.md`。
### 4.2 MB1 — Phase interference（chunked-prefill on, 默认 baseline）
### 4.3 MB1 — Phase interference（PD-disagg 的潜在 benefit 上界）
dash1 GPU 0 单 instance，D（concurrent decodes）× P（prefill size）扫描。
dash1 GPU 0 单 instance（无 kv_connector），chunked-prefill 默认开启，D × P 扫描。D=8 结果：
D=8（最 agentic-realistic）的结果：
| Prefill | prefill_ttft | per-stream TPOT during | penalty |
| Prefill | T_prefill | per-stream TPOT during | penalty |
|---|---:|---:|---:|
| 2k tok | 143 ms | 32 ms | 4× |
| 8k | 583 ms | 114 ms | 15× |
| 32k | 4.5 s | 388 ms | **52×** |
| 65k | 15.6 s | 757 ms | **99×** |
| 131k | 57 s | 1419 ms | **183×** |
| 32k tok | 4.5 s | 388 ms | **52×** |
| 131k tok | 57 s | 1419 ms | **183×** |
baseline TPOT 7.7 ms。**Decode 在大 prefill 期间基本被 halted**。chunked-prefill 已经默认开启，PD-disagg 在它之上能额外提供的 phase isolation = **decode 在 prefill 期间被 halted 的那部分时间**。
**Decode 在 prefill 期间被几乎完全 halted**，单 stream 损失 ≈ `T_prefill` 的时间。**每个 prefill event 总 decode 损失 ≈ `D × T_prefill`**。
参考图：`figs/mb1_interference.png`、doc `analysis/mb1/README.md`。
### 4.3 联合结论
### 4.4 联合 cost-benefit（per-prefill event）
| | Per-request |
|---|---|
| **Max PD-disagg benefit**（救回来的 decode 时间）| ≤ **decode 时长 = 50–200 ms**（agentic tool-call output）|
| **PD-disagg cost**（MB2 transfer p50）| 80 MiB ≈ 8 ms · 3 GiB ≈ 320 ms · 11.5 GiB ≈ **1.9 s**（p99 实测最差 10 s）|
| Cost / Benefit | **每个 KV ≥ 80 MiB 的请求都输**；trace 平均 KV 192 MiB → 已经输 |
| Prefill (KV size) | T_prefill | Cost = T_transfer | Benefit = D × T_prefill (D=8) | Cost / Benefit |
|---:|---:|---:|---:|---:|
| 2k tok (192 MiB) | 0.14 s | 8 ms | 1.1 s | **0.7%** |
| 33k tok (3 GiB, trace mean) | 4.5 s | 0.32 s | 36 s | **0.9%** |
| 125k tok (12 GiB, ~p99) | 57 s | 1.9 s | 456 s | **0.4%** |
**结论**：在 agentic 上 **PD-disaggregation 是结构性失败的**。Chunked-prefill 默认已经在 colocation 内做了 first-order phase isolation；PD-disagg 在此之上能额外补的（decode 短时段没被 prefill 挤）小于它新带来的（每个 routed 请求都付 KV transfer）。这个结论与拓扑无关（intra-node 和 inter-node 一样）。
**PD-disagg 在 phase-isolation 这一维赢 100×–250×**。但**这不是 §3.2 该用的论证**，因为 §3.2 真正的 dominant failure 是 §4.1 的 D 池容量天花板（颠覆了上面的全部数学）。
参考图：`figs/pd_cost_vs_benefit.png`（§3.2 headline）。
**总结**：
- D 侧 KV 容量天花板（§4.1）→ PD-disagg 在 agentic 上**结构性失败**。
- MB1 + MB2 的合计 cost-benefit 在 phase isolation 维度上 PD-disagg 是赢的，**但这件事被容量天花板压倒**。
- Paper §3.2 论证应该聚焦"D 池装不下"，MB1/MB2 数据用作 supporting context（per-request transfer charge 量化、phase isolation benefit 量化）而不是 main argument。
## 5. EAR 设计的实证状态（§4）
@@ -96,10 +107,12 @@ baseline TPOT 7.7 ms。**Decode 在大 prefill 期间基本被 halted**。chunke
## 6. 已经能写的 paper 主张（按 confidence 排序）
1. **Agentic vs chatbot 在调度上是不同 regime**（dispatch coupling + sub-second tool-call mass）—— 实证完整
2. **PD-disaggregation 在 agentic 下输**（cost > benefit，跨拓扑）—— **MB1 + MB2 实证完整**
3. **三类现有调度 baseline 各自的失败模式** —— 实证完整
4. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整
5. **Hot-triggered migration 修复 sticky 的 hot pin** —— **design 完整、e2e 待验证**
2. **三类现有调度 baseline 各自的失败模式**（load-balance / static PD-disagg / pure sticky）—— 实证完整
3. **Static PD-disagg 在 agentic 下失败的 dominant 根因是 D 侧 KV 容量**（不是 phase-isolation cost-benefit）—— 实证完整（`f4b` + colleague 4P+4D 数据）
4. **Mooncake transfer cost 拓扑无关**（intra ≈ inter，~9.7 GB/s NIC 上限）—— 实证完整（MB2）
5. **Phase isolation interference 在 chunked-prefill on 下仍然显著**（per-stream TPOT during prefill 15×–2000× baseline）—— 实证完整（MB1）。**注意**：这条数据本身不直接论证 "PD-disagg 失败"，因为算正确账后 PD-disagg 反而在这一维上赢；它的用途是给 §3.2 提供 phase-isolation benefit 上界的量化。
6. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整
7. **Hot-triggered migration 修复 sticky 的 hot pin** —— design 完整、e2e 待验证
## 7. 待做

74
analysis/mb1/README.md
View File

@@ -15,20 +15,28 @@ bottom; the **Summary** block is what gets cited.
| Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** |
| Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** |
| Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** |
| Maximum PD-disagg benefit per agentic decode | **≤ 50–200 ms** (= decode duration) |
**§3.2 headline (cost vs benefit, this run + MB2)**:
**What MB1 actually measures**:
> Under chunked-prefill, every ongoing decode stream is essentially
> **halted while a prefill chunk is in flight** — per-stream effective
> TPOT during the burst is 15× to 2000× baseline, scaling with prefill
> size. PD-disagg can recover this stall, but the recovery is bounded by
> the **decode duration** of the request being protected. For agentic,
> decode is 50–200 ms (tool-call output). MB2 shows PD-disagg pays
> 300 ms – 10 s of KV-transfer cost per request to do that recovery. The
> cost exceeds the benefit ceiling for any per-request KV ≥ ~80 MiB
> (~830 tokens) — well below all agentic operating points. The benefit
> never beats the cost in this workload.
> During a prefill burst, every ongoing decode stream is essentially
> halted (per-stream effective TPOT is 15×–2000× baseline, scaling with
> prefill size). The **total decode time lost per prefill event is
> `D × T_prefill`** (D concurrent decodes each lose ~T_prefill of useful
> work). For the trace mean (P ≈ 33k tokens, T_prefill ≈ 4.5 s) at D=8
> that's **~36 seconds of decode-equivalent work lost per request**.
> This is the **upper bound on what PD-disaggregation's phase isolation
> could recover** on the decode side.
**⚠ Correction (2026-05-27)**: an earlier version of this README framed
the §3.2 PD-disagg argument as "phase-isolation benefit is capped at
the decode duration of the new request (~50–200 ms), so MB2 transfer
cost dominates". That framing was wrong. The correct accounting is
benefit-per-prefill-event = D × T_prefill (aggregate decode time saved
across all stalled streams), which is **much larger than per-request
transfer cost**. The actual reason static PD-disagg fails in agentic
is **D-side KV pool capacity** (`figs/f4b_pdsep_kv_wall.png`), not a
cost-vs-benefit imbalance on phase isolation. See `RESULTS_SUMMARY.md`
section 4 for the corrected framing.
## Setup
@@ -107,25 +115,30 @@ the cleanest "average over the whole burst window" number).
halts decode). This is the entire prefill duration's worth of decode
time that could in principle be recovered.
Two big caveats for **agentic** application:
**Connecting to the §3.2 PD-disagg argument** (corrected):
1. **Decode is short** (~50–200 ms for tool-call output). The actual
recoverable benefit per request is bounded by the decode duration,
not by `prefill_ttft`. If a decode lasts 100 ms and a 5-second prefill
collides with it, PD-disagg can save at most 100 ms — not 5 s.
2. **PD-disagg pays KV-transfer cost** (MB2: 300 ms – 10 s per request
for agentic sizes). For any KV ≥ ~80 MiB the cost already exceeds the
~100 ms benefit ceiling. Cost > benefit across the whole agentic
distribution.
PD-disagg's promised phase-isolation benefit is **per prefill event**,
not per request. When a new prefill arrives, it stalls every concurrent
decode stream on the same GPU. The aggregate decode time lost across
those D streams is `D × T_prefill`. PD-disagg moving prefill off-decode-GPU
recovers all of it.
## §3.2 cost-vs-benefit figure
Plugging numbers per prefill event:
`figs/pd_cost_vs_benefit.png` overlays MB1 benefit ceiling (50–200 ms
band, capped by decode duration) on top of MB2 transfer cost curve. The
cost curve crosses the benefit ceiling somewhere around **80 MiB / 830
tokens** of KV — well below the trace mean (192 MiB / 2k tok ≈ trace
mean per request KV, and we know agentic averages 33k tokens, p99
125k). For anything bigger PD-disagg pays more than it can recover.
| Prefill size | T_prefill | PD-disagg cost (MB2 T_transfer) | PD-disagg benefit (D=8 × T_prefill) | Ratio |
|---:|---:|---:|---:|---:|
| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s | 0.7 % |
| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s | 0.9 % |
| 125k tok (~p99) | 57 s | 1.9 s | 456 s | 0.4 % |
On the **phase-isolation axis alone**, PD-disagg wins by 100×–250×.
The reason static PD-disagg nonetheless **fails in agentic** is a
*different* failure mode: the D-side KV pool cannot fit p90+ requests
(p99 = 11.5 GiB; D-instance pool ≈ 38 GiB; 4P+4D halves system-wide
decode capacity → TTFT p50 62×, success rate 99.5% → 52% in colleague's
4P+4D experiment). The structural problem is **capacity** (see
`figs/f4b_pdsep_kv_wall.png`), not transfer-cost vs phase-isolation
trade-off.
## Reproduction
@@ -174,4 +187,7 @@ ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop
3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on
dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`.
Figures: `figs/mb1_interference.png`, `figs/pd_cost_vs_benefit.png`.
Figure: `figs/mb1_interference.png`. The figure
`figs/pd_cost_vs_benefit.png` from the original commit `029821c` was
based on the wrong "benefit ≤ decode duration" accounting; **deleted in
the correction commit**.

78
analysis/mb2/README.md
View File

@@ -24,12 +24,25 @@ get cheaper by co-locating P and D on the same node — the ~9.7 GB/s
ceiling applies regardless. Halving the transfer cost cannot be bought
back by topology.
**Headline for the paper §3.2**: at the agentic tail, **pure KV transfer
takes 1.5 – 10 s**. A median agentic decode is **50 – 200 ms** of tool-call
output. So **PD-disaggregation adds 8 – 100 × decode-time of transfer on
top of every routed request**. Phase isolation (the thing PD-disagg
trades transfer cost for) can only win back at most one decode duration
— for agentic that's negligible. The arithmetic is one-sided.
**What MB2 actually measures**: the **per-request charge** that
PD-disagg pays for every routed request — `T_transfer ≈ KV_size / 9.7
GB/s`. For agentic this is **8 ms (192 MiB / trace lower) – 1.9 s
(11.5 GiB / p99)**.
**⚠ Correction (2026-05-27)**: an earlier version of this README
framed §3.2 as "transfer cost (1.5–10 s) >> decode duration (50–200 ms),
so PD-disagg loses on cost-vs-benefit." That accounting was wrong:
PD-disagg's phase-isolation benefit is **per-prefill-event** and equals
`D × T_prefill` (aggregate across stalled decode streams), not the
single-request decode duration. With trace-mean `T_prefill = 4.5 s` and
D = 8, the benefit is ~36 s — far larger than the ~0.32 s transfer
cost. PD-disagg's phase-isolation axis is a *win*, not a loss.
The actual reason static PD-disagg fails in agentic is **D-side KV
capacity** (`figs/f4b_pdsep_kv_wall.png`), not a cost-vs-benefit
imbalance. See `RESULTS_SUMMARY.md` section 4 for the corrected
framing. MB2 still serves as the source of the per-request transfer
cost number used in that analysis.
---
@@ -137,43 +150,44 @@ analysis; not done yet.
treats them as additional samples (same sizes); the per-size
aggregates use all of them.
## Implications for §3.2 PD-disagg cost argument
## Implications for §3.2 PD-disagg argument
For each PD-disagg-routed request, transfer wall-time is:
```
T_transfer(KV_size) = max( pure_transfer(KV_size), rx_overhead )
≈ KV_size / 9.7 GB/s for KV_size <= 3 GiB
T_transfer(KV_size) ≈ KV_size / 9.7 GB/s for KV_size ≤ 3 GiB
≈ 0.3 – 10 s for KV_size in [3, 12] GiB
```
Agentic decode wall-time is typically 50 – 200 ms (tool-call output of
a few tens of tokens at ~50 tok/s). So the **transfer/decode ratio**
under intra-node best-case Mooncake is:
This is the **per-request transfer charge** of PD-disagg. It's a
real cost, but in the context of phase-isolation accounting it is
*small* compared to the benefit:
| KV size | T_transfer @9.7 GB/s | typical decode | T_transfer / T_decode |
|---|---:|---:|---:|
| 192 MiB (2k tok) | 20 ms | 100 ms | 0.2× |
| 768 MiB (8k tok) | 84 ms | 100 ms | 0.8× |
| 3 GiB (33k tok ≈ trace mean) | 321 ms | 100 ms | **3.2×** |
| 6 GiB (~p90) | 1900 ms | 100 ms | **19×** |
| 12 GiB (~p99) | 2800 ms | 100 ms | **28×** (median) – **100×** (p99 variance) |
| Prefill | T_prefill (MB1) | T_transfer (MB2) | Phase-isolation benefit at D=8 = D × T_prefill |
|---:|---:|---:|---:|
| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s |
| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s |
| 125k tok (~p99) | 57 s | 1.9 s | 456 s |
PD-disagg's promised payoff is *eliminating prefill–decode interference
on the decode instance*. The maximum benefit it can buy is bounded
above by the **decode duration itself** (you cannot recover more time
than the decode existed). For agentic that's 50 – 200 ms. The cost is
the table column above — 0.3 – 10 s of transfer per routed request.
On the phase-isolation axis alone, PD-disagg recovers two orders of
magnitude more decode time than it pays in transfer. **It is NOT this
axis that defeats static PD-disagg in agentic** — see colleague's
4P+4D experiment (TTFT p50 62×, success rate 99.5% → 52%) which is
driven by **D-side KV-pool overflow** on long-context requests
(`figs/f4b_pdsep_kv_wall.png`), not by transfer latency.
**Cost > Benefit by 5× to 100× across the agentic distribution.** Below
~3 GiB the ratio is small (≤1×); above 3 GiB the ratio explodes; above
6 GiB even individual draws can take 10 s for a single transfer.
What MB2 contributes to the paper is therefore:
- The **per-request transfer cost number** (used as the cost input
to the cost-benefit accounting above).
- The empirical observation that **Mooncake's transfer cost is
topology-independent** — intra-node and inter-node both go through
the RDMA NIC and hit the same 9.7 GB/s ceiling. PD-disagg's
transfer cost does not get cheaper by co-locating P and D.
This data alone is not the whole §3.2 argument — we still need to
account for D-side KV capacity (`f4b`, separate axis), cache reuse loss,
and static-partition mismatch (MB3 / MB4 / MB5). But it nails one
of the two key cost axes with measured numbers from vanilla mooncake,
not the dash0 patched build.
The dominant §3.2 failure mode of static PD-disagg in agentic is
**capacity**, not transfer cost. MB3 / MB4 / MB5 will quantify the
remaining axes (D-pool occupancy, cache reuse degradation under PD
routing, static-partition mismatch).
## Open questions / next runs

BIN
figs/mb1_interference.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 122 KiB

After

Width:  |  Height:  |  Size: 114 KiB

BIN
figs/pd_cost_vs_benefit.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 161 KiB

106
microbench/fresh_setup/plot_mb1.py
View File

@@ -1,20 +1,19 @@
#!/usr/bin/env python3
"""Plot MB1 interference results + the §3.2 cost-vs-benefit headline figure.
"""Plot MB1 phase-interference data.
Two outputs:
Single output: figs/mb1_interference.png — effective per-stream TPOT
during a prefill burst, vs prefill size, one line per concurrent decode
batch size D.
mb1_interference.png
Effective TPOT during prefill vs prefill size, one line per D.
Log-log. Annotates typical agentic decode duration (~100 ms) as a
horizontal band so reader can spot when decode would be stalled.
pd_cost_vs_benefit.png
The §3.2 headline. X axis: KV size (MiB). Two stacked curves:
- benefit ceiling (MB1) — at most one decode-duration per request
of phase isolation can be recovered. Drawn as a flat 100 ms line.
- cost (MB2) — Mooncake pure_transfer p50 at that size.
Anywhere the cost curve sits ABOVE the benefit ceiling, PD-disagg
structurally loses.
Earlier versions of this script also produced figs/pd_cost_vs_benefit.png
which composed a "max PD-disagg benefit = decode duration (50–200 ms)
band" against the MB2 transfer-cost curve. That accounting was wrong
(see RESULTS_SUMMARY.md §4 correction): phase-isolation benefit is
per-prefill-event, equal to D × T_prefill across stalled streams, not
capped by a single request's decode duration. That figure has been
removed; the math it implied was structurally backwards. The dominant
reason static PD-disagg fails in agentic is **D-side KV capacity**
(see figs/f4b_pdsep_kv_wall.png), not cost-vs-benefit on phase isolation.
"""
from __future__ import annotations
@@ -25,21 +24,16 @@ from pathlib import Path
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
def main() -> None:
p = argparse.ArgumentParser()
p.add_argument("--mb1", type=Path, required=True)
p.add_argument("--mb2-intra", type=Path, required=True)
p.add_argument("--mb2-inter", type=Path, default=None)
p.add_argument("--out-interf", type=Path, default=Path("figs/mb1_interference.png"))
p.add_argument("--out-cb", type=Path, default=Path("figs/pd_cost_vs_benefit.png"))
p.add_argument("--out", type=Path, default=Path("figs/mb1_interference.png"))
args = p.parse_args()
mb1 = json.loads(args.mb1.read_text())["summary"]
# ---- mb1_interference.png ----
fig, ax = plt.subplots(figsize=(9, 5.5))
Ds = sorted({s["decode_batch_size"] for s in mb1})
colors = {1: "#1f77b4", 4: "#ff7f0e", 8: "#d62728"}
@@ -50,79 +44,19 @@ def main() -> None:
ys = [s["effective_tpot_during_ms"] for s in rows]
ax.plot(xs, ys, "o-", lw=2, markersize=7,
color=colors.get(D, "gray"),
label=f"D={D} (baseline {rows[0]['baseline_tpot_ms']:.1f} ms)")
for tdec, lbl in [(50, "tool-call decode (~50 ms)"),
(100, "agentic decode (~100 ms)"),
(200, "long agentic decode (~200 ms)")]:
ax.axhline(tdec, color="#444", lw=0.6, ls=":", alpha=0.6)
ax.text(2200, tdec * 1.1, lbl, fontsize=8, color="#444")
label=f"D={D} (baseline TPOT {rows[0]['baseline_tpot_ms']:.1f} ms)")
ax.set_xscale("log"); ax.set_yscale("log")
ax.set_xlabel("Prefill burst size (tokens, log)")
ax.set_ylabel("Per-stream effective TPOT during prefill burst (ms, log)")
ax.set_title("MB1: each ongoing decode is essentially halted while prefill runs\n"
"(chunked-prefill ON, vLLM 0.18.1 default, single H20)")
"(chunked-prefill ON, vLLM 0.18.1 default, single H20). "
"Per-prefill aggregate stall = D × T_prefill.")
ax.grid(True, which="both", alpha=0.3)
ax.legend(loc="upper left", fontsize=9)
args.out_interf.parent.mkdir(parents=True, exist_ok=True)
fig.tight_layout(); fig.savefig(args.out_interf, dpi=150); plt.close(fig)
print(f"wrote {args.out_interf}")
# ---- pd_cost_vs_benefit.png ----
mb2_intra = json.loads(args.mb2_intra.read_text())["summary"]
mb2_intra = [s for s in mb2_intra if s["input_tokens"] >= 64]
intra_x_mib = [s["kv_mib"] for s in mb2_intra]
intra_y_ms = [s["pure_transfer_ms_p50"] for s in mb2_intra]
fig, ax = plt.subplots(figsize=(9, 5.5))
ax.plot(intra_x_mib, intra_y_ms, "o-", color="#d62728", lw=2.4,
markersize=8, label="MB2 PD-disagg KV transfer cost (Mooncake, p50)")
if args.mb2_inter:
mb2_inter = json.loads(args.mb2_inter.read_text())["summary"]
mb2_inter = [s for s in mb2_inter if s["input_tokens"] >= 64]
inter_x = [s["kv_mib"] for s in mb2_inter]
inter_y = [s["pure_transfer_ms_p50"] for s in mb2_inter]
ax.plot(inter_x, inter_y, "s--", color="#7a1d1d", lw=2, markersize=7,
alpha=0.7, label="MB2 inter-node (same numbers)")
# Benefit ceiling: typical agentic decode duration (PD-disagg max savings).
ax.axhline(100, color="#2ca02c", lw=2.4, ls="-",
label="MB1 max benefit ≤ agentic decode (~100 ms)")
ax.axhspan(50, 200, alpha=0.15, color="#2ca02c",
label="benefit range (50–200 ms decode)")
# Mark agentic-tail request sizes
for kv_mib, lbl in [(192, "trace mean\n(~2k tok)"),
(3072, "p90\n(~33k tok)"),
(6144, "p95\n(~65k tok)"),
(11500, "p99\n(11.5 GiB)")]:
ax.axvline(kv_mib, color="#666", lw=0.5, ls=":", alpha=0.5)
ax.text(kv_mib, 2, lbl, fontsize=8, color="#444",
ha="center", va="bottom")
ax.set_xscale("log"); ax.set_yscale("log")
ax.set_xlim(40, 14000)
ax.set_ylim(1, 12000)
ax.set_xlabel("Per-request KV size (MiB, log)")
ax.set_ylabel("Time per request (ms, log)")
ax.set_title("§3.2 headline — PD-disagg KV transfer cost vs phase-isolation benefit\n"
"(both measured on vanilla vLLM 0.18.1 + Mooncake 0.3.11, agentic regime)")
ax.grid(True, which="both", alpha=0.3)
ax.legend(loc="upper left", fontsize=9)
# Add explanatory annotation
ax.text(10000, 5000,
"Cost > benefit for ANY KV size above\n"
"the green band (~80 MiB / ~830 tokens).\n"
"Below: cost is marginal (<10 ms) but\n"
"benefit is also small (decode is short).",
fontsize=9, color="#333",
ha="right", va="top",
bbox=dict(boxstyle="round,pad=0.4", facecolor="#fffacd", alpha=0.9, edgecolor="#888"))
fig.tight_layout(); fig.savefig(args.out_cb, dpi=150); plt.close(fig)
print(f"wrote {args.out_cb}")
args.out.parent.mkdir(parents=True, exist_ok=True)
fig.tight_layout(); fig.savefig(args.out, dpi=150); plt.close(fig)
print(f"wrote {args.out}")
if __name__ == "__main__":

17
scripts/b3_isolated_policy.sh
View File

@@ -32,10 +32,14 @@ RUNDIR="${3:?usage: $0 <policy> <trace> <rundir>}"
# Auto-enable kv_both when the policy requires it.
# KV_CONNECTOR (Mooncake|Nixl) selects the underlying connector when KV_BOTH=1.
_KV_CONNECTOR_EXPLICIT="${KV_CONNECTOR:-}"
KV_CONNECTOR="${KV_CONNECTOR:-Mooncake}"
if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_kv_both" ]; then
if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_v3" ] || [ "$POLICY" = "unified_kv_both" ]; then
ENABLE_KV_BOTH=1
KV_CONNECTOR="Mooncake"
# honor explicit KV_CONNECTOR override (e.g. =Nixl); otherwise default Mooncake.
if [ -z "$_KV_CONNECTOR_EXPLICIT" ]; then
KV_CONNECTOR="Mooncake"
fi
fi
if [ "$POLICY" = "unified_nixl_both" ]; then
ENABLE_KV_BOTH=1
@@ -143,7 +147,10 @@ for i in $(seq 0 $((N_INSTANCES - 1))); do
combined_args="$combined_args http://127.0.0.1:$((BASE_PORT + i))"
done
proxy_extra=""
if [ "$ENABLE_KV_BOTH" = "1" ]; then
# Bootstrap ports only needed for Mooncake handshake. Nixl uses its own
# UCX side-channel and the proxy forwards kv_transfer_params from src's
# response body instead of pre-baking engine_id/bootstrap_addr.
if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Mooncake" ]; then
bp_list=""
for i in $(seq 0 $((N_INSTANCES - 1))); do
if [ -z "$bp_list" ]; then
@@ -154,11 +161,15 @@ if [ "$ENABLE_KV_BOTH" = "1" ]; then
done
proxy_extra="--bootstrap-ports $bp_list"
fi
if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Nixl" ]; then
proxy_extra="--connector-type nixl"
fi
nohup "$VENV/python" "$ROOT/scripts/cache_aware_proxy.py" \
--port "$PROXY_PORT" \
--combined $combined_args \
--policy "$POLICY" \
$proxy_extra \
${EXTRA_PROXY_ARGS:-} \
> "$RUNDIR/proxy.log" 2>&1 &
disown
tries=0

303
scripts/cache_aware_proxy.py
View File

@@ -82,6 +82,36 @@ class Settings:
# Patch 6.6: per-request KV-xfer wall-clock timeout (proxy side).
pd_sep_xfer_timeout_s: float = 60.0
# --- unified_v3 (offload-decode) gating knobs -----------------------
# v3 differs from v2 in *direction*: prefill stays on the session-
# affinity host (which holds the prefix cache); decode is migrated to
# a less-loaded target. KV transfer flows prefill_host → decode_target.
# The target doesn't need cache — we're shipping the post-prefill KV
# over anyway. After successful migration the session affinity table
# rotates to decode_target so the *next* turn lands where the KV now
# lives.
v3_min_new_tokens: int = 8000 # same as v2: don't migrate tiny prefills
v3_min_prefill_decode_busy: int = 1 # prefill_host must have ≥ this many concurrent decode tokens to justify migrating
v3_target_load_ratio: float = 0.7 # target.num_requests must be < prefill_host.num_requests × this
v3_min_load_gap: int = 1 # target.num_requests must also be ≤ prefill_host - this (absolute slack)
v3_rotate_affinity: bool = True # after migration, set session affinity to decode_target.
# Empirically False is better — see cache_miss_audit (next turn hits 9.5%
# with rotation vs ~80% without), because delay_free_blocks doesn't
# actually preserve cross-turn KV on decode_target.
v3_prefer_cache_target: bool = True # Mechanism B: among low-load candidates, prefer the one
# with the most prefix cache for this prompt — vLLM's connector
# auto-transfers only the missing portion (verified via
# smoke_partial_transfer: cache-rich dst is 77% faster than
# cold dst at 33k tokens, +512 ext).
# --- KV connector selection (governs PD-sep handshake) -------------
# "mooncake": pre-baked kv_transfer_params (bootstrap_addr+engine_id+transfer_id).
# Requires --bootstrap-ports and vLLMs launched with MooncakeConnector.
# "nixl" : response-forward handshake. src returns kv_transfer_params via
# response body, proxy forwards to dst. Nixl auto-selects transport
# via UCX (CUDA IPC / NVLink on intra-node, RDMA across nodes).
connector_type: str = "mooncake"
SETTINGS = Settings()
@@ -538,6 +568,122 @@ def pick_instance_unified_v2(
return chosen, chosen_idx, decision, None
def pick_instance_unified_v3(
instances: list[InstanceState],
token_ids: list[int] | None,
session_id: str | None,
input_length: int,
affinity: dict[str, int],
) -> tuple[InstanceState, int, dict, tuple[InstanceState, int] | None]:
"""unified_v3 = unified hybrid + selective DECODE migration.
Direction-reversed from unified_v2:
- prefill stays on session-affinity host (`prefill_host`) so we keep
the 93%-intra-session prefix-cache reuse intact.
- decode is migrated to a lower-load `decode_target` when the
affinity host is busy with concurrent decodes.
- KV transfer flows prefill_host → decode_target (the opposite of
v2's src → chosen).
- target does NOT need pre-existing cache — we're shipping the
post-prefill KV over anyway.
- On successful migration the *caller* must rotate
`affinity[session_id] = decode_target_idx` so the next turn lands
where the KV now lives (decode_target retains the blocks after
completion, since mooncake defaults to delay_free_blocks=True).
Decision is purely load-based on the target side:
1. new_local ≥ v3_min_new_tokens (don't pay RDMA for tiny prefills)
2. prefill_host.ongoing_decode_tokens ≥ v3_min_prefill_decode_busy
(the host is actually busy decoding; migration buys decode-bw)
3. ∃ target with both num_requests < prefill_host.num_requests × ratio
and num_requests ≤ prefill_host.num_requests − v3_min_load_gap
Returns (prefill_host, prefill_idx, decision, migrate). When migrate
is None the request is fully local on prefill_host. When migrate is
(decode_target_inst, decode_target_idx), the handler should run
prefill on prefill_host and ship KV to decode_target for decode.
"""
prefill_host, prefill_idx, decision = pick_instance_unified_hybrid(
instances, token_ids, session_id, input_length, affinity)
decision["v3_migrate"] = False
decision["v3_decision"] = "local"
decision["v3_reason"] = None
if not token_ids:
decision["v3_reason"] = "no_token_ids"
return prefill_host, prefill_idx, decision, None
prefill_cache_hit = prefill_host.estimate_cache_hit(token_ids)
new_local = max(0, input_length - prefill_cache_hit)
decision["v3_prefill_cache_hit"] = prefill_cache_hit
decision["v3_new_local"] = new_local
# Gate 1: prefill must be large enough to amortise RDMA setup.
if new_local < SETTINGS.v3_min_new_tokens:
decision["v3_reason"] = (
f"new_local_below_threshold ({new_local} < {SETTINGS.v3_min_new_tokens})"
)
return prefill_host, prefill_idx, decision, None
# Gate 2: affinity host must be busy with concurrent decodes — that's
# what migrating decode-traffic-away buys us. If the host is idle
# there's no point.
if prefill_host.ongoing_decode_tokens < SETTINGS.v3_min_prefill_decode_busy:
decision["v3_reason"] = (
f"prefill_host_not_busy "
f"(ongoing_decode_tokens={prefill_host.ongoing_decode_tokens} < "
f"{SETTINGS.v3_min_prefill_decode_busy})"
)
return prefill_host, prefill_idx, decision, None
# Gate 3: pick the lowest-load target that is materially less loaded
# than the prefill_host. Cache content irrelevant — KV ships over.
threshold_loaded = max(1,
int(prefill_host.num_requests * SETTINGS.v3_target_load_ratio))
candidates = [
(i, inst) for i, inst in enumerate(instances)
if i != prefill_idx
and inst.num_requests < threshold_loaded
and inst.num_requests <= prefill_host.num_requests - SETTINGS.v3_min_load_gap
]
if not candidates:
decision["v3_reason"] = (
f"no_low_load_target "
f"(prefill_host.num_req={prefill_host.num_requests} "
f"threshold={threshold_loaded})"
)
return prefill_host, prefill_idx, decision, None
# Mechanism B (v3_prefer_cache_target=True): rank candidates first by
# cache_hit DESC (more cache = less KV to transfer), then by load. vLLM
# auto-skips transferring overlapping prefix when dst's local cache
# matches — verified in smoke_partial_transfer: 77% faster on a 33k
# prompt when dst has the prefix already.
if SETTINGS.v3_prefer_cache_target:
decode_target_idx, decode_target = min(
candidates,
key=lambda x: (-x[1].estimate_cache_hit(token_ids),
x[1].num_requests, x[1].ongoing_tokens))
else:
decode_target_idx, decode_target = min(
candidates, key=lambda x: (x[1].num_requests, x[1].ongoing_tokens))
target_cache_hit = decode_target.estimate_cache_hit(token_ids)
decision["v3_migrate"] = True
decision["v3_decision"] = "migrate_decode"
decision["v3_target_idx"] = decode_target_idx
decision["v3_target_num_req"] = decode_target.num_requests
decision["v3_target_cache_hit"] = target_cache_hit
decision["v3_prefill_num_req"] = prefill_host.num_requests
decision["v3_reason"] = (
f"prefill_host.num_req={prefill_host.num_requests} busy; "
f"target.num_req={decode_target.num_requests} cache_hit={target_cache_hit}, "
f"transferring KV after prefill"
)
return prefill_host, prefill_idx, decision, (decode_target, decode_target_idx)
def _extract_output_token_ids_from_sse(
buffer: str,
chunk: bytes,
@@ -764,10 +910,11 @@ async def lifespan(app: FastAPI):
policy = getattr(global_args, 'policy', 'linear')
# Mooncake-based modes still need bootstrap discovery; NIXL uses
# its own UCX side-channel and doesn't go through our proxy
# bootstrap path (and unified_nixl_both never PD-seps anyway).
# bootstrap path. With --connector-type=nixl, v3 also skips bootstrap.
needs_bootstrap = (
global_args.offload
or policy in ("unified_v2", "unified_kv_both")
or (policy in ("unified_v2", "unified_v3", "unified_kv_both")
and getattr(global_args, 'connector_type', 'mooncake') == 'mooncake')
)
if needs_bootstrap and bp_list:
await init_prefill_bootstrap(combined_instances, app.state.ready)
@@ -953,6 +1100,26 @@ async def _handle_combined(api, req_data, token_ids, input_length, session_id, h
breakdown.update(decision)
if session_id:
session_affinity_combined[session_id] = best_idx
elif policy == "unified_v3":
# v3: prefill on affinity (cache reuse), decode migrated to a
# low-load target. KV flows prefill_host → decode_target.
# Reuses _handle_combined_pd_sep_v2 with src=prefill_host,
# dst=decode_target (the handler is direction-agnostic).
chosen, best_idx, decision, pd_sep_v2 = pick_instance_unified_v3(
combined_instances, token_ids, session_id, input_length,
session_affinity_combined)
breakdown.update(decision)
if session_id:
if pd_sep_v2 is not None and SETTINGS.v3_rotate_affinity:
# Migration + rotation: redirect next turn to decode_target,
# assuming KV will live there. (Empirically wrong — see
# cache_miss_audit. Keep behind a flag.)
_decode_target_inst, decode_target_idx = pd_sep_v2
session_affinity_combined[session_id] = decode_target_idx
else:
# No rotation: keep affinity on prefill_host (where the prefix
# cache lives). This is the empirically correct choice.
session_affinity_combined[session_id] = best_idx
else: # linear (default)
chosen, best_idx = pick_instance(
combined_instances, token_ids, session_id, input_length,
@@ -983,13 +1150,35 @@ async def _handle_combined(api, req_data, token_ids, input_length, session_id, h
})
if pd_sep_v2 is not None:
src_inst, src_idx = pd_sep_v2
breakdown["v2_src_url"] = src_inst.url
breakdown["v2_src_idx"] = src_idx
return await _handle_combined_pd_sep_v2(
api, req_data, headers, token_ids, input_length,
src_inst, chosen, breakdown,
request_id=request_id)
# Handler contract: first arg = prefill source (does same-worker
# prefill with do_remote_decode=True, max_tokens=1), second arg =
# decode target (does do_remote_prefill=True, pulls KV via
# Mooncake, decodes).
#
# v2 contract: pd_sep_v2 = (src_inst, src_idx); chosen = decode
# → src does prefill (it has more cache), chosen decodes.
# v3 contract: chosen = prefill_host (affinity, has cache);
# pd_sep_v2 = (decode_target_inst, decode_target_idx)
# → chosen does prefill (cache reuse), decode_target decodes.
if policy == "unified_v3":
decode_target_inst, decode_target_idx = pd_sep_v2
prefill_inst = chosen
breakdown["v2_src_url"] = prefill_inst.url
breakdown["v2_src_idx"] = best_idx
breakdown["v3_decode_target_url"] = decode_target_inst.url
breakdown["v3_decode_target_idx"] = decode_target_idx
return await _handle_combined_pd_sep_v2(
api, req_data, headers, token_ids, input_length,
prefill_inst, decode_target_inst, breakdown,
request_id=request_id)
else:
src_inst, src_idx = pd_sep_v2
breakdown["v2_src_url"] = src_inst.url
breakdown["v2_src_idx"] = src_idx
return await _handle_combined_pd_sep_v2(
api, req_data, headers, token_ids, input_length,
src_inst, chosen, breakdown,
request_id=request_id)
return await _handle_local_request(
api, req_data, headers, token_ids, input_length,
@@ -1011,11 +1200,12 @@ async def _handle_combined_pd_sep_v2(
of SETTINGS.pd_sep_xfer_timeout_s, so a stuck KV transfer fails
the request instead of hanging for 600 s.
"""
if src.bootstrap_port is None:
connector = SETTINGS.connector_type
if connector == "mooncake" and src.bootstrap_port is None:
raise HTTPException(
status_code=500,
detail=(
"unified_v2 PD-sep triggered but src instance "
"Mooncake PD-sep triggered but src instance "
f"{src.url} has no bootstrap_port; launch with "
"kv_role=kv_both and pass --bootstrap-ports"
),
@@ -1029,12 +1219,16 @@ async def _handle_combined_pd_sep_v2(
src_load_held = True
dst_load_held = True
# Build prefill kv_transfer_params per connector.
prefill_data = req_data.copy()
prefill_data["kv_transfer_params"] = {
"do_remote_decode": True,
"do_remote_prefill": False,
"transfer_id": f"xfer-{request_id}",
}
if connector == "mooncake":
prefill_data["kv_transfer_params"] = {
"do_remote_decode": True,
"do_remote_prefill": False,
"transfer_id": f"xfer-{request_id}",
}
else: # nixl: src just signals it'll produce KV for remote decode
prefill_data["kv_transfer_params"] = {"do_remote_decode": True}
prefill_data["stream"] = False
prefill_data["max_tokens"] = 1
prefill_data["min_tokens"] = 1
@@ -1044,11 +1238,23 @@ async def _handle_combined_pd_sep_v2(
breakdown["t_prefill_sent"] = _time.monotonic()
breakdown["t_prefill_sent_unix"] = _time.time()
forwarded_params: dict | None = None
try:
resp = await src.client.post(api, json=prefill_data, headers=p_headers)
breakdown["t_prefill_done"] = _time.monotonic()
breakdown["t_prefill_done_unix"] = _time.time()
resp.raise_for_status()
if connector == "nixl":
# Nixl populates kv_transfer_params in the response body with
# remote_block_ids / remote_engine_id / remote_host / remote_port.
# We must read the body BEFORE aclose.
src_resp_json = resp.json()
forwarded_params = src_resp_json.get("kv_transfer_params")
if not forwarded_params or not forwarded_params.get("remote_block_ids"):
raise HTTPException(
status_code=502,
detail=f"Nixl src returned no remote_block_ids: {forwarded_params}",
)
await resp.aclose()
src.record_prefix(token_ids)
except Exception as e:
@@ -1057,11 +1263,16 @@ async def _handle_combined_pd_sep_v2(
breakdown["prefill_error"] = True
breakdown["error_detail"] = repr(e)[:300]
_breakdown_log.append(breakdown)
# Release reservations on failure.
src.ongoing_tokens -= input_length
src.num_requests -= 1
dst.ongoing_tokens -= input_length
dst.num_requests -= 1
# Release reservations on failure. Clear load_held flags so the
# finally block below does not double-decrement (CRITICAL audit #1).
if src_load_held:
src.ongoing_tokens -= input_length
src.num_requests -= 1
src_load_held = False
if dst_load_held:
dst.ongoing_tokens -= input_length
dst.num_requests -= 1
dst_load_held = False
raise HTTPException(status_code=502, detail=f"Prefill failed: {e}")
finally:
if src_load_held:
@@ -1069,17 +1280,19 @@ async def _handle_combined_pd_sep_v2(
src.num_requests -= 1
src_load_held = False
parsed = urllib.parse.urlparse(str(src.client.base_url))
bootstrap_addr = f"http://{parsed.hostname}:{src.bootstrap_port}"
decode_data = req_data.copy()
decode_data["kv_transfer_params"] = {
"do_remote_decode": False,
"do_remote_prefill": True,
"remote_bootstrap_addr": bootstrap_addr,
"remote_engine_id": src.engine_id.get(0, ""),
"transfer_id": f"xfer-{request_id}",
}
if connector == "mooncake":
parsed = urllib.parse.urlparse(str(src.client.base_url))
bootstrap_addr = f"http://{parsed.hostname}:{src.bootstrap_port}"
decode_data["kv_transfer_params"] = {
"do_remote_decode": False,
"do_remote_prefill": True,
"remote_bootstrap_addr": bootstrap_addr,
"remote_engine_id": src.engine_id.get(0, ""),
"transfer_id": f"xfer-{request_id}",
}
else: # nixl: forward what src returned
decode_data["kv_transfer_params"] = forwarded_params
breakdown["t_decode_sent"] = _time.monotonic()
breakdown["t_decode_sent_unix"] = _time.time()
@@ -1304,7 +1517,8 @@ def parse_args():
p.add_argument("--policy", type=str, default="linear",
choices=["linear", "lmetric", "load_only", "sticky",
"unified", "unified_kv_both",
"unified_nixl_both", "unified_v2"],
"unified_nixl_both", "unified_v2",
"unified_v3"],
help="Routing policy: linear (cache-aware), lmetric (P_tokens × BS), "
"load_only (B3 control: pure min-num_requests), "
"sticky (B3 control: hard session affinity), "
@@ -1317,6 +1531,20 @@ def parse_args():
"or unified_v2 (unified + selective per-request PD-sep "
"via Mooncake; requires --bootstrap-ports and "
"kv_role=kv_both vLLM launch)")
p.add_argument("--v3-rotate-affinity", type=int, default=1, choices=[0,1],
help="unified_v3 only: 1 = rotate session affinity to decode_target "
"after migration (original behavior, empirically loses prefix cache); "
"0 = keep affinity on prefill_host so next turn hits its cache.")
p.add_argument("--connector-type", type=str, default="mooncake",
choices=["mooncake", "nixl"],
help="PD-sep handshake protocol. 'mooncake' uses pre-baked engine_id"
" + bootstrap_addr (requires --bootstrap-ports). 'nixl' uses"
" response-forward (src returns kv_transfer_params, proxy"
" relays to dst; Nixl/UCX auto-picks NVLink intra-node).")
p.add_argument("--v3-prefer-cache-target", type=int, default=1, choices=[0,1],
help="Mechanism B: unified_v3 picks decode_target with the most"
" prefix cache among low-load candidates (default 1). Set 0"
" to fall back to pure-load tie-break (cache-blind).")
p.add_argument("--overload-factor", type=float, default=2.0,
help="Break session affinity when instance load > factor * avg")
# The four flags below are accepted for bench.sh backward compatibility but
@@ -1354,7 +1582,14 @@ if __name__ == "__main__":
SETTINGS.max_offload_inflight = global_args.max_offload_inflight
SETTINGS.cache_gate_ratio = global_args.cache_gate_ratio
SETTINGS.decode_iteration_s = getattr(global_args, 'decode_iteration_s', 0.05)
print("SETTINGS: throughput=%.0f rdma_overhead=%.2f offload=%s" % (
SETTINGS.v3_rotate_affinity = bool(getattr(global_args, 'v3_rotate_affinity', 1))
SETTINGS.connector_type = getattr(global_args, 'connector_type', 'mooncake')
SETTINGS.v3_prefer_cache_target = bool(getattr(global_args, 'v3_prefer_cache_target', 1))
print("SETTINGS: throughput=%.0f rdma_overhead=%.2f offload=%s v3_rotate_affinity=%s "
"connector_type=%s v3_prefer_cache_target=%s" % (
SETTINGS.prefill_throughput, SETTINGS.rdma_overhead_s,
getattr(global_args, 'offload', False)))
getattr(global_args, 'offload', False),
SETTINGS.v3_rotate_affinity,
SETTINGS.connector_type,
SETTINGS.v3_prefer_cache_target))
uvicorn.run(app, host=global_args.host, port=global_args.port)