diff --git a/RESULTS_SUMMARY.md b/RESULTS_SUMMARY.md index d9f790b..e8ec1d9 100644 --- a/RESULTS_SUMMARY.md +++ b/RESULTS_SUMMARY.md @@ -39,52 +39,63 @@ Production trace = Qwen3-Coder agentic，1.3 M sessions / 2.1 M reqs / 7200 s。 参考图：`figs/f4a_apc_loss.png`、`figs/f4b_pdsep_kv_wall.png`、`figs/f4c_per_worker_ttft.png`、`figs/f6_e2e_latency_bars.png`、`figs/f6_e2e_latency_full_grid.png`。 -## 4. PD-disagg 在 agentic 下输——cost vs benefit（§3.2） +## 4. Static PD-disagg 为什么失败（§3.2）—— 容量问题，不是 cost-benefit 问题 -由两个独立 microbench 钉死（**全用 vanilla vLLM 0.18.1 + Mooncake 0.3.11，fresh venv，无 patch**）。 +⚠ **2026-05-27 纠正**：本节前一版本论证"PD-disagg 因为 transfer cost > phase isolation benefit 而失败"。**这个论证算错了**。正确的 phase-isolation benefit 是**每个 prefill 事件 × D 个 concurrent stream** 的总和（≈ `D × T_prefill`），不是单个 request 的 decode 时长。用正确公式，PD-disagg 在 phase-isolation 这一维上**赢 colo 一两个数量级**。Static PD-disagg 在 agentic 上失败的**真正根因是 D 侧 KV pool 容量**，不是这一维。 -### 4.1 MB2 — KV transfer cost +### 4.1 真正的失败模式：D 侧 KV 容量天花板 -dash1 GPU 0+1（intra-node）和 dash1 ↔ dash2（inter-node, 200 Gbps RoCE）扫 9 个 size × 5 reps。 +| | 8C colo | 4P+4D PD-disagg | +|---|---:|---:| +| Per-D-instance KV pool（0.4 × 96 GiB） | 38 GiB | 38 GiB | +| 系统总 decode 容量（D 实例数 × 单池） | 8 × 38 = **304 GiB** | 4 × 38 = **152 GiB** | +| p99 单请求 KV = 11.5 GiB → 最多并发 decode | 24 | **12（减半）** | -| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求（11.5 GiB）transfer 时间 | +Colleague 4P+4D 实测：TTFT p50 0.91 s → **62.8 s（62×）**、success rate **99.5% → 52%**。失败模式：**D 池溢出 + 排队**，不是 transfer 延迟。 + +参考图：`figs/f4b_pdsep_kv_wall.png`（pdf 版本是高质量 paper figure）。 + +### 4.2 MB2 — KV transfer cost（per-request 一次性成本，**不**是 dominant cost） + +dash1 GPU 0+1（intra）和 dash1 ↔ dash2（inter, 200 Gbps RoCE）扫 9 个 size × 5 reps。 + +| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求 11.5 GiB transfer | |---|---|---| -| Intra-node | **9.7 GB/s** | p50 **1.9 s** · min 1.5 s · max 10 s | -| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · min 1.3 s · max 9.2 s | +| Intra-node | **9.7 GB/s** | p50 **1.9 s** · max 10 s | +| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · max 9.2 s | -**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC，包括 intra-node loopback**，不走 NVLink。200 Gbps NIC 是天花板，**PD-disagg 的 transfer cost 与拓扑无关**。 +**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC**，不走 NVLink。200 Gbps NIC 是天花板。**PD-disagg transfer cost 与拓扑无关**。 -参考图：`figs/mb2_transfer_time_compare.png`、`figs/mb2_transfer_bw_compare.png`、doc `analysis/mb2/README.md`。 +参考图：`figs/mb2_transfer_time_compare.png`、doc `analysis/mb2/README.md`。 -### 4.2 MB1 — Phase interference（chunked-prefill on, 默认 baseline） +### 4.3 MB1 — Phase interference（PD-disagg 的潜在 benefit 上界） -dash1 GPU 0 单 instance，D（concurrent decodes）× P（prefill size）扫描。 +dash1 GPU 0 单 instance（无 kv_connector），chunked-prefill 默认开启，D × P 扫描。D=8 结果： -D=8（最 agentic-realistic）的结果： - -| Prefill | prefill_ttft | per-stream TPOT during | penalty | +| Prefill | T_prefill | per-stream TPOT during | penalty | |---|---:|---:|---:| | 2k tok | 143 ms | 32 ms | 4× | -| 8k | 583 ms | 114 ms | 15× | -| 32k | 4.5 s | 388 ms | **52×** | -| 65k | 15.6 s | 757 ms | **99×** | -| 131k | 57 s | 1419 ms | **183×** | +| 32k tok | 4.5 s | 388 ms | **52×** | +| 131k tok | 57 s | 1419 ms | **183×** | -baseline TPOT 7.7 ms。**Decode 在大 prefill 期间基本被 halted**。chunked-prefill 已经默认开启，PD-disagg 在它之上能额外提供的 phase isolation = **decode 在 prefill 期间被 halted 的那部分时间**。 +**Decode 在 prefill 期间被几乎完全 halted**，单 stream 损失 ≈ `T_prefill` 的时间。**每个 prefill event 总 decode 损失 ≈ `D × T_prefill`**。 参考图：`figs/mb1_interference.png`、doc `analysis/mb1/README.md`。 -### 4.3 联合结论 +### 4.4 联合 cost-benefit（per-prefill event） -| | Per-request | -|---|---| -| **Max PD-disagg benefit**（救回来的 decode 时间）| ≤ **decode 时长 = 50–200 ms**（agentic tool-call output）| -| **PD-disagg cost**（MB2 transfer p50）| 80 MiB ≈ 8 ms · 3 GiB ≈ 320 ms · 11.5 GiB ≈ **1.9 s**（p99 实测最差 10 s）| -| Cost / Benefit | **每个 KV ≥ 80 MiB 的请求都输**；trace 平均 KV 192 MiB → 已经输 | +| Prefill (KV size) | T_prefill | Cost = T_transfer | Benefit = D × T_prefill (D=8) | Cost / Benefit | +|---:|---:|---:|---:|---:| +| 2k tok (192 MiB) | 0.14 s | 8 ms | 1.1 s | **0.7%** | +| 33k tok (3 GiB, trace mean) | 4.5 s | 0.32 s | 36 s | **0.9%** | +| 125k tok (12 GiB, ~p99) | 57 s | 1.9 s | 456 s | **0.4%** | -**结论**：在 agentic 上 **PD-disaggregation 是结构性失败的**。Chunked-prefill 默认已经在 colocation 内做了 first-order phase isolation；PD-disagg 在此之上能额外补的（decode 短时段没被 prefill 挤）小于它新带来的（每个 routed 请求都付 KV transfer）。这个结论与拓扑无关（intra-node 和 inter-node 一样）。 +**PD-disagg 在 phase-isolation 这一维赢 100×–250×**。但**这不是 §3.2 该用的论证**，因为 §3.2 真正的 dominant failure 是 §4.1 的 D 池容量天花板（颠覆了上面的全部数学）。 -参考图：`figs/pd_cost_vs_benefit.png`（§3.2 headline）。 +**总结**： +- D 侧 KV 容量天花板（§4.1）→ PD-disagg 在 agentic 上**结构性失败**。 +- MB1 + MB2 的合计 cost-benefit 在 phase isolation 维度上 PD-disagg 是赢的，**但这件事被容量天花板压倒**。 +- Paper §3.2 论证应该聚焦"D 池装不下"，MB1/MB2 数据用作 supporting context（per-request transfer charge 量化、phase isolation benefit 量化）而不是 main argument。 ## 5. EAR 设计的实证状态（§4） @@ -96,10 +107,12 @@ baseline TPOT 7.7 ms。**Decode 在大 prefill 期间基本被 halted**。chunke ## 6. 已经能写的 paper 主张（按 confidence 排序） 1. **Agentic vs chatbot 在调度上是不同 regime**（dispatch coupling + sub-second tool-call mass）—— 实证完整 -2. **PD-disaggregation 在 agentic 下输**（cost > benefit，跨拓扑）—— **MB1 + MB2 实证完整** -3. **三类现有调度 baseline 各自的失败模式** —— 实证完整 -4. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整 -5. **Hot-triggered migration 修复 sticky 的 hot pin** —— **design 完整、e2e 待验证** +2. **三类现有调度 baseline 各自的失败模式**（load-balance / static PD-disagg / pure sticky）—— 实证完整 +3. **Static PD-disagg 在 agentic 下失败的 dominant 根因是 D 侧 KV 容量**（不是 phase-isolation cost-benefit）—— 实证完整（`f4b` + colleague 4P+4D 数据） +4. **Mooncake transfer cost 拓扑无关**（intra ≈ inter，~9.7 GB/s NIC 上限）—— 实证完整（MB2） +5. **Phase isolation interference 在 chunked-prefill on 下仍然显著**（per-stream TPOT during prefill 15×–2000× baseline）—— 实证完整（MB1）。**注意**：这条数据本身不直接论证 "PD-disagg 失败"，因为算正确账后 PD-disagg 反而在这一维上赢；它的用途是给 §3.2 提供 phase-isolation benefit 上界的量化。 +6. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整 +7. **Hot-triggered migration 修复 sticky 的 hot pin** —— design 完整、e2e 待验证 ## 7. 待做 diff --git a/analysis/mb1/README.md b/analysis/mb1/README.md index 1d56b0d..cd8af64 100644 --- a/analysis/mb1/README.md +++ b/analysis/mb1/README.md @@ -15,20 +15,28 @@ bottom; the **Summary** block is what gets cited. | Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** | | Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** | | Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** | -| Maximum PD-disagg benefit per agentic decode | **≤ 50–200 ms** (= decode duration) | -**§3.2 headline (cost vs benefit, this run + MB2)**: +**What MB1 actually measures**: -> Under chunked-prefill, every ongoing decode stream is essentially -> **halted while a prefill chunk is in flight** — per-stream effective -> TPOT during the burst is 15× to 2000× baseline, scaling with prefill -> size. PD-disagg can recover this stall, but the recovery is bounded by -> the **decode duration** of the request being protected. For agentic, -> decode is 50–200 ms (tool-call output). MB2 shows PD-disagg pays -> 300 ms – 10 s of KV-transfer cost per request to do that recovery. The -> cost exceeds the benefit ceiling for any per-request KV ≥ ~80 MiB -> (~830 tokens) — well below all agentic operating points. The benefit -> never beats the cost in this workload. +> During a prefill burst, every ongoing decode stream is essentially +> halted (per-stream effective TPOT is 15×–2000× baseline, scaling with +> prefill size). The **total decode time lost per prefill event is +> `D × T_prefill`** (D concurrent decodes each lose ~T_prefill of useful +> work). For the trace mean (P ≈ 33k tokens, T_prefill ≈ 4.5 s) at D=8 +> that's **~36 seconds of decode-equivalent work lost per request**. +> This is the **upper bound on what PD-disaggregation's phase isolation +> could recover** on the decode side. + +**⚠ Correction (2026-05-27)**: an earlier version of this README framed +the §3.2 PD-disagg argument as "phase-isolation benefit is capped at +the decode duration of the new request (~50–200 ms), so MB2 transfer +cost dominates". That framing was wrong. The correct accounting is +benefit-per-prefill-event = D × T_prefill (aggregate decode time saved +across all stalled streams), which is **much larger than per-request +transfer cost**. The actual reason static PD-disagg fails in agentic +is **D-side KV pool capacity** (`figs/f4b_pdsep_kv_wall.png`), not a +cost-vs-benefit imbalance on phase isolation. See `RESULTS_SUMMARY.md` +section 4 for the corrected framing. ## Setup @@ -107,25 +115,30 @@ the cleanest "average over the whole burst window" number). halts decode). This is the entire prefill duration's worth of decode time that could in principle be recovered. -Two big caveats for **agentic** application: +**Connecting to the §3.2 PD-disagg argument** (corrected): -1. **Decode is short** (~50–200 ms for tool-call output). The actual - recoverable benefit per request is bounded by the decode duration, - not by `prefill_ttft`. If a decode lasts 100 ms and a 5-second prefill - collides with it, PD-disagg can save at most 100 ms — not 5 s. -2. **PD-disagg pays KV-transfer cost** (MB2: 300 ms – 10 s per request - for agentic sizes). For any KV ≥ ~80 MiB the cost already exceeds the - ~100 ms benefit ceiling. Cost > benefit across the whole agentic - distribution. +PD-disagg's promised phase-isolation benefit is **per prefill event**, +not per request. When a new prefill arrives, it stalls every concurrent +decode stream on the same GPU. The aggregate decode time lost across +those D streams is `D × T_prefill`. PD-disagg moving prefill off-decode-GPU +recovers all of it. -## §3.2 cost-vs-benefit figure +Plugging numbers per prefill event: -`figs/pd_cost_vs_benefit.png` overlays MB1 benefit ceiling (50–200 ms -band, capped by decode duration) on top of MB2 transfer cost curve. The -cost curve crosses the benefit ceiling somewhere around **80 MiB / 830 -tokens** of KV — well below the trace mean (192 MiB / 2k tok ≈ trace -mean per request KV, and we know agentic averages 33k tokens, p99 -125k). For anything bigger PD-disagg pays more than it can recover. +| Prefill size | T_prefill | PD-disagg cost (MB2 T_transfer) | PD-disagg benefit (D=8 × T_prefill) | Ratio | +|---:|---:|---:|---:|---:| +| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s | 0.7 % | +| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s | 0.9 % | +| 125k tok (~p99) | 57 s | 1.9 s | 456 s | 0.4 % | + +On the **phase-isolation axis alone**, PD-disagg wins by 100×–250×. +The reason static PD-disagg nonetheless **fails in agentic** is a +*different* failure mode: the D-side KV pool cannot fit p90+ requests +(p99 = 11.5 GiB; D-instance pool ≈ 38 GiB; 4P+4D halves system-wide +decode capacity → TTFT p50 62×, success rate 99.5% → 52% in colleague's +4P+4D experiment). The structural problem is **capacity** (see +`figs/f4b_pdsep_kv_wall.png`), not transfer-cost vs phase-isolation +trade-off. ## Reproduction @@ -174,4 +187,7 @@ ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop 3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`. -Figures: `figs/mb1_interference.png`, `figs/pd_cost_vs_benefit.png`. +Figure: `figs/mb1_interference.png`. The figure +`figs/pd_cost_vs_benefit.png` from the original commit `029821c` was +based on the wrong "benefit ≤ decode duration" accounting; **deleted in +the correction commit**. diff --git a/analysis/mb2/README.md b/analysis/mb2/README.md index 549d6e3..41b7ef1 100644 --- a/analysis/mb2/README.md +++ b/analysis/mb2/README.md @@ -24,12 +24,25 @@ get cheaper by co-locating P and D on the same node — the ~9.7 GB/s ceiling applies regardless. Halving the transfer cost cannot be bought back by topology. -**Headline for the paper §3.2**: at the agentic tail, **pure KV transfer -takes 1.5 – 10 s**. A median agentic decode is **50 – 200 ms** of tool-call -output. So **PD-disaggregation adds 8 – 100 × decode-time of transfer on -top of every routed request**. Phase isolation (the thing PD-disagg -trades transfer cost for) can only win back at most one decode duration -— for agentic that's negligible. The arithmetic is one-sided. +**What MB2 actually measures**: the **per-request charge** that +PD-disagg pays for every routed request — `T_transfer ≈ KV_size / 9.7 +GB/s`. For agentic this is **8 ms (192 MiB / trace lower) – 1.9 s +(11.5 GiB / p99)**. + +**⚠ Correction (2026-05-27)**: an earlier version of this README +framed §3.2 as "transfer cost (1.5–10 s) >> decode duration (50–200 ms), +so PD-disagg loses on cost-vs-benefit." That accounting was wrong: +PD-disagg's phase-isolation benefit is **per-prefill-event** and equals +`D × T_prefill` (aggregate across stalled decode streams), not the +single-request decode duration. With trace-mean `T_prefill = 4.5 s` and +D = 8, the benefit is ~36 s — far larger than the ~0.32 s transfer +cost. PD-disagg's phase-isolation axis is a *win*, not a loss. + +The actual reason static PD-disagg fails in agentic is **D-side KV +capacity** (`figs/f4b_pdsep_kv_wall.png`), not a cost-vs-benefit +imbalance. See `RESULTS_SUMMARY.md` section 4 for the corrected +framing. MB2 still serves as the source of the per-request transfer +cost number used in that analysis. --- @@ -137,43 +150,44 @@ analysis; not done yet. treats them as additional samples (same sizes); the per-size aggregates use all of them. -## Implications for §3.2 PD-disagg cost argument +## Implications for §3.2 PD-disagg argument For each PD-disagg-routed request, transfer wall-time is: ``` -T_transfer(KV_size) = max( pure_transfer(KV_size), rx_overhead ) - ≈ KV_size / 9.7 GB/s for KV_size <= 3 GiB +T_transfer(KV_size) ≈ KV_size / 9.7 GB/s for KV_size ≤ 3 GiB ≈ 0.3 – 10 s for KV_size in [3, 12] GiB ``` -Agentic decode wall-time is typically 50 – 200 ms (tool-call output of -a few tens of tokens at ~50 tok/s). So the **transfer/decode ratio** -under intra-node best-case Mooncake is: +This is the **per-request transfer charge** of PD-disagg. It's a +real cost, but in the context of phase-isolation accounting it is +*small* compared to the benefit: -| KV size | T_transfer @9.7 GB/s | typical decode | T_transfer / T_decode | -|---|---:|---:|---:| -| 192 MiB (2k tok) | 20 ms | 100 ms | 0.2× | -| 768 MiB (8k tok) | 84 ms | 100 ms | 0.8× | -| 3 GiB (33k tok ≈ trace mean) | 321 ms | 100 ms | **3.2×** | -| 6 GiB (~p90) | 1900 ms | 100 ms | **19×** | -| 12 GiB (~p99) | 2800 ms | 100 ms | **28×** (median) – **100×** (p99 variance) | +| Prefill | T_prefill (MB1) | T_transfer (MB2) | Phase-isolation benefit at D=8 = D × T_prefill | +|---:|---:|---:|---:| +| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s | +| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s | +| 125k tok (~p99) | 57 s | 1.9 s | 456 s | -PD-disagg's promised payoff is *eliminating prefill–decode interference -on the decode instance*. The maximum benefit it can buy is bounded -above by the **decode duration itself** (you cannot recover more time -than the decode existed). For agentic that's 50 – 200 ms. The cost is -the table column above — 0.3 – 10 s of transfer per routed request. +On the phase-isolation axis alone, PD-disagg recovers two orders of +magnitude more decode time than it pays in transfer. **It is NOT this +axis that defeats static PD-disagg in agentic** — see colleague's +4P+4D experiment (TTFT p50 62×, success rate 99.5% → 52%) which is +driven by **D-side KV-pool overflow** on long-context requests +(`figs/f4b_pdsep_kv_wall.png`), not by transfer latency. -**Cost > Benefit by 5× to 100× across the agentic distribution.** Below -~3 GiB the ratio is small (≤1×); above 3 GiB the ratio explodes; above -6 GiB even individual draws can take 10 s for a single transfer. +What MB2 contributes to the paper is therefore: +- The **per-request transfer cost number** (used as the cost input + to the cost-benefit accounting above). +- The empirical observation that **Mooncake's transfer cost is + topology-independent** — intra-node and inter-node both go through + the RDMA NIC and hit the same 9.7 GB/s ceiling. PD-disagg's + transfer cost does not get cheaper by co-locating P and D. -This data alone is not the whole §3.2 argument — we still need to -account for D-side KV capacity (`f4b`, separate axis), cache reuse loss, -and static-partition mismatch (MB3 / MB4 / MB5). But it nails one -of the two key cost axes with measured numbers from vanilla mooncake, -not the dash0 patched build. +The dominant §3.2 failure mode of static PD-disagg in agentic is +**capacity**, not transfer cost. MB3 / MB4 / MB5 will quantify the +remaining axes (D-pool occupancy, cache reuse degradation under PD +routing, static-partition mismatch). ## Open questions / next runs diff --git a/figs/mb1_interference.png b/figs/mb1_interference.png index 8e660d6..1931735 100644 Binary files a/figs/mb1_interference.png and b/figs/mb1_interference.png differ diff --git a/figs/pd_cost_vs_benefit.png b/figs/pd_cost_vs_benefit.png deleted file mode 100644 index b1f181f..0000000 Binary files a/figs/pd_cost_vs_benefit.png and /dev/null differ diff --git a/microbench/fresh_setup/plot_mb1.py b/microbench/fresh_setup/plot_mb1.py index 488437e..6563a0b 100644 --- a/microbench/fresh_setup/plot_mb1.py +++ b/microbench/fresh_setup/plot_mb1.py @@ -1,20 +1,19 @@ #!/usr/bin/env python3 -"""Plot MB1 interference results + the §3.2 cost-vs-benefit headline figure. +"""Plot MB1 phase-interference data. -Two outputs: +Single output: figs/mb1_interference.png — effective per-stream TPOT +during a prefill burst, vs prefill size, one line per concurrent decode +batch size D. - mb1_interference.png - Effective TPOT during prefill vs prefill size, one line per D. - Log-log. Annotates typical agentic decode duration (~100 ms) as a - horizontal band so reader can spot when decode would be stalled. - - pd_cost_vs_benefit.png - The §3.2 headline. X axis: KV size (MiB). Two stacked curves: - - benefit ceiling (MB1) — at most one decode-duration per request - of phase isolation can be recovered. Drawn as a flat 100 ms line. - - cost (MB2) — Mooncake pure_transfer p50 at that size. - Anywhere the cost curve sits ABOVE the benefit ceiling, PD-disagg - structurally loses. +Earlier versions of this script also produced figs/pd_cost_vs_benefit.png +which composed a "max PD-disagg benefit = decode duration (50–200 ms) +band" against the MB2 transfer-cost curve. That accounting was wrong +(see RESULTS_SUMMARY.md §4 correction): phase-isolation benefit is +per-prefill-event, equal to D × T_prefill across stalled streams, not +capped by a single request's decode duration. That figure has been +removed; the math it implied was structurally backwards. The dominant +reason static PD-disagg fails in agentic is **D-side KV capacity** +(see figs/f4b_pdsep_kv_wall.png), not cost-vs-benefit on phase isolation. """ from __future__ import annotations @@ -25,21 +24,16 @@ from pathlib import Path import matplotlib matplotlib.use("Agg") import matplotlib.pyplot as plt -import numpy as np def main() -> None: p = argparse.ArgumentParser() p.add_argument("--mb1", type=Path, required=True) - p.add_argument("--mb2-intra", type=Path, required=True) - p.add_argument("--mb2-inter", type=Path, default=None) - p.add_argument("--out-interf", type=Path, default=Path("figs/mb1_interference.png")) - p.add_argument("--out-cb", type=Path, default=Path("figs/pd_cost_vs_benefit.png")) + p.add_argument("--out", type=Path, default=Path("figs/mb1_interference.png")) args = p.parse_args() mb1 = json.loads(args.mb1.read_text())["summary"] - # ---- mb1_interference.png ---- fig, ax = plt.subplots(figsize=(9, 5.5)) Ds = sorted({s["decode_batch_size"] for s in mb1}) colors = {1: "#1f77b4", 4: "#ff7f0e", 8: "#d62728"} @@ -50,79 +44,19 @@ def main() -> None: ys = [s["effective_tpot_during_ms"] for s in rows] ax.plot(xs, ys, "o-", lw=2, markersize=7, color=colors.get(D, "gray"), - label=f"D={D} (baseline {rows[0]['baseline_tpot_ms']:.1f} ms)") - - for tdec, lbl in [(50, "tool-call decode (~50 ms)"), - (100, "agentic decode (~100 ms)"), - (200, "long agentic decode (~200 ms)")]: - ax.axhline(tdec, color="#444", lw=0.6, ls=":", alpha=0.6) - ax.text(2200, tdec * 1.1, lbl, fontsize=8, color="#444") + label=f"D={D} (baseline TPOT {rows[0]['baseline_tpot_ms']:.1f} ms)") ax.set_xscale("log"); ax.set_yscale("log") ax.set_xlabel("Prefill burst size (tokens, log)") ax.set_ylabel("Per-stream effective TPOT during prefill burst (ms, log)") ax.set_title("MB1: each ongoing decode is essentially halted while prefill runs\n" - "(chunked-prefill ON, vLLM 0.18.1 default, single H20)") + "(chunked-prefill ON, vLLM 0.18.1 default, single H20). " + "Per-prefill aggregate stall = D × T_prefill.") ax.grid(True, which="both", alpha=0.3) ax.legend(loc="upper left", fontsize=9) - args.out_interf.parent.mkdir(parents=True, exist_ok=True) - fig.tight_layout(); fig.savefig(args.out_interf, dpi=150); plt.close(fig) - print(f"wrote {args.out_interf}") - - # ---- pd_cost_vs_benefit.png ---- - mb2_intra = json.loads(args.mb2_intra.read_text())["summary"] - mb2_intra = [s for s in mb2_intra if s["input_tokens"] >= 64] - intra_x_mib = [s["kv_mib"] for s in mb2_intra] - intra_y_ms = [s["pure_transfer_ms_p50"] for s in mb2_intra] - - fig, ax = plt.subplots(figsize=(9, 5.5)) - ax.plot(intra_x_mib, intra_y_ms, "o-", color="#d62728", lw=2.4, - markersize=8, label="MB2 PD-disagg KV transfer cost (Mooncake, p50)") - if args.mb2_inter: - mb2_inter = json.loads(args.mb2_inter.read_text())["summary"] - mb2_inter = [s for s in mb2_inter if s["input_tokens"] >= 64] - inter_x = [s["kv_mib"] for s in mb2_inter] - inter_y = [s["pure_transfer_ms_p50"] for s in mb2_inter] - ax.plot(inter_x, inter_y, "s--", color="#7a1d1d", lw=2, markersize=7, - alpha=0.7, label="MB2 inter-node (same numbers)") - - # Benefit ceiling: typical agentic decode duration (PD-disagg max savings). - ax.axhline(100, color="#2ca02c", lw=2.4, ls="-", - label="MB1 max benefit ≤ agentic decode (~100 ms)") - ax.axhspan(50, 200, alpha=0.15, color="#2ca02c", - label="benefit range (50–200 ms decode)") - - # Mark agentic-tail request sizes - for kv_mib, lbl in [(192, "trace mean\n(~2k tok)"), - (3072, "p90\n(~33k tok)"), - (6144, "p95\n(~65k tok)"), - (11500, "p99\n(11.5 GiB)")]: - ax.axvline(kv_mib, color="#666", lw=0.5, ls=":", alpha=0.5) - ax.text(kv_mib, 2, lbl, fontsize=8, color="#444", - ha="center", va="bottom") - - ax.set_xscale("log"); ax.set_yscale("log") - ax.set_xlim(40, 14000) - ax.set_ylim(1, 12000) - ax.set_xlabel("Per-request KV size (MiB, log)") - ax.set_ylabel("Time per request (ms, log)") - ax.set_title("§3.2 headline — PD-disagg KV transfer cost vs phase-isolation benefit\n" - "(both measured on vanilla vLLM 0.18.1 + Mooncake 0.3.11, agentic regime)") - ax.grid(True, which="both", alpha=0.3) - ax.legend(loc="upper left", fontsize=9) - - # Add explanatory annotation - ax.text(10000, 5000, - "Cost > benefit for ANY KV size above\n" - "the green band (~80 MiB / ~830 tokens).\n" - "Below: cost is marginal (<10 ms) but\n" - "benefit is also small (decode is short).", - fontsize=9, color="#333", - ha="right", va="top", - bbox=dict(boxstyle="round,pad=0.4", facecolor="#fffacd", alpha=0.9, edgecolor="#888")) - - fig.tight_layout(); fig.savefig(args.out_cb, dpi=150); plt.close(fig) - print(f"wrote {args.out_cb}") + args.out.parent.mkdir(parents=True, exist_ok=True) + fig.tight_layout(); fig.savefig(args.out, dpi=150); plt.close(fig) + print(f"wrote {args.out}") if __name__ == "__main__": diff --git a/scripts/b3_isolated_policy.sh b/scripts/b3_isolated_policy.sh index 08a888c..9634af0 100755 --- a/scripts/b3_isolated_policy.sh +++ b/scripts/b3_isolated_policy.sh @@ -32,10 +32,14 @@ RUNDIR="${3:?usage: $0 }" # Auto-enable kv_both when the policy requires it. # KV_CONNECTOR (Mooncake|Nixl) selects the underlying connector when KV_BOTH=1. +_KV_CONNECTOR_EXPLICIT="${KV_CONNECTOR:-}" KV_CONNECTOR="${KV_CONNECTOR:-Mooncake}" -if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_kv_both" ]; then +if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_v3" ] || [ "$POLICY" = "unified_kv_both" ]; then ENABLE_KV_BOTH=1 - KV_CONNECTOR="Mooncake" + # honor explicit KV_CONNECTOR override (e.g. =Nixl); otherwise default Mooncake. + if [ -z "$_KV_CONNECTOR_EXPLICIT" ]; then + KV_CONNECTOR="Mooncake" + fi fi if [ "$POLICY" = "unified_nixl_both" ]; then ENABLE_KV_BOTH=1 @@ -143,7 +147,10 @@ for i in $(seq 0 $((N_INSTANCES - 1))); do combined_args="$combined_args http://127.0.0.1:$((BASE_PORT + i))" done proxy_extra="" -if [ "$ENABLE_KV_BOTH" = "1" ]; then +# Bootstrap ports only needed for Mooncake handshake. Nixl uses its own +# UCX side-channel and the proxy forwards kv_transfer_params from src's +# response body instead of pre-baking engine_id/bootstrap_addr. +if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Mooncake" ]; then bp_list="" for i in $(seq 0 $((N_INSTANCES - 1))); do if [ -z "$bp_list" ]; then @@ -154,11 +161,15 @@ if [ "$ENABLE_KV_BOTH" = "1" ]; then done proxy_extra="--bootstrap-ports $bp_list" fi +if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Nixl" ]; then + proxy_extra="--connector-type nixl" +fi nohup "$VENV/python" "$ROOT/scripts/cache_aware_proxy.py" \ --port "$PROXY_PORT" \ --combined $combined_args \ --policy "$POLICY" \ $proxy_extra \ + ${EXTRA_PROXY_ARGS:-} \ > "$RUNDIR/proxy.log" 2>&1 & disown tries=0 diff --git a/scripts/cache_aware_proxy.py b/scripts/cache_aware_proxy.py index d70ea9e..ac7d5ae 100644 --- a/scripts/cache_aware_proxy.py +++ b/scripts/cache_aware_proxy.py @@ -82,6 +82,36 @@ class Settings: # Patch 6.6: per-request KV-xfer wall-clock timeout (proxy side). pd_sep_xfer_timeout_s: float = 60.0 + # --- unified_v3 (offload-decode) gating knobs ----------------------- + # v3 differs from v2 in *direction*: prefill stays on the session- + # affinity host (which holds the prefix cache); decode is migrated to + # a less-loaded target. KV transfer flows prefill_host → decode_target. + # The target doesn't need cache — we're shipping the post-prefill KV + # over anyway. After successful migration the session affinity table + # rotates to decode_target so the *next* turn lands where the KV now + # lives. + v3_min_new_tokens: int = 8000 # same as v2: don't migrate tiny prefills + v3_min_prefill_decode_busy: int = 1 # prefill_host must have ≥ this many concurrent decode tokens to justify migrating + v3_target_load_ratio: float = 0.7 # target.num_requests must be < prefill_host.num_requests × this + v3_min_load_gap: int = 1 # target.num_requests must also be ≤ prefill_host - this (absolute slack) + v3_rotate_affinity: bool = True # after migration, set session affinity to decode_target. + # Empirically False is better — see cache_miss_audit (next turn hits 9.5% + # with rotation vs ~80% without), because delay_free_blocks doesn't + # actually preserve cross-turn KV on decode_target. + v3_prefer_cache_target: bool = True # Mechanism B: among low-load candidates, prefer the one + # with the most prefix cache for this prompt — vLLM's connector + # auto-transfers only the missing portion (verified via + # smoke_partial_transfer: cache-rich dst is 77% faster than + # cold dst at 33k tokens, +512 ext). + + # --- KV connector selection (governs PD-sep handshake) ------------- + # "mooncake": pre-baked kv_transfer_params (bootstrap_addr+engine_id+transfer_id). + # Requires --bootstrap-ports and vLLMs launched with MooncakeConnector. + # "nixl" : response-forward handshake. src returns kv_transfer_params via + # response body, proxy forwards to dst. Nixl auto-selects transport + # via UCX (CUDA IPC / NVLink on intra-node, RDMA across nodes). + connector_type: str = "mooncake" + SETTINGS = Settings() @@ -538,6 +568,122 @@ def pick_instance_unified_v2( return chosen, chosen_idx, decision, None +def pick_instance_unified_v3( + instances: list[InstanceState], + token_ids: list[int] | None, + session_id: str | None, + input_length: int, + affinity: dict[str, int], +) -> tuple[InstanceState, int, dict, tuple[InstanceState, int] | None]: + """unified_v3 = unified hybrid + selective DECODE migration. + + Direction-reversed from unified_v2: + - prefill stays on session-affinity host (`prefill_host`) so we keep + the 93%-intra-session prefix-cache reuse intact. + - decode is migrated to a lower-load `decode_target` when the + affinity host is busy with concurrent decodes. + - KV transfer flows prefill_host → decode_target (the opposite of + v2's src → chosen). + - target does NOT need pre-existing cache — we're shipping the + post-prefill KV over anyway. + - On successful migration the *caller* must rotate + `affinity[session_id] = decode_target_idx` so the next turn lands + where the KV now lives (decode_target retains the blocks after + completion, since mooncake defaults to delay_free_blocks=True). + + Decision is purely load-based on the target side: + 1. new_local ≥ v3_min_new_tokens (don't pay RDMA for tiny prefills) + 2. prefill_host.ongoing_decode_tokens ≥ v3_min_prefill_decode_busy + (the host is actually busy decoding; migration buys decode-bw) + 3. ∃ target with both num_requests < prefill_host.num_requests × ratio + and num_requests ≤ prefill_host.num_requests − v3_min_load_gap + + Returns (prefill_host, prefill_idx, decision, migrate). When migrate + is None the request is fully local on prefill_host. When migrate is + (decode_target_inst, decode_target_idx), the handler should run + prefill on prefill_host and ship KV to decode_target for decode. + """ + prefill_host, prefill_idx, decision = pick_instance_unified_hybrid( + instances, token_ids, session_id, input_length, affinity) + + decision["v3_migrate"] = False + decision["v3_decision"] = "local" + decision["v3_reason"] = None + + if not token_ids: + decision["v3_reason"] = "no_token_ids" + return prefill_host, prefill_idx, decision, None + + prefill_cache_hit = prefill_host.estimate_cache_hit(token_ids) + new_local = max(0, input_length - prefill_cache_hit) + decision["v3_prefill_cache_hit"] = prefill_cache_hit + decision["v3_new_local"] = new_local + + # Gate 1: prefill must be large enough to amortise RDMA setup. + if new_local < SETTINGS.v3_min_new_tokens: + decision["v3_reason"] = ( + f"new_local_below_threshold ({new_local} < {SETTINGS.v3_min_new_tokens})" + ) + return prefill_host, prefill_idx, decision, None + + # Gate 2: affinity host must be busy with concurrent decodes — that's + # what migrating decode-traffic-away buys us. If the host is idle + # there's no point. + if prefill_host.ongoing_decode_tokens < SETTINGS.v3_min_prefill_decode_busy: + decision["v3_reason"] = ( + f"prefill_host_not_busy " + f"(ongoing_decode_tokens={prefill_host.ongoing_decode_tokens} < " + f"{SETTINGS.v3_min_prefill_decode_busy})" + ) + return prefill_host, prefill_idx, decision, None + + # Gate 3: pick the lowest-load target that is materially less loaded + # than the prefill_host. Cache content irrelevant — KV ships over. + threshold_loaded = max(1, + int(prefill_host.num_requests * SETTINGS.v3_target_load_ratio)) + candidates = [ + (i, inst) for i, inst in enumerate(instances) + if i != prefill_idx + and inst.num_requests < threshold_loaded + and inst.num_requests <= prefill_host.num_requests - SETTINGS.v3_min_load_gap + ] + if not candidates: + decision["v3_reason"] = ( + f"no_low_load_target " + f"(prefill_host.num_req={prefill_host.num_requests} " + f"threshold={threshold_loaded})" + ) + return prefill_host, prefill_idx, decision, None + + # Mechanism B (v3_prefer_cache_target=True): rank candidates first by + # cache_hit DESC (more cache = less KV to transfer), then by load. vLLM + # auto-skips transferring overlapping prefix when dst's local cache + # matches — verified in smoke_partial_transfer: 77% faster on a 33k + # prompt when dst has the prefix already. + if SETTINGS.v3_prefer_cache_target: + decode_target_idx, decode_target = min( + candidates, + key=lambda x: (-x[1].estimate_cache_hit(token_ids), + x[1].num_requests, x[1].ongoing_tokens)) + else: + decode_target_idx, decode_target = min( + candidates, key=lambda x: (x[1].num_requests, x[1].ongoing_tokens)) + + target_cache_hit = decode_target.estimate_cache_hit(token_ids) + decision["v3_migrate"] = True + decision["v3_decision"] = "migrate_decode" + decision["v3_target_idx"] = decode_target_idx + decision["v3_target_num_req"] = decode_target.num_requests + decision["v3_target_cache_hit"] = target_cache_hit + decision["v3_prefill_num_req"] = prefill_host.num_requests + decision["v3_reason"] = ( + f"prefill_host.num_req={prefill_host.num_requests} busy; " + f"target.num_req={decode_target.num_requests} cache_hit={target_cache_hit}, " + f"transferring KV after prefill" + ) + return prefill_host, prefill_idx, decision, (decode_target, decode_target_idx) + + def _extract_output_token_ids_from_sse( buffer: str, chunk: bytes, @@ -764,10 +910,11 @@ async def lifespan(app: FastAPI): policy = getattr(global_args, 'policy', 'linear') # Mooncake-based modes still need bootstrap discovery; NIXL uses # its own UCX side-channel and doesn't go through our proxy - # bootstrap path (and unified_nixl_both never PD-seps anyway). + # bootstrap path. With --connector-type=nixl, v3 also skips bootstrap. needs_bootstrap = ( global_args.offload - or policy in ("unified_v2", "unified_kv_both") + or (policy in ("unified_v2", "unified_v3", "unified_kv_both") + and getattr(global_args, 'connector_type', 'mooncake') == 'mooncake') ) if needs_bootstrap and bp_list: await init_prefill_bootstrap(combined_instances, app.state.ready) @@ -953,6 +1100,26 @@ async def _handle_combined(api, req_data, token_ids, input_length, session_id, h breakdown.update(decision) if session_id: session_affinity_combined[session_id] = best_idx + elif policy == "unified_v3": + # v3: prefill on affinity (cache reuse), decode migrated to a + # low-load target. KV flows prefill_host → decode_target. + # Reuses _handle_combined_pd_sep_v2 with src=prefill_host, + # dst=decode_target (the handler is direction-agnostic). + chosen, best_idx, decision, pd_sep_v2 = pick_instance_unified_v3( + combined_instances, token_ids, session_id, input_length, + session_affinity_combined) + breakdown.update(decision) + if session_id: + if pd_sep_v2 is not None and SETTINGS.v3_rotate_affinity: + # Migration + rotation: redirect next turn to decode_target, + # assuming KV will live there. (Empirically wrong — see + # cache_miss_audit. Keep behind a flag.) + _decode_target_inst, decode_target_idx = pd_sep_v2 + session_affinity_combined[session_id] = decode_target_idx + else: + # No rotation: keep affinity on prefill_host (where the prefix + # cache lives). This is the empirically correct choice. + session_affinity_combined[session_id] = best_idx else: # linear (default) chosen, best_idx = pick_instance( combined_instances, token_ids, session_id, input_length, @@ -983,13 +1150,35 @@ async def _handle_combined(api, req_data, token_ids, input_length, session_id, h }) if pd_sep_v2 is not None: - src_inst, src_idx = pd_sep_v2 - breakdown["v2_src_url"] = src_inst.url - breakdown["v2_src_idx"] = src_idx - return await _handle_combined_pd_sep_v2( - api, req_data, headers, token_ids, input_length, - src_inst, chosen, breakdown, - request_id=request_id) + # Handler contract: first arg = prefill source (does same-worker + # prefill with do_remote_decode=True, max_tokens=1), second arg = + # decode target (does do_remote_prefill=True, pulls KV via + # Mooncake, decodes). + # + # v2 contract: pd_sep_v2 = (src_inst, src_idx); chosen = decode + # → src does prefill (it has more cache), chosen decodes. + # v3 contract: chosen = prefill_host (affinity, has cache); + # pd_sep_v2 = (decode_target_inst, decode_target_idx) + # → chosen does prefill (cache reuse), decode_target decodes. + if policy == "unified_v3": + decode_target_inst, decode_target_idx = pd_sep_v2 + prefill_inst = chosen + breakdown["v2_src_url"] = prefill_inst.url + breakdown["v2_src_idx"] = best_idx + breakdown["v3_decode_target_url"] = decode_target_inst.url + breakdown["v3_decode_target_idx"] = decode_target_idx + return await _handle_combined_pd_sep_v2( + api, req_data, headers, token_ids, input_length, + prefill_inst, decode_target_inst, breakdown, + request_id=request_id) + else: + src_inst, src_idx = pd_sep_v2 + breakdown["v2_src_url"] = src_inst.url + breakdown["v2_src_idx"] = src_idx + return await _handle_combined_pd_sep_v2( + api, req_data, headers, token_ids, input_length, + src_inst, chosen, breakdown, + request_id=request_id) return await _handle_local_request( api, req_data, headers, token_ids, input_length, @@ -1011,11 +1200,12 @@ async def _handle_combined_pd_sep_v2( of SETTINGS.pd_sep_xfer_timeout_s, so a stuck KV transfer fails the request instead of hanging for 600 s. """ - if src.bootstrap_port is None: + connector = SETTINGS.connector_type + if connector == "mooncake" and src.bootstrap_port is None: raise HTTPException( status_code=500, detail=( - "unified_v2 PD-sep triggered but src instance " + "Mooncake PD-sep triggered but src instance " f"{src.url} has no bootstrap_port; launch with " "kv_role=kv_both and pass --bootstrap-ports" ), @@ -1029,12 +1219,16 @@ async def _handle_combined_pd_sep_v2( src_load_held = True dst_load_held = True + # Build prefill kv_transfer_params per connector. prefill_data = req_data.copy() - prefill_data["kv_transfer_params"] = { - "do_remote_decode": True, - "do_remote_prefill": False, - "transfer_id": f"xfer-{request_id}", - } + if connector == "mooncake": + prefill_data["kv_transfer_params"] = { + "do_remote_decode": True, + "do_remote_prefill": False, + "transfer_id": f"xfer-{request_id}", + } + else: # nixl: src just signals it'll produce KV for remote decode + prefill_data["kv_transfer_params"] = {"do_remote_decode": True} prefill_data["stream"] = False prefill_data["max_tokens"] = 1 prefill_data["min_tokens"] = 1 @@ -1044,11 +1238,23 @@ async def _handle_combined_pd_sep_v2( breakdown["t_prefill_sent"] = _time.monotonic() breakdown["t_prefill_sent_unix"] = _time.time() + forwarded_params: dict | None = None try: resp = await src.client.post(api, json=prefill_data, headers=p_headers) breakdown["t_prefill_done"] = _time.monotonic() breakdown["t_prefill_done_unix"] = _time.time() resp.raise_for_status() + if connector == "nixl": + # Nixl populates kv_transfer_params in the response body with + # remote_block_ids / remote_engine_id / remote_host / remote_port. + # We must read the body BEFORE aclose. + src_resp_json = resp.json() + forwarded_params = src_resp_json.get("kv_transfer_params") + if not forwarded_params or not forwarded_params.get("remote_block_ids"): + raise HTTPException( + status_code=502, + detail=f"Nixl src returned no remote_block_ids: {forwarded_params}", + ) await resp.aclose() src.record_prefix(token_ids) except Exception as e: @@ -1057,11 +1263,16 @@ async def _handle_combined_pd_sep_v2( breakdown["prefill_error"] = True breakdown["error_detail"] = repr(e)[:300] _breakdown_log.append(breakdown) - # Release reservations on failure. - src.ongoing_tokens -= input_length - src.num_requests -= 1 - dst.ongoing_tokens -= input_length - dst.num_requests -= 1 + # Release reservations on failure. Clear load_held flags so the + # finally block below does not double-decrement (CRITICAL audit #1). + if src_load_held: + src.ongoing_tokens -= input_length + src.num_requests -= 1 + src_load_held = False + if dst_load_held: + dst.ongoing_tokens -= input_length + dst.num_requests -= 1 + dst_load_held = False raise HTTPException(status_code=502, detail=f"Prefill failed: {e}") finally: if src_load_held: @@ -1069,17 +1280,19 @@ async def _handle_combined_pd_sep_v2( src.num_requests -= 1 src_load_held = False - parsed = urllib.parse.urlparse(str(src.client.base_url)) - bootstrap_addr = f"http://{parsed.hostname}:{src.bootstrap_port}" - decode_data = req_data.copy() - decode_data["kv_transfer_params"] = { - "do_remote_decode": False, - "do_remote_prefill": True, - "remote_bootstrap_addr": bootstrap_addr, - "remote_engine_id": src.engine_id.get(0, ""), - "transfer_id": f"xfer-{request_id}", - } + if connector == "mooncake": + parsed = urllib.parse.urlparse(str(src.client.base_url)) + bootstrap_addr = f"http://{parsed.hostname}:{src.bootstrap_port}" + decode_data["kv_transfer_params"] = { + "do_remote_decode": False, + "do_remote_prefill": True, + "remote_bootstrap_addr": bootstrap_addr, + "remote_engine_id": src.engine_id.get(0, ""), + "transfer_id": f"xfer-{request_id}", + } + else: # nixl: forward what src returned + decode_data["kv_transfer_params"] = forwarded_params breakdown["t_decode_sent"] = _time.monotonic() breakdown["t_decode_sent_unix"] = _time.time() @@ -1304,7 +1517,8 @@ def parse_args(): p.add_argument("--policy", type=str, default="linear", choices=["linear", "lmetric", "load_only", "sticky", "unified", "unified_kv_both", - "unified_nixl_both", "unified_v2"], + "unified_nixl_both", "unified_v2", + "unified_v3"], help="Routing policy: linear (cache-aware), lmetric (P_tokens × BS), " "load_only (B3 control: pure min-num_requests), " "sticky (B3 control: hard session affinity), " @@ -1317,6 +1531,20 @@ def parse_args(): "or unified_v2 (unified + selective per-request PD-sep " "via Mooncake; requires --bootstrap-ports and " "kv_role=kv_both vLLM launch)") + p.add_argument("--v3-rotate-affinity", type=int, default=1, choices=[0,1], + help="unified_v3 only: 1 = rotate session affinity to decode_target " + "after migration (original behavior, empirically loses prefix cache); " + "0 = keep affinity on prefill_host so next turn hits its cache.") + p.add_argument("--connector-type", type=str, default="mooncake", + choices=["mooncake", "nixl"], + help="PD-sep handshake protocol. 'mooncake' uses pre-baked engine_id" + " + bootstrap_addr (requires --bootstrap-ports). 'nixl' uses" + " response-forward (src returns kv_transfer_params, proxy" + " relays to dst; Nixl/UCX auto-picks NVLink intra-node).") + p.add_argument("--v3-prefer-cache-target", type=int, default=1, choices=[0,1], + help="Mechanism B: unified_v3 picks decode_target with the most" + " prefix cache among low-load candidates (default 1). Set 0" + " to fall back to pure-load tie-break (cache-blind).") p.add_argument("--overload-factor", type=float, default=2.0, help="Break session affinity when instance load > factor * avg") # The four flags below are accepted for bench.sh backward compatibility but @@ -1354,7 +1582,14 @@ if __name__ == "__main__": SETTINGS.max_offload_inflight = global_args.max_offload_inflight SETTINGS.cache_gate_ratio = global_args.cache_gate_ratio SETTINGS.decode_iteration_s = getattr(global_args, 'decode_iteration_s', 0.05) - print("SETTINGS: throughput=%.0f rdma_overhead=%.2f offload=%s" % ( + SETTINGS.v3_rotate_affinity = bool(getattr(global_args, 'v3_rotate_affinity', 1)) + SETTINGS.connector_type = getattr(global_args, 'connector_type', 'mooncake') + SETTINGS.v3_prefer_cache_target = bool(getattr(global_args, 'v3_prefer_cache_target', 1)) + print("SETTINGS: throughput=%.0f rdma_overhead=%.2f offload=%s v3_rotate_affinity=%s " + "connector_type=%s v3_prefer_cache_target=%s" % ( SETTINGS.prefill_throughput, SETTINGS.rdma_overhead_s, - getattr(global_args, 'offload', False))) + getattr(global_args, 'offload', False), + SETTINGS.v3_rotate_affinity, + SETTINGS.connector_type, + SETTINGS.v3_prefer_cache_target)) uvicorn.run(app, host=global_args.host, port=global_args.port)