Correct PD-disagg cost/benefit framing across repo

The §3.2 cost-vs-benefit math in commits 029821c (MB1 plot +
pd_cost_vs_benefit.png) and abde010 (RESULTS_SUMMARY.md) was wrong.

What was wrong:
  I framed PD-disagg's max phase-isolation benefit as "≤ decode duration
  of the new request (~50–200 ms)" — implicitly treating the benefit as
  per-request and bounded by that request's own decode. The correct
  accounting is per-prefill-event across all stalled streams:

      benefit_per_prefill = D × T_prefill × (1 − TPOT_baseline/TPOT_during)
                          ≈ D × T_prefill

  which the user pointed out follows from the chunked-prefill math
  (each of L/N chunks slows D ongoing decode steps from ~10 ms to t ms,
  summing to D × T_prefill).

Plug MB1 + MB2 numbers in:

  prefill size | T_prefill | T_transfer | D=8 benefit | cost/benefit
   2k tok      | 0.14 s    |     8 ms   |   1.1 s     |    0.7 %
  33k tok      | 4.5  s    |  320 ms    |  36   s     |    0.9 %
 125k tok      | 57   s    |  1.9 s     | 456   s     |    0.4 %

On the phase-isolation axis alone, PD-disagg WINS by 100×–250× — the
opposite of what the deleted figure showed.

The actual dominant reason static PD-disagg fails in agentic is the
D-side KV pool capacity wall (figs/f4b_pdsep_kv_wall.png) — p99
single-request KV is 11.5 GiB, per-D-instance pool is 38 GiB, so 4P+4D
halves system decode capacity. Colleague's 4P+4D experiment showed
TTFT p50 62× worse and success rate 99.5% → 52%, driven by pool
overflow + queueing, not by transfer latency.

Changes:
- figs/pd_cost_vs_benefit.png : DELETED (figure built on wrong math)
- microbench/fresh_setup/plot_mb1.py : drop the pd_cost_vs_benefit
  function; keep mb1_interference.png and update its title to note
  per-prefill aggregate stall = D × T_prefill (not capped by decode)
- figs/mb1_interference.png : regenerated, no misleading band annotation
- analysis/mb1/README.md : Summary block rewritten ("what MB1 measures";
  no more "max benefit = decode duration" claim); §3.2 implications
  section replaced with the corrected per-prefill-event table; explicit
  ⚠ Correction note documents what was wrong
- analysis/mb2/README.md : Summary block + §3.2 implications section
  rewritten the same way; ⚠ Correction note links to RESULTS_SUMMARY §4
- RESULTS_SUMMARY.md §4 + §6 : §4 reordered to lead with the D-side
  capacity argument (the real failure mode), MB1/MB2 demoted from
  "kill-shot for PD-disagg" to "supporting context inputs to a
  cost-benefit table that actually favors PD-disagg on this axis";
  §6 paper-claims list reordered to remove the wrong "PD-disagg loses
  on cost-vs-benefit" claim and replace with the corrected ones

PAPER_OUTLINE.md and MEETING.md were checked and never picked up this
specific wrong claim — they already (correctly) frame §3.2 around the
D-side KV memory wall.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

RESULTS_SUMMARY.md

@@ -39,52 +39,63 @@ Production trace = Qwen3-Coder agentic，1.3 M sessions / 2.1 M reqs / 7200 s。
 
 参考图：`figs/f4a_apc_loss.png`、`figs/f4b_pdsep_kv_wall.png`、`figs/f4c_per_worker_ttft.png`、`figs/f6_e2e_latency_bars.png`、`figs/f6_e2e_latency_full_grid.png`。
 
-## 4. PD-disagg 在 agentic 下输——cost vs benefit（§3.2）
+## 4. Static PD-disagg 为什么失败（§3.2）—— 容量问题，不是 cost-benefit 问题
 
-由两个独立 microbench 钉死（**全用 vanilla vLLM 0.18.1 + Mooncake 0.3.11，fresh venv，无 patch**）。
+⚠ **2026-05-27 纠正**：本节前一版本论证"PD-disagg 因为 transfer cost > phase isolation benefit 而失败"。**这个论证算错了**。正确的 phase-isolation benefit 是**每个 prefill 事件 × D 个 concurrent stream** 的总和（≈ `D × T_prefill`），不是单个 request 的 decode 时长。用正确公式，PD-disagg 在 phase-isolation 这一维上**赢 colo 一两个数量级**。Static PD-disagg 在 agentic 上失败的**真正根因是 D 侧 KV pool 容量**，不是这一维。
 
-### 4.1 MB2 — KV transfer cost
+### 4.1 真正的失败模式：D 侧 KV 容量天花板
 
-dash1 GPU 0+1（intra-node）和 dash1 ↔ dash2（inter-node, 200 Gbps RoCE）扫 9 个 size × 5 reps。
+| | 8C colo | 4P+4D PD-disagg |
+|---|---:|---:|
+| Per-D-instance KV pool（0.4 × 96 GiB） | 38 GiB | 38 GiB |
+| 系统总 decode 容量（D 实例数 × 单池） | 8 × 38 = **304 GiB** | 4 × 38 = **152 GiB** |
+| p99 单请求 KV = 11.5 GiB → 最多并发 decode | 24 | **12（减半）** |
 
-| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求（11.5 GiB）transfer 时间 |
+Colleague 4P+4D 实测：TTFT p50 0.91 s → **62.8 s（62×）**、success rate **99.5% → 52%**。失败模式：**D 池溢出 + 排队**，不是 transfer 延迟。
+
+参考图：`figs/f4b_pdsep_kv_wall.png`（pdf 版本是高质量 paper figure）。
+
+### 4.2 MB2 — KV transfer cost（per-request 一次性成本，**不**是 dominant cost）
+
+dash1 GPU 0+1（intra）和 dash1 ↔ dash2（inter, 200 Gbps RoCE）扫 9 个 size × 5 reps。
+
+| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求 11.5 GiB transfer |
 |---|---|---|
-| Intra-node | **9.7 GB/s** | p50 **1.9 s** · min 1.5 s · max 10 s |
+| Intra-node | **9.7 GB/s** | p50 **1.9 s** · max 10 s |
-| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · min 1.3 s · max 9.2 s |
+| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · max 9.2 s |
 
-**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC，包括 intra-node loopback**，不走 NVLink。200 Gbps NIC 是天花板，**PD-disagg 的 transfer cost 与拓扑无关**。
+**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC**，不走 NVLink。200 Gbps NIC 是天花板。**PD-disagg transfer cost 与拓扑无关**。
 
-参考图：`figs/mb2_transfer_time_compare.png`、`figs/mb2_transfer_bw_compare.png`、doc `analysis/mb2/README.md`。
+参考图：`figs/mb2_transfer_time_compare.png`、doc `analysis/mb2/README.md`。
 
-### 4.2 MB1 — Phase interference（chunked-prefill on, 默认 baseline）
+### 4.3 MB1 — Phase interference（PD-disagg 的潜在 benefit 上界）
 
-dash1 GPU 0 单 instance，D（concurrent decodes）× P（prefill size）扫描。
+dash1 GPU 0 单 instance（无 kv_connector），chunked-prefill 默认开启，D × P 扫描。D=8 结果：
 
-D=8（最 agentic-realistic）的结果：
+| Prefill | T_prefill | per-stream TPOT during | penalty |
-
-| Prefill | prefill_ttft | per-stream TPOT during | penalty |
 |---|---:|---:|---:|
 | 2k tok | 143 ms | 32 ms | 4× |
-| 8k | 583 ms | 114 ms | 15× |
+| 32k tok | 4.5 s | 388 ms | **52×** |
-| 32k | 4.5 s | 388 ms | **52×** |
+| 131k tok | 57 s | 1419 ms | **183×** |
-| 65k | 15.6 s | 757 ms | **99×** |
-| 131k | 57 s | 1419 ms | **183×** |
 
-baseline TPOT 7.7 ms。**Decode 在大 prefill 期间基本被 halted**。chunked-prefill 已经默认开启，PD-disagg 在它之上能额外提供的 phase isolation = **decode 在 prefill 期间被 halted 的那部分时间**。
+**Decode 在 prefill 期间被几乎完全 halted**，单 stream 损失 ≈ `T_prefill` 的时间。**每个 prefill event 总 decode 损失 ≈ `D × T_prefill`**。
 
 参考图：`figs/mb1_interference.png`、doc `analysis/mb1/README.md`。
 
-### 4.3 联合结论
+### 4.4 联合 cost-benefit（per-prefill event）
 
-| | Per-request |
+| Prefill (KV size) | T_prefill | Cost = T_transfer | Benefit = D × T_prefill (D=8) | Cost / Benefit |
-|---|---|
+|---:|---:|---:|---:|---:|
-| **Max PD-disagg benefit**（救回来的 decode 时间）| ≤ **decode 时长 = 50–200 ms**（agentic tool-call output）|
+| 2k tok (192 MiB) | 0.14 s | 8 ms | 1.1 s | **0.7%** |
-| **PD-disagg cost**（MB2 transfer p50）| 80 MiB ≈ 8 ms · 3 GiB ≈ 320 ms · 11.5 GiB ≈ **1.9 s**（p99 实测最差 10 s）|
+| 33k tok (3 GiB, trace mean) | 4.5 s | 0.32 s | 36 s | **0.9%** |
-| Cost / Benefit | **每个 KV ≥ 80 MiB 的请求都输**；trace 平均 KV 192 MiB → 已经输 |
+| 125k tok (12 GiB, ~p99) | 57 s | 1.9 s | 456 s | **0.4%** |
 
-**结论**：在 agentic 上 **PD-disaggregation 是结构性失败的**。Chunked-prefill 默认已经在 colocation 内做了 first-order phase isolation；PD-disagg 在此之上能额外补的（decode 短时段没被 prefill 挤）小于它新带来的（每个 routed 请求都付 KV transfer）。这个结论与拓扑无关（intra-node 和 inter-node 一样）。
+**PD-disagg 在 phase-isolation 这一维赢 100×–250×**。但**这不是 §3.2 该用的论证**，因为 §3.2 真正的 dominant failure 是 §4.1 的 D 池容量天花板（颠覆了上面的全部数学）。
 
-参考图：`figs/pd_cost_vs_benefit.png`（§3.2 headline）。
+**总结**：
+- D 侧 KV 容量天花板（§4.1）→ PD-disagg 在 agentic 上**结构性失败**。
+- MB1 + MB2 的合计 cost-benefit 在 phase isolation 维度上 PD-disagg 是赢的，**但这件事被容量天花板压倒**。
+- Paper §3.2 论证应该聚焦"D 池装不下"，MB1/MB2 数据用作 supporting context（per-request transfer charge 量化、phase isolation benefit 量化）而不是 main argument。
 
 ## 5. EAR 设计的实证状态（§4）
 
@@ -96,10 +107,12 @@ baseline TPOT 7.7 ms。**Decode 在大 prefill 期间基本被 halted**。chunke
 ## 6. 已经能写的 paper 主张（按 confidence 排序）
 
 1. **Agentic vs chatbot 在调度上是不同 regime**（dispatch coupling + sub-second tool-call mass）—— 实证完整
-2. **PD-disaggregation 在 agentic 下输**（cost > benefit，跨拓扑）—— **MB1 + MB2 实证完整**
+2. **三类现有调度 baseline 各自的失败模式**（load-balance / static PD-disagg / pure sticky）—— 实证完整
-3. **三类现有调度 baseline 各自的失败模式** —— 实证完整
+3. **Static PD-disagg 在 agentic 下失败的 dominant 根因是 D 侧 KV 容量**（不是 phase-isolation cost-benefit）—— 实证完整（`f4b` + colleague 4P+4D 数据）
-4. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整
+4. **Mooncake transfer cost 拓扑无关**（intra ≈ inter，~9.7 GB/s NIC 上限）—— 实证完整（MB2）
-5. **Hot-triggered migration 修复 sticky 的 hot pin** —— **design 完整、e2e 待验证**
+5. **Phase isolation interference 在 chunked-prefill on 下仍然显著**（per-stream TPOT during prefill 15×–2000× baseline）—— 实证完整（MB1）。**注意**：这条数据本身不直接论证 "PD-disagg 失败"，因为算正确账后 PD-disagg 反而在这一维上赢；它的用途是给 §3.2 提供 phase-isolation benefit 上界的量化。
+6. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整
+7. **Hot-triggered migration 修复 sticky 的 hot pin** —— design 完整、e2e 待验证
 
 ## 7. 待做
 

analysis/mb1/README.md

@@ -15,20 +15,28 @@ bottom; the **Summary** block is what gets cited.
 | Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** |
 | Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** |
 | Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** |
-| Maximum PD-disagg benefit per agentic decode | **≤ 50–200 ms** (= decode duration) |
 
-**§3.2 headline (cost vs benefit, this run + MB2)**:
+**What MB1 actually measures**:
 
-> Under chunked-prefill, every ongoing decode stream is essentially
+> During a prefill burst, every ongoing decode stream is essentially
-> **halted while a prefill chunk is in flight** — per-stream effective
+> halted (per-stream effective TPOT is 15×–2000× baseline, scaling with
-> TPOT during the burst is 15× to 2000× baseline, scaling with prefill
+> prefill size). The **total decode time lost per prefill event is
-> size. PD-disagg can recover this stall, but the recovery is bounded by
+> `D × T_prefill`** (D concurrent decodes each lose ~T_prefill of useful
-> the **decode duration** of the request being protected. For agentic,
+> work). For the trace mean (P ≈ 33k tokens, T_prefill ≈ 4.5 s) at D=8
-> decode is 50–200 ms (tool-call output). MB2 shows PD-disagg pays
+> that's **~36 seconds of decode-equivalent work lost per request**.
-> 300 ms – 10 s of KV-transfer cost per request to do that recovery. The
+> This is the **upper bound on what PD-disaggregation's phase isolation
-> cost exceeds the benefit ceiling for any per-request KV ≥ ~80 MiB
+> could recover** on the decode side.
-> (~830 tokens) — well below all agentic operating points. The benefit
+
-> never beats the cost in this workload.
+**⚠ Correction (2026-05-27)**: an earlier version of this README framed
+the §3.2 PD-disagg argument as "phase-isolation benefit is capped at
+the decode duration of the new request (~50–200 ms), so MB2 transfer
+cost dominates". That framing was wrong. The correct accounting is
+benefit-per-prefill-event = D × T_prefill (aggregate decode time saved
+across all stalled streams), which is **much larger than per-request
+transfer cost**. The actual reason static PD-disagg fails in agentic
+is **D-side KV pool capacity** (`figs/f4b_pdsep_kv_wall.png`), not a
+cost-vs-benefit imbalance on phase isolation. See `RESULTS_SUMMARY.md`
+section 4 for the corrected framing.
 
 ## Setup
 
@@ -107,25 +115,30 @@ the cleanest "average over the whole burst window" number).
   halts decode). This is the entire prefill duration's worth of decode
   time that could in principle be recovered.
 
-Two big caveats for **agentic** application:
+**Connecting to the §3.2 PD-disagg argument** (corrected):
 
-1. **Decode is short** (~50–200 ms for tool-call output). The actual
+PD-disagg's promised phase-isolation benefit is **per prefill event**,
-   recoverable benefit per request is bounded by the decode duration,
+not per request. When a new prefill arrives, it stalls every concurrent
-   not by `prefill_ttft`. If a decode lasts 100 ms and a 5-second prefill
+decode stream on the same GPU. The aggregate decode time lost across
-   collides with it, PD-disagg can save at most 100 ms — not 5 s.
+those D streams is `D × T_prefill`. PD-disagg moving prefill off-decode-GPU
-2. **PD-disagg pays KV-transfer cost** (MB2: 300 ms – 10 s per request
+recovers all of it.
-   for agentic sizes). For any KV ≥ ~80 MiB the cost already exceeds the
-   ~100 ms benefit ceiling. Cost > benefit across the whole agentic
-   distribution.
 
-## §3.2 cost-vs-benefit figure
+Plugging numbers per prefill event:
 
-`figs/pd_cost_vs_benefit.png` overlays MB1 benefit ceiling (50–200 ms
+| Prefill size | T_prefill | PD-disagg cost (MB2 T_transfer) | PD-disagg benefit (D=8 × T_prefill) | Ratio |
-band, capped by decode duration) on top of MB2 transfer cost curve. The
+|---:|---:|---:|---:|---:|
-cost curve crosses the benefit ceiling somewhere around **80 MiB / 830
+| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s | 0.7 % |
-tokens** of KV — well below the trace mean (192 MiB / 2k tok ≈ trace
+| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s | 0.9 % |
-mean per request KV, and we know agentic averages 33k tokens, p99
+| 125k tok (~p99) | 57 s | 1.9 s | 456 s | 0.4 % |
-125k). For anything bigger PD-disagg pays more than it can recover.
+
+On the **phase-isolation axis alone**, PD-disagg wins by 100×–250×.
+The reason static PD-disagg nonetheless **fails in agentic** is a
+*different* failure mode: the D-side KV pool cannot fit p90+ requests
+(p99 = 11.5 GiB; D-instance pool ≈ 38 GiB; 4P+4D halves system-wide
+decode capacity → TTFT p50 62×, success rate 99.5% → 52% in colleague's
+4P+4D experiment). The structural problem is **capacity** (see
+`figs/f4b_pdsep_kv_wall.png`), not transfer-cost vs phase-isolation
+trade-off.
 
 ## Reproduction
 
@@ -174,4 +187,7 @@ ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop
 
 3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on
 dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`.
-Figures: `figs/mb1_interference.png`, `figs/pd_cost_vs_benefit.png`.
+Figure: `figs/mb1_interference.png`. The figure
+`figs/pd_cost_vs_benefit.png` from the original commit `029821c` was
+based on the wrong "benefit ≤ decode duration" accounting; **deleted in
+the correction commit**.

analysis/mb2/README.md

@@ -24,12 +24,25 @@ get cheaper by co-locating P and D on the same node — the ~9.7 GB/s
 ceiling applies regardless. Halving the transfer cost cannot be bought
 back by topology.
 
-**Headline for the paper §3.2**: at the agentic tail, **pure KV transfer
+**What MB2 actually measures**: the **per-request charge** that
-takes 1.5 – 10 s**. A median agentic decode is **50 – 200 ms** of tool-call
+PD-disagg pays for every routed request — `T_transfer ≈ KV_size / 9.7
-output. So **PD-disaggregation adds 8 – 100 × decode-time of transfer on
+GB/s`. For agentic this is **8 ms (192 MiB / trace lower) – 1.9 s
-top of every routed request**. Phase isolation (the thing PD-disagg
+(11.5 GiB / p99)**.
-trades transfer cost for) can only win back at most one decode duration
+
-— for agentic that's negligible. The arithmetic is one-sided.
+**⚠ Correction (2026-05-27)**: an earlier version of this README
+framed §3.2 as "transfer cost (1.5–10 s) >> decode duration (50–200 ms),
+so PD-disagg loses on cost-vs-benefit." That accounting was wrong:
+PD-disagg's phase-isolation benefit is **per-prefill-event** and equals
+`D × T_prefill` (aggregate across stalled decode streams), not the
+single-request decode duration. With trace-mean `T_prefill = 4.5 s` and
+D = 8, the benefit is ~36 s — far larger than the ~0.32 s transfer
+cost. PD-disagg's phase-isolation axis is a *win*, not a loss.
+
+The actual reason static PD-disagg fails in agentic is **D-side KV
+capacity** (`figs/f4b_pdsep_kv_wall.png`), not a cost-vs-benefit
+imbalance. See `RESULTS_SUMMARY.md` section 4 for the corrected
+framing. MB2 still serves as the source of the per-request transfer
+cost number used in that analysis.
 
 ---
 
@@ -137,43 +150,44 @@ analysis; not done yet.
   treats them as additional samples (same sizes); the per-size
   aggregates use all of them.
 
-## Implications for §3.2 PD-disagg cost argument
+## Implications for §3.2 PD-disagg argument
 
 For each PD-disagg-routed request, transfer wall-time is:
 
 ```
-T_transfer(KV_size) = max(  pure_transfer(KV_size),  rx_overhead  )
+T_transfer(KV_size) ≈ KV_size / 9.7 GB/s   for KV_size ≤ 3 GiB
-                    ≈ KV_size / 9.7 GB/s   for KV_size <= 3 GiB
                     ≈ 0.3 – 10 s            for KV_size in [3, 12] GiB
 ```
 
-Agentic decode wall-time is typically 50 – 200 ms (tool-call output of
+This is the **per-request transfer charge** of PD-disagg. It's a
-a few tens of tokens at ~50 tok/s). So the **transfer/decode ratio**
+real cost, but in the context of phase-isolation accounting it is
-under intra-node best-case Mooncake is:
+*small* compared to the benefit:
 
-| KV size | T_transfer @9.7 GB/s | typical decode | T_transfer / T_decode |
+| Prefill | T_prefill (MB1) | T_transfer (MB2) | Phase-isolation benefit at D=8 = D × T_prefill |
-|---|---:|---:|---:|
+|---:|---:|---:|---:|
-| 192 MiB (2k tok)   |   20 ms | 100 ms | 0.2× |
+| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s |
-| 768 MiB (8k tok)   |   84 ms | 100 ms | 0.8× |
+| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s |
-| 3 GiB (33k tok ≈ trace mean) |  321 ms | 100 ms | **3.2×** |
+| 125k tok (~p99) | 57 s | 1.9 s | 456 s |
-| 6 GiB (~p90)       | 1900 ms | 100 ms | **19×**  |
-| 12 GiB (~p99)      | 2800 ms | 100 ms | **28×** (median) – **100×** (p99 variance) |
 
-PD-disagg's promised payoff is *eliminating prefill–decode interference
+On the phase-isolation axis alone, PD-disagg recovers two orders of
-on the decode instance*. The maximum benefit it can buy is bounded
+magnitude more decode time than it pays in transfer. **It is NOT this
-above by the **decode duration itself** (you cannot recover more time
+axis that defeats static PD-disagg in agentic** — see colleague's
-than the decode existed). For agentic that's 50 – 200 ms. The cost is
+4P+4D experiment (TTFT p50 62×, success rate 99.5% → 52%) which is
-the table column above — 0.3 – 10 s of transfer per routed request.
+driven by **D-side KV-pool overflow** on long-context requests
+(`figs/f4b_pdsep_kv_wall.png`), not by transfer latency.
 
-**Cost > Benefit by 5× to 100× across the agentic distribution.** Below
+What MB2 contributes to the paper is therefore:
-~3 GiB the ratio is small (≤1×); above 3 GiB the ratio explodes; above
+- The **per-request transfer cost number** (used as the cost input
-6 GiB even individual draws can take 10 s for a single transfer.
+  to the cost-benefit accounting above).
+- The empirical observation that **Mooncake's transfer cost is
+  topology-independent** — intra-node and inter-node both go through
+  the RDMA NIC and hit the same 9.7 GB/s ceiling. PD-disagg's
+  transfer cost does not get cheaper by co-locating P and D.
 
-This data alone is not the whole §3.2 argument — we still need to
+The dominant §3.2 failure mode of static PD-disagg in agentic is
-account for D-side KV capacity (`f4b`, separate axis), cache reuse loss,
+**capacity**, not transfer cost. MB3 / MB4 / MB5 will quantify the
-and static-partition mismatch (MB3 / MB4 / MB5). But it nails one
+remaining axes (D-pool occupancy, cache reuse degradation under PD
-of the two key cost axes with measured numbers from vanilla mooncake,
+routing, static-partition mismatch).
-not the dash0 patched build.
 
 ## Open questions / next runs
 

microbench/fresh_setup/plot_mb1.py

@@ -1,20 +1,19 @@
 #!/usr/bin/env python3
-"""Plot MB1 interference results + the §3.2 cost-vs-benefit headline figure.
+"""Plot MB1 phase-interference data.
 
-Two outputs:
+Single output: figs/mb1_interference.png — effective per-stream TPOT
+during a prefill burst, vs prefill size, one line per concurrent decode
+batch size D.
 
-  mb1_interference.png
+Earlier versions of this script also produced figs/pd_cost_vs_benefit.png
-    Effective TPOT during prefill vs prefill size, one line per D.
+which composed a "max PD-disagg benefit = decode duration (50–200 ms)
-    Log-log. Annotates typical agentic decode duration (~100 ms) as a
+band" against the MB2 transfer-cost curve. That accounting was wrong
-    horizontal band so reader can spot when decode would be stalled.
+(see RESULTS_SUMMARY.md §4 correction): phase-isolation benefit is
-
+per-prefill-event, equal to D × T_prefill across stalled streams, not
-  pd_cost_vs_benefit.png
+capped by a single request's decode duration. That figure has been
-    The §3.2 headline. X axis: KV size (MiB). Two stacked curves:
+removed; the math it implied was structurally backwards. The dominant
-      - benefit ceiling (MB1) — at most one decode-duration per request
+reason static PD-disagg fails in agentic is **D-side KV capacity**
-        of phase isolation can be recovered. Drawn as a flat 100 ms line.
+(see figs/f4b_pdsep_kv_wall.png), not cost-vs-benefit on phase isolation.
-      - cost (MB2) — Mooncake pure_transfer p50 at that size.
-    Anywhere the cost curve sits ABOVE the benefit ceiling, PD-disagg
-    structurally loses.
 """
 from __future__ import annotations
 
@@ -25,21 +24,16 @@ from pathlib import Path
 import matplotlib
 matplotlib.use("Agg")
 import matplotlib.pyplot as plt
-import numpy as np
 
 
 def main() -> None:
     p = argparse.ArgumentParser()
     p.add_argument("--mb1", type=Path, required=True)
-    p.add_argument("--mb2-intra", type=Path, required=True)
+    p.add_argument("--out", type=Path, default=Path("figs/mb1_interference.png"))
-    p.add_argument("--mb2-inter", type=Path, default=None)
-    p.add_argument("--out-interf", type=Path, default=Path("figs/mb1_interference.png"))
-    p.add_argument("--out-cb", type=Path, default=Path("figs/pd_cost_vs_benefit.png"))
     args = p.parse_args()
 
     mb1 = json.loads(args.mb1.read_text())["summary"]
 
-    # ---- mb1_interference.png ----
     fig, ax = plt.subplots(figsize=(9, 5.5))
     Ds = sorted({s["decode_batch_size"] for s in mb1})
     colors = {1: "#1f77b4", 4: "#ff7f0e", 8: "#d62728"}
@@ -50,79 +44,19 @@ def main() -> None:
         ys = [s["effective_tpot_during_ms"] for s in rows]
         ax.plot(xs, ys, "o-", lw=2, markersize=7,
                 color=colors.get(D, "gray"),
-                label=f"D={D} (baseline {rows[0]['baseline_tpot_ms']:.1f} ms)")
+                label=f"D={D} (baseline TPOT {rows[0]['baseline_tpot_ms']:.1f} ms)")
-
-    for tdec, lbl in [(50, "tool-call decode (~50 ms)"),
-                       (100, "agentic decode (~100 ms)"),
-                       (200, "long agentic decode (~200 ms)")]:
-        ax.axhline(tdec, color="#444", lw=0.6, ls=":", alpha=0.6)
-        ax.text(2200, tdec * 1.1, lbl, fontsize=8, color="#444")
 
     ax.set_xscale("log"); ax.set_yscale("log")
     ax.set_xlabel("Prefill burst size (tokens, log)")
     ax.set_ylabel("Per-stream effective TPOT during prefill burst (ms, log)")
     ax.set_title("MB1: each ongoing decode is essentially halted while prefill runs\n"
-                 "(chunked-prefill ON, vLLM 0.18.1 default, single H20)")
+                 "(chunked-prefill ON, vLLM 0.18.1 default, single H20). "
+                 "Per-prefill aggregate stall = D × T_prefill.")
     ax.grid(True, which="both", alpha=0.3)
     ax.legend(loc="upper left", fontsize=9)
-    args.out_interf.parent.mkdir(parents=True, exist_ok=True)
+    args.out.parent.mkdir(parents=True, exist_ok=True)
-    fig.tight_layout(); fig.savefig(args.out_interf, dpi=150); plt.close(fig)
+    fig.tight_layout(); fig.savefig(args.out, dpi=150); plt.close(fig)
-    print(f"wrote {args.out_interf}")
+    print(f"wrote {args.out}")
-
-    # ---- pd_cost_vs_benefit.png ----
-    mb2_intra = json.loads(args.mb2_intra.read_text())["summary"]
-    mb2_intra = [s for s in mb2_intra if s["input_tokens"] >= 64]
-    intra_x_mib = [s["kv_mib"] for s in mb2_intra]
-    intra_y_ms  = [s["pure_transfer_ms_p50"] for s in mb2_intra]
-
-    fig, ax = plt.subplots(figsize=(9, 5.5))
-    ax.plot(intra_x_mib, intra_y_ms, "o-", color="#d62728", lw=2.4,
-            markersize=8, label="MB2 PD-disagg KV transfer cost (Mooncake, p50)")
-    if args.mb2_inter:
-        mb2_inter = json.loads(args.mb2_inter.read_text())["summary"]
-        mb2_inter = [s for s in mb2_inter if s["input_tokens"] >= 64]
-        inter_x = [s["kv_mib"] for s in mb2_inter]
-        inter_y = [s["pure_transfer_ms_p50"] for s in mb2_inter]
-        ax.plot(inter_x, inter_y, "s--", color="#7a1d1d", lw=2, markersize=7,
-                alpha=0.7, label="MB2 inter-node (same numbers)")
-
-    # Benefit ceiling: typical agentic decode duration (PD-disagg max savings).
-    ax.axhline(100, color="#2ca02c", lw=2.4, ls="-",
-               label="MB1 max benefit ≤ agentic decode (~100 ms)")
-    ax.axhspan(50, 200, alpha=0.15, color="#2ca02c",
-               label="benefit range (50–200 ms decode)")
-
-    # Mark agentic-tail request sizes
-    for kv_mib, lbl in [(192, "trace mean\n(~2k tok)"),
-                         (3072, "p90\n(~33k tok)"),
-                         (6144, "p95\n(~65k tok)"),
-                         (11500, "p99\n(11.5 GiB)")]:
-        ax.axvline(kv_mib, color="#666", lw=0.5, ls=":", alpha=0.5)
-        ax.text(kv_mib, 2, lbl, fontsize=8, color="#444",
-                ha="center", va="bottom")
-
-    ax.set_xscale("log"); ax.set_yscale("log")
-    ax.set_xlim(40, 14000)
-    ax.set_ylim(1, 12000)
-    ax.set_xlabel("Per-request KV size (MiB, log)")
-    ax.set_ylabel("Time per request (ms, log)")
-    ax.set_title("§3.2 headline — PD-disagg KV transfer cost vs phase-isolation benefit\n"
-                 "(both measured on vanilla vLLM 0.18.1 + Mooncake 0.3.11, agentic regime)")
-    ax.grid(True, which="both", alpha=0.3)
-    ax.legend(loc="upper left", fontsize=9)
-
-    # Add explanatory annotation
-    ax.text(10000, 5000,
-            "Cost > benefit for ANY KV size above\n"
-            "the green band (~80 MiB / ~830 tokens).\n"
-            "Below: cost is marginal (<10 ms) but\n"
-            "benefit is also small (decode is short).",
-            fontsize=9, color="#333",
-            ha="right", va="top",
-            bbox=dict(boxstyle="round,pad=0.4", facecolor="#fffacd", alpha=0.9, edgecolor="#888"))
-
-    fig.tight_layout(); fig.savefig(args.out_cb, dpi=150); plt.close(fig)
-    print(f"wrote {args.out_cb}")
 
 
 if __name__ == "__main__":

scripts/b3_isolated_policy.sh

@@ -32,10 +32,14 @@ RUNDIR="${3:?usage: $0 <policy> <trace> <rundir>}"
 
 # Auto-enable kv_both when the policy requires it.
 # KV_CONNECTOR (Mooncake|Nixl) selects the underlying connector when KV_BOTH=1.
+_KV_CONNECTOR_EXPLICIT="${KV_CONNECTOR:-}"
 KV_CONNECTOR="${KV_CONNECTOR:-Mooncake}"
-if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_kv_both" ]; then
+if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_v3" ] || [ "$POLICY" = "unified_kv_both" ]; then
     ENABLE_KV_BOTH=1
-    KV_CONNECTOR="Mooncake"
+    # honor explicit KV_CONNECTOR override (e.g. =Nixl); otherwise default Mooncake.
+    if [ -z "$_KV_CONNECTOR_EXPLICIT" ]; then
+        KV_CONNECTOR="Mooncake"
+    fi
 fi
 if [ "$POLICY" = "unified_nixl_both" ]; then
     ENABLE_KV_BOTH=1
@@ -143,7 +147,10 @@ for i in $(seq 0 $((N_INSTANCES - 1))); do
     combined_args="$combined_args http://127.0.0.1:$((BASE_PORT + i))"
 done
 proxy_extra=""
-if [ "$ENABLE_KV_BOTH" = "1" ]; then
+# Bootstrap ports only needed for Mooncake handshake. Nixl uses its own
+# UCX side-channel and the proxy forwards kv_transfer_params from src's
+# response body instead of pre-baking engine_id/bootstrap_addr.
+if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Mooncake" ]; then
     bp_list=""
     for i in $(seq 0 $((N_INSTANCES - 1))); do
         if [ -z "$bp_list" ]; then
@@ -154,11 +161,15 @@ if [ "$ENABLE_KV_BOTH" = "1" ]; then
     done
     proxy_extra="--bootstrap-ports $bp_list"
 fi
+if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Nixl" ]; then
+    proxy_extra="--connector-type nixl"
+fi
 nohup "$VENV/python" "$ROOT/scripts/cache_aware_proxy.py" \
     --port "$PROXY_PORT" \
     --combined $combined_args \
     --policy "$POLICY" \
     $proxy_extra \
+    ${EXTRA_PROXY_ARGS:-} \
     > "$RUNDIR/proxy.log" 2>&1 &
 disown
 tries=0

scripts/cache_aware_proxy.py

@@ -82,6 +82,36 @@ class Settings:
     # Patch 6.6: per-request KV-xfer wall-clock timeout (proxy side).
     pd_sep_xfer_timeout_s: float = 60.0
 
+    # --- unified_v3 (offload-decode) gating knobs -----------------------
+    # v3 differs from v2 in *direction*: prefill stays on the session-
+    # affinity host (which holds the prefix cache); decode is migrated to
+    # a less-loaded target. KV transfer flows prefill_host → decode_target.
+    # The target doesn't need cache — we're shipping the post-prefill KV
+    # over anyway. After successful migration the session affinity table
+    # rotates to decode_target so the *next* turn lands where the KV now
+    # lives.
+    v3_min_new_tokens: int = 8000               # same as v2: don't migrate tiny prefills
+    v3_min_prefill_decode_busy: int = 1         # prefill_host must have ≥ this many concurrent decode tokens to justify migrating
+    v3_target_load_ratio: float = 0.7           # target.num_requests must be < prefill_host.num_requests × this
+    v3_min_load_gap: int = 1                    # target.num_requests must also be ≤ prefill_host - this (absolute slack)
+    v3_rotate_affinity: bool = True             # after migration, set session affinity to decode_target.
+                                                # Empirically False is better — see cache_miss_audit (next turn hits 9.5%
+                                                # with rotation vs ~80% without), because delay_free_blocks doesn't
+                                                # actually preserve cross-turn KV on decode_target.
+    v3_prefer_cache_target: bool = True         # Mechanism B: among low-load candidates, prefer the one
+                                                # with the most prefix cache for this prompt — vLLM's connector
+                                                # auto-transfers only the missing portion (verified via
+                                                # smoke_partial_transfer: cache-rich dst is 77% faster than
+                                                # cold dst at 33k tokens, +512 ext).
+
+    # --- KV connector selection (governs PD-sep handshake) -------------
+    # "mooncake": pre-baked kv_transfer_params (bootstrap_addr+engine_id+transfer_id).
+    #             Requires --bootstrap-ports and vLLMs launched with MooncakeConnector.
+    # "nixl"    : response-forward handshake. src returns kv_transfer_params via
+    #             response body, proxy forwards to dst. Nixl auto-selects transport
+    #             via UCX (CUDA IPC / NVLink on intra-node, RDMA across nodes).
+    connector_type: str = "mooncake"
+
 
 SETTINGS = Settings()
 
@@ -538,6 +568,122 @@ def pick_instance_unified_v2(
     return chosen, chosen_idx, decision, None
 
 
+def pick_instance_unified_v3(
+    instances: list[InstanceState],
+    token_ids: list[int] | None,
+    session_id: str | None,
+    input_length: int,
+    affinity: dict[str, int],
+) -> tuple[InstanceState, int, dict, tuple[InstanceState, int] | None]:
+    """unified_v3 = unified hybrid + selective DECODE migration.
+
+    Direction-reversed from unified_v2:
+      - prefill stays on session-affinity host (`prefill_host`) so we keep
+        the 93%-intra-session prefix-cache reuse intact.
+      - decode is migrated to a lower-load `decode_target` when the
+        affinity host is busy with concurrent decodes.
+      - KV transfer flows prefill_host → decode_target (the opposite of
+        v2's src → chosen).
+      - target does NOT need pre-existing cache — we're shipping the
+        post-prefill KV over anyway.
+      - On successful migration the *caller* must rotate
+        `affinity[session_id] = decode_target_idx` so the next turn lands
+        where the KV now lives (decode_target retains the blocks after
+        completion, since mooncake defaults to delay_free_blocks=True).
+
+    Decision is purely load-based on the target side:
+      1. new_local ≥ v3_min_new_tokens   (don't pay RDMA for tiny prefills)
+      2. prefill_host.ongoing_decode_tokens ≥ v3_min_prefill_decode_busy
+         (the host is actually busy decoding; migration buys decode-bw)
+      3. ∃ target with both num_requests < prefill_host.num_requests × ratio
+         and num_requests ≤ prefill_host.num_requests − v3_min_load_gap
+
+    Returns (prefill_host, prefill_idx, decision, migrate). When migrate
+    is None the request is fully local on prefill_host. When migrate is
+    (decode_target_inst, decode_target_idx), the handler should run
+    prefill on prefill_host and ship KV to decode_target for decode.
+    """
+    prefill_host, prefill_idx, decision = pick_instance_unified_hybrid(
+        instances, token_ids, session_id, input_length, affinity)
+
+    decision["v3_migrate"] = False
+    decision["v3_decision"] = "local"
+    decision["v3_reason"] = None
+
+    if not token_ids:
+        decision["v3_reason"] = "no_token_ids"
+        return prefill_host, prefill_idx, decision, None
+
+    prefill_cache_hit = prefill_host.estimate_cache_hit(token_ids)
+    new_local = max(0, input_length - prefill_cache_hit)
+    decision["v3_prefill_cache_hit"] = prefill_cache_hit
+    decision["v3_new_local"] = new_local
+
+    # Gate 1: prefill must be large enough to amortise RDMA setup.
+    if new_local < SETTINGS.v3_min_new_tokens:
+        decision["v3_reason"] = (
+            f"new_local_below_threshold ({new_local} < {SETTINGS.v3_min_new_tokens})"
+        )
+        return prefill_host, prefill_idx, decision, None
+
+    # Gate 2: affinity host must be busy with concurrent decodes — that's
+    # what migrating decode-traffic-away buys us. If the host is idle
+    # there's no point.
+    if prefill_host.ongoing_decode_tokens < SETTINGS.v3_min_prefill_decode_busy:
+        decision["v3_reason"] = (
+            f"prefill_host_not_busy "
+            f"(ongoing_decode_tokens={prefill_host.ongoing_decode_tokens} < "
+            f"{SETTINGS.v3_min_prefill_decode_busy})"
+        )
+        return prefill_host, prefill_idx, decision, None
+
+    # Gate 3: pick the lowest-load target that is materially less loaded
+    # than the prefill_host. Cache content irrelevant — KV ships over.
+    threshold_loaded = max(1,
+        int(prefill_host.num_requests * SETTINGS.v3_target_load_ratio))
+    candidates = [
+        (i, inst) for i, inst in enumerate(instances)
+        if i != prefill_idx
+        and inst.num_requests < threshold_loaded
+        and inst.num_requests <= prefill_host.num_requests - SETTINGS.v3_min_load_gap
+    ]
+    if not candidates:
+        decision["v3_reason"] = (
+            f"no_low_load_target "
+            f"(prefill_host.num_req={prefill_host.num_requests} "
+            f"threshold={threshold_loaded})"
+        )
+        return prefill_host, prefill_idx, decision, None
+
+    # Mechanism B (v3_prefer_cache_target=True): rank candidates first by
+    # cache_hit DESC (more cache = less KV to transfer), then by load. vLLM
+    # auto-skips transferring overlapping prefix when dst's local cache
+    # matches — verified in smoke_partial_transfer: 77% faster on a 33k
+    # prompt when dst has the prefix already.
+    if SETTINGS.v3_prefer_cache_target:
+        decode_target_idx, decode_target = min(
+            candidates,
+            key=lambda x: (-x[1].estimate_cache_hit(token_ids),
+                           x[1].num_requests, x[1].ongoing_tokens))
+    else:
+        decode_target_idx, decode_target = min(
+            candidates, key=lambda x: (x[1].num_requests, x[1].ongoing_tokens))
+
+    target_cache_hit = decode_target.estimate_cache_hit(token_ids)
+    decision["v3_migrate"] = True
+    decision["v3_decision"] = "migrate_decode"
+    decision["v3_target_idx"] = decode_target_idx
+    decision["v3_target_num_req"] = decode_target.num_requests
+    decision["v3_target_cache_hit"] = target_cache_hit
+    decision["v3_prefill_num_req"] = prefill_host.num_requests
+    decision["v3_reason"] = (
+        f"prefill_host.num_req={prefill_host.num_requests} busy; "
+        f"target.num_req={decode_target.num_requests} cache_hit={target_cache_hit}, "
+        f"transferring KV after prefill"
+    )
+    return prefill_host, prefill_idx, decision, (decode_target, decode_target_idx)
+
+
 def _extract_output_token_ids_from_sse(
     buffer: str,
     chunk: bytes,
@@ -764,10 +910,11 @@ async def lifespan(app: FastAPI):
         policy = getattr(global_args, 'policy', 'linear')
         # Mooncake-based modes still need bootstrap discovery; NIXL uses
         # its own UCX side-channel and doesn't go through our proxy
-        # bootstrap path (and unified_nixl_both never PD-seps anyway).
+        # bootstrap path. With --connector-type=nixl, v3 also skips bootstrap.
         needs_bootstrap = (
             global_args.offload
-            or policy in ("unified_v2", "unified_kv_both")
+            or (policy in ("unified_v2", "unified_v3", "unified_kv_both")
+                and getattr(global_args, 'connector_type', 'mooncake') == 'mooncake')
         )
         if needs_bootstrap and bp_list:
             await init_prefill_bootstrap(combined_instances, app.state.ready)
@@ -953,6 +1100,26 @@ async def _handle_combined(api, req_data, token_ids, input_length, session_id, h
         breakdown.update(decision)
         if session_id:
             session_affinity_combined[session_id] = best_idx
+    elif policy == "unified_v3":
+        # v3: prefill on affinity (cache reuse), decode migrated to a
+        # low-load target. KV flows prefill_host → decode_target.
+        # Reuses _handle_combined_pd_sep_v2 with src=prefill_host,
+        # dst=decode_target (the handler is direction-agnostic).
+        chosen, best_idx, decision, pd_sep_v2 = pick_instance_unified_v3(
+            combined_instances, token_ids, session_id, input_length,
+            session_affinity_combined)
+        breakdown.update(decision)
+        if session_id:
+            if pd_sep_v2 is not None and SETTINGS.v3_rotate_affinity:
+                # Migration + rotation: redirect next turn to decode_target,
+                # assuming KV will live there. (Empirically wrong — see
+                # cache_miss_audit. Keep behind a flag.)
+                _decode_target_inst, decode_target_idx = pd_sep_v2
+                session_affinity_combined[session_id] = decode_target_idx
+            else:
+                # No rotation: keep affinity on prefill_host (where the prefix
+                # cache lives). This is the empirically correct choice.
+                session_affinity_combined[session_id] = best_idx
     else:  # linear (default)
         chosen, best_idx = pick_instance(
             combined_instances, token_ids, session_id, input_length,
@@ -983,13 +1150,35 @@ async def _handle_combined(api, req_data, token_ids, input_length, session_id, h
     })
 
     if pd_sep_v2 is not None:
-        src_inst, src_idx = pd_sep_v2
+        # Handler contract: first arg = prefill source (does same-worker
-        breakdown["v2_src_url"] = src_inst.url
+        # prefill with do_remote_decode=True, max_tokens=1), second arg =
-        breakdown["v2_src_idx"] = src_idx
+        # decode target (does do_remote_prefill=True, pulls KV via
-        return await _handle_combined_pd_sep_v2(
+        # Mooncake, decodes).
-            api, req_data, headers, token_ids, input_length,
+        #
-            src_inst, chosen, breakdown,
+        # v2 contract:  pd_sep_v2 = (src_inst, src_idx); chosen = decode
-            request_id=request_id)
+        #               → src does prefill (it has more cache), chosen decodes.
+        # v3 contract:  chosen = prefill_host (affinity, has cache);
+        #               pd_sep_v2 = (decode_target_inst, decode_target_idx)
+        #               → chosen does prefill (cache reuse), decode_target decodes.
+        if policy == "unified_v3":
+            decode_target_inst, decode_target_idx = pd_sep_v2
+            prefill_inst = chosen
+            breakdown["v2_src_url"] = prefill_inst.url
+            breakdown["v2_src_idx"] = best_idx
+            breakdown["v3_decode_target_url"] = decode_target_inst.url
+            breakdown["v3_decode_target_idx"] = decode_target_idx
+            return await _handle_combined_pd_sep_v2(
+                api, req_data, headers, token_ids, input_length,
+                prefill_inst, decode_target_inst, breakdown,
+                request_id=request_id)
+        else:
+            src_inst, src_idx = pd_sep_v2
+            breakdown["v2_src_url"] = src_inst.url
+            breakdown["v2_src_idx"] = src_idx
+            return await _handle_combined_pd_sep_v2(
+                api, req_data, headers, token_ids, input_length,
+                src_inst, chosen, breakdown,
+                request_id=request_id)
 
     return await _handle_local_request(
         api, req_data, headers, token_ids, input_length,
@@ -1011,11 +1200,12 @@ async def _handle_combined_pd_sep_v2(
     of SETTINGS.pd_sep_xfer_timeout_s, so a stuck KV transfer fails
     the request instead of hanging for 600 s.
     """
-    if src.bootstrap_port is None:
+    connector = SETTINGS.connector_type
+    if connector == "mooncake" and src.bootstrap_port is None:
         raise HTTPException(
             status_code=500,
             detail=(
-                "unified_v2 PD-sep triggered but src instance "
+                "Mooncake PD-sep triggered but src instance "
                 f"{src.url} has no bootstrap_port; launch with "
                 "kv_role=kv_both and pass --bootstrap-ports"
             ),
@@ -1029,12 +1219,16 @@ async def _handle_combined_pd_sep_v2(
     src_load_held = True
     dst_load_held = True
 
+    # Build prefill kv_transfer_params per connector.
     prefill_data = req_data.copy()
-    prefill_data["kv_transfer_params"] = {
+    if connector == "mooncake":
-        "do_remote_decode": True,
+        prefill_data["kv_transfer_params"] = {
-        "do_remote_prefill": False,
+            "do_remote_decode": True,
-        "transfer_id": f"xfer-{request_id}",
+            "do_remote_prefill": False,
-    }
+            "transfer_id": f"xfer-{request_id}",
+        }
+    else:  # nixl: src just signals it'll produce KV for remote decode
+        prefill_data["kv_transfer_params"] = {"do_remote_decode": True}
     prefill_data["stream"] = False
     prefill_data["max_tokens"] = 1
     prefill_data["min_tokens"] = 1
@@ -1044,11 +1238,23 @@ async def _handle_combined_pd_sep_v2(
 
     breakdown["t_prefill_sent"] = _time.monotonic()
     breakdown["t_prefill_sent_unix"] = _time.time()
+    forwarded_params: dict | None = None
     try:
         resp = await src.client.post(api, json=prefill_data, headers=p_headers)
         breakdown["t_prefill_done"] = _time.monotonic()
         breakdown["t_prefill_done_unix"] = _time.time()
         resp.raise_for_status()
+        if connector == "nixl":
+            # Nixl populates kv_transfer_params in the response body with
+            # remote_block_ids / remote_engine_id / remote_host / remote_port.
+            # We must read the body BEFORE aclose.
+            src_resp_json = resp.json()
+            forwarded_params = src_resp_json.get("kv_transfer_params")
+            if not forwarded_params or not forwarded_params.get("remote_block_ids"):
+                raise HTTPException(
+                    status_code=502,
+                    detail=f"Nixl src returned no remote_block_ids: {forwarded_params}",
+                )
         await resp.aclose()
         src.record_prefix(token_ids)
     except Exception as e:
@@ -1057,11 +1263,16 @@ async def _handle_combined_pd_sep_v2(
         breakdown["prefill_error"] = True
         breakdown["error_detail"] = repr(e)[:300]
         _breakdown_log.append(breakdown)
-        # Release reservations on failure.
+        # Release reservations on failure. Clear load_held flags so the
-        src.ongoing_tokens -= input_length
+        # finally block below does not double-decrement (CRITICAL audit #1).
-        src.num_requests -= 1
+        if src_load_held:
-        dst.ongoing_tokens -= input_length
+            src.ongoing_tokens -= input_length
-        dst.num_requests -= 1
+            src.num_requests -= 1
+            src_load_held = False
+        if dst_load_held:
+            dst.ongoing_tokens -= input_length
+            dst.num_requests -= 1
+            dst_load_held = False
         raise HTTPException(status_code=502, detail=f"Prefill failed: {e}")
     finally:
         if src_load_held:
@@ -1069,17 +1280,19 @@ async def _handle_combined_pd_sep_v2(
             src.num_requests -= 1
             src_load_held = False
 
-    parsed = urllib.parse.urlparse(str(src.client.base_url))
-    bootstrap_addr = f"http://{parsed.hostname}:{src.bootstrap_port}"
-
     decode_data = req_data.copy()
-    decode_data["kv_transfer_params"] = {
+    if connector == "mooncake":
-        "do_remote_decode": False,
+        parsed = urllib.parse.urlparse(str(src.client.base_url))
-        "do_remote_prefill": True,
+        bootstrap_addr = f"http://{parsed.hostname}:{src.bootstrap_port}"
-        "remote_bootstrap_addr": bootstrap_addr,
+        decode_data["kv_transfer_params"] = {
-        "remote_engine_id": src.engine_id.get(0, ""),
+            "do_remote_decode": False,
-        "transfer_id": f"xfer-{request_id}",
+            "do_remote_prefill": True,
-    }
+            "remote_bootstrap_addr": bootstrap_addr,
+            "remote_engine_id": src.engine_id.get(0, ""),
+            "transfer_id": f"xfer-{request_id}",
+        }
+    else:  # nixl: forward what src returned
+        decode_data["kv_transfer_params"] = forwarded_params
 
     breakdown["t_decode_sent"] = _time.monotonic()
     breakdown["t_decode_sent_unix"] = _time.time()
@@ -1304,7 +1517,8 @@ def parse_args():
     p.add_argument("--policy", type=str, default="linear",
                    choices=["linear", "lmetric", "load_only", "sticky",
                              "unified", "unified_kv_both",
-                             "unified_nixl_both", "unified_v2"],
+                             "unified_nixl_both", "unified_v2",
+                             "unified_v3"],
                    help="Routing policy: linear (cache-aware), lmetric (P_tokens × BS), "
                         "load_only (B3 control: pure min-num_requests), "
                         "sticky (B3 control: hard session affinity), "
@@ -1317,6 +1531,20 @@ def parse_args():
                         "or unified_v2 (unified + selective per-request PD-sep "
                         "via Mooncake; requires --bootstrap-ports and "
                         "kv_role=kv_both vLLM launch)")
+    p.add_argument("--v3-rotate-affinity", type=int, default=1, choices=[0,1],
+                   help="unified_v3 only: 1 = rotate session affinity to decode_target "
+                        "after migration (original behavior, empirically loses prefix cache); "
+                        "0 = keep affinity on prefill_host so next turn hits its cache.")
+    p.add_argument("--connector-type", type=str, default="mooncake",
+                   choices=["mooncake", "nixl"],
+                   help="PD-sep handshake protocol. 'mooncake' uses pre-baked engine_id"
+                        " + bootstrap_addr (requires --bootstrap-ports). 'nixl' uses"
+                        " response-forward (src returns kv_transfer_params, proxy"
+                        " relays to dst; Nixl/UCX auto-picks NVLink intra-node).")
+    p.add_argument("--v3-prefer-cache-target", type=int, default=1, choices=[0,1],
+                   help="Mechanism B: unified_v3 picks decode_target with the most"
+                        " prefix cache among low-load candidates (default 1). Set 0"
+                        " to fall back to pure-load tie-break (cache-blind).")
     p.add_argument("--overload-factor", type=float, default=2.0,
                    help="Break session affinity when instance load > factor * avg")
     # The four flags below are accepted for bench.sh backward compatibility but
@@ -1354,7 +1582,14 @@ if __name__ == "__main__":
     SETTINGS.max_offload_inflight = global_args.max_offload_inflight
     SETTINGS.cache_gate_ratio = global_args.cache_gate_ratio
     SETTINGS.decode_iteration_s = getattr(global_args, 'decode_iteration_s', 0.05)
-    print("SETTINGS: throughput=%.0f rdma_overhead=%.2f offload=%s" % (
+    SETTINGS.v3_rotate_affinity = bool(getattr(global_args, 'v3_rotate_affinity', 1))
+    SETTINGS.connector_type = getattr(global_args, 'connector_type', 'mooncake')
+    SETTINGS.v3_prefer_cache_target = bool(getattr(global_args, 'v3_prefer_cache_target', 1))
+    print("SETTINGS: throughput=%.0f rdma_overhead=%.2f offload=%s v3_rotate_affinity=%s "
+          "connector_type=%s v3_prefer_cache_target=%s" % (
         SETTINGS.prefill_throughput, SETTINGS.rdma_overhead_s,
-        getattr(global_args, 'offload', False)))
+        getattr(global_args, 'offload', False),
+        SETTINGS.v3_rotate_affinity,
+        SETTINGS.connector_type,
+        SETTINGS.v3_prefer_cache_target))
     uvicorn.run(app, host=global_args.host, port=global_args.port)