MB5 PD ablation v2 results: concurrency axis + reuse 3-way writeup

- fig3_conc32k.json + fig3_concurrency_axis.png: agentic-corner concurrency
  sweep (in=32768, reuse=0.984, out=128), N 8->128, PD capped 600s / colo
  uncapped. colo completes 100% at every N (graceful, E2E 2.4s->81s); every
  static PD split collapses, earlier as N rises (viable only N<=16; <27% by N=32).
- analysis/mb5_pd_ablation/README.md: self-contained v2 writeup. Reuse axis
  3-way (A=d2048/o256, C=d2048/o128, B=d1024/o128) decomposes shape: output
  ~negligible, delta (real prefill/turn) dominant; crossover to colo at reuse
  ~90-95% robust. Run on dash2 (dash0 NICs faulted for Mooncake).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.gitignore

@@ -3,7 +3,14 @@ __pycache__/
 .venv/
 *.egg-info/
 outputs/
-traces/
+traces/*
+# ship the anonymized sampled trace + its provenance (metadata only, no cleartext)
+!traces/w600_r0.0015_st30.jsonl
+!traces/README.md
 *.log
 .claude/
 # third_party/vllm tracked in git for patch management
+!traces/w600_r0.0015_st30_first600s.jsonl
+# + time_to_parent_chat annotation (for --dispatch-mode thinktime); same anon data
+!traces/w600_r0.0015_st30_ttp.jsonl
+!traces/w600_r0.0015_st30_first600s_ttp.jsonl

PAPER_OUTLINE.md

@@ -1,423 +1,362 @@
-# EAR: Elastic Affinity Routing for Agentic LLM Serving
+# GPU-Hit-First: Serving Agentic LLM Workloads by Keeping the Working Set in HBM
 
-> **One-liner**: Agentic LLM workload 的 KV reuse 93% 是 intra-session 的，且 turn 间 tool-call 反馈耦合把单 request 的延迟差放大成 throughput 差距 —— locality 因此成为主导调度杠杆；现有 load-balance 丢 locality、静态 PD-disagg 撞 D 侧 KV 墙、pure session-sticky 造 hot pin，我们提出 session-affinity routing + hot-instance 触发 session migration 的调度器 **EAR (Elastic Affinity Router)**，单一方案同时拿到 locality 和 balance。
+> **Thesis (one-liner)**: 对 agentic LLM 负载，用户感受到的端到端 metric 是 **request latency / TPS / GPU
+> utilization**，而它们由一件事主导 —— **KV cache 命中是否发生在 GPU HBM 上**。Agentic 的 KV reuse 93% 在
+> session 内、且活跃 working set 小到一个节点就能常驻 HBM；命中层级 `GPU ≫ CPU-local > remote-RDMA-store ≫
+> recompute` 的代价差随 context 拉大。由此得到一条统一原则 —— **GPU-hit-first**：把活跃 working set 留在 HBM，
+> 而不是建深的 CPU/storage hierarchy 去追长尾。三个推论分别修复现有系统的三处失配：(3.1) 让 PD-colocation 重新
+> 成为默认；(3.2) 在全局路由里做 biased KV-cache-awareness；(3.3) 用 KV migration 而非 replication 做跨实例
+> GPU 去重。
+
+> **Framing note (2026-05-30)**：本 outline 取代早期的 "EAR: Elastic Affinity Routing" 版本（保留在 git
+> 历史里）。原 EAR 的 dispatch-coupling 形式化在此 **降级为 §2 的 metric 论证**（解释"为什么是 request latency
+> 而不是 TTFT/TPOT"），不再是 headline；headline 升格为 GPU-hit-first 原则，affinity routing / migration
+> 成为该原则的两个推论（§3.2 / §3.3）。
 
 ---
 
-## 📊 Validation Status (2026-05-27)
+## 📊 Validation Status (2026-05-30)
 
-| 部分 | 现有数据 | 待补 |
-|---|---|---|
-| §2 Workload characterization | ✅ 完整 (3 张图复用) | — |
-| §3.1 Load-balance 丢 locality | ✅ 完整 (`f4a`) | — |
-| §3.2 静态 PD-disagg 撞 KV 墙 | ✅ 完整 (`f4b`) | — |
-| §3.3 Sticky 造 hot pin | ✅ 完整 (`f4c`, `f4d`) | — |
-| §4.1-2 Affinity routing | ✅ 已实现（current `unified` 算法）| — |
-| `kv_both` substrate cost | ✅ **VERIFIED net-positive** (2026-05-27, commit `ef9e010`) | TTFT p90 −18.6% w/o DR-fix, −36.6% w/ DR-fix |
-| §4.3 Migration mechanism (e2e) | 🚧 **PARTIAL** | substrate 已通；e2e trigger + target selection 实验未跑 |
-| §5.2 End-to-end | ⚠️ 5/6 baseline 有数据 (`f6`) | 缺 static PD-disagg；EAR 列待 migration |
-| §5.3 Ablation | 🚧 **PARTIAL DEFER** | 仅 affinity-only 现可做，full 待 migration |
-| §5.4 Dispatch coupling validation | 🚧 **NEW DATA NEEDED** | 5 baseline wall-clock 重跑（Phase 1 patch 后）|
-| §5.5 Sensitivity | 🚧 **PARTIAL DEFER** | λ/skew/KV pool 可做；`T_hot`/`T_cool` 待 migration |
-| §5.6 Migration microbench | 🚧 **FULL DEFER** | 完全依赖 migration validation |
-
-**前提背景**：team 之前 4 次尝试 migration 都因 transfer overhead 被还原（见 `analysis/unified_routing_fix_review.md`）；2026-05-27 的 trace-replay A/B/C（`microbench/connector_tax/cache_sweep/REPORT_TRACE_REPLAY.md`）证明 `kv_both` substrate 已经反转 —— 不仅 +45% penalty obsolete，substrate 本身就是 net positive（TTFT p90 −18.6% vs plain，DR-fix 后 −36.6%）。**之前 4 次 migration revert 的最大根因消失，但 e2e migration 策略层（trigger + target selection 在反馈环里的真实收益）仍未直接验证 ——  EAR 的 migration 部分实验已无 substrate 风险，只剩策略层风险。**
+| 章节 | 论点 | 证据 | 状态 |
+|---|---|---|---|
+| §1 | 背景：PD-colo / PD-disagg / KV storage hierarchy | — | 写作 |
+| §2 metric | request latency 而非 TTFT/TPOT；TPS；GPU util | dispatch coupling + amplification；`bench_report.py` | 🟡 论证全，wall-clock sweep 待补 |
+| §2.1 | KV hit 普遍且关键 | `f2a` intra 93.2%、APC 上界 79.6%、C1/C2 | ✅ |
+| **§2.2** | **GPU hit > CPU hit > RDMA-store hit ≫ miss** | **`v2/figs/exp_a_tier_latency.png`（四层实测）** | ✅ **NEW** |
+| **§2.2 Ev#1** | **GPU 足以常驻"有价值的" working set** | **`v2/figs/exp_b_capacity_knee.png`（knee）+ working_set + cluster-scale 校正** | ✅ **NEW** |
+| §3.1 | Make PD-colocation great again | `PD_DISAGG_RESULTS`、`crossover_pd_advantage`、MB1/MB2、C2/C3 | ✅ |
+| §3.2 | Biased KV-aware global routing | LPWL（TTFT p90 −31%）、LMetric 乘性稀释、sticky hot-pin、ES ablation | ✅ |
+| §3.3 | GPU dedup via migration not replication | substrate net-positive（−18.6%/−36.6%）；correctness smoke tests | 🟡 substrate 通，policy e2e 待验证 |
+| §4 | 集成系统端到端 eval | 散落 mb1/mb2/mb5/crossover/lpwl，需统一 | 🚧 |
+| §5 | Related work（含 storage hierarchy 正面回应）| — | 写作 |
 
 ---
 
-## §1 Introduction
+## §1 Background and System Setup
 
-Agentic LLM workload —— 由 LLM 通过 tool call 自驱、多 turn 完成任务 —— 已经成为推理系统的主导负载，但现有为 chatbot 设计的调度策略在 agentic 下普遍失败。本文先刻画 agentic 与 chatbot 的本质区别，然后说明为什么三类主流调度都不够，最后给出 EAR 设计。
+### §1.1 LLM 与 KV cache
 
-**Contributions**:
+Transformer 自回归推理分两段：**prefill**（一次性算完 prompt 的全部 KV，compute-bound）与 **decode**（逐 token
+生成，memory-bandwidth-bound）。每个 token 的 KV 常驻 GPU HBM 才能被后续 attention 复用。Prefix caching（APC）让
+相同 prompt 前缀直接命中已算好的 KV，省掉重复 prefill —— 这是本文全部优化的物理基础。
 
-- **C1 Dispatch coupling 论证**：我们形式化一个 agentic workload 独有的反馈环 —— 单 turn 服务时间通过 Little's Law 隐式方程影响并发 session 数，从而把 per-request 延迟差放大成 throughput 差距。实测：load-balance baseline 在 600s trace 上跑出 **8x** wall-clock amplification；EAR 跑出 **TBDx**。
-- **C2 EAR 设计**：两个 pillar 的调度器 —— affinity-default routing 抓 intra-session locality，hot-instance 触发的 session migration 在 hotspot 出现时把整个 session 的 KV 搬到更轻的 instance，避免 hot pin。
-- **C3 评估**：在真实 Qwen3-Coder agentic trace 上，EAR 同时 dominate 5 个 baseline 的 TTFT、TPOT、APC、worst-worker p90、wall-clock 五个维度。
+> Qwen3-Coder-30B-A3B（GQA, 48 层, 4 KV heads, head_dim 128, bf16）：**KV = 96 KiB/token**，1 GiB = 10,923
+> token，block(16 tok) = 1.573 MB。
 
-**Figure 1: Teaser — wall-clock vs trace-time across schedulers** — `figs/f1_teaser.png` **🚧 TBD (NEW DATA NEEDED)**
-> Needs Phase 3 measurements: 5 baselines × 3 runs of trace replay, extract `amplification = wall_clock_s / trace_span_s` from each summary (Phase 1 patch already exposes the field). Plot as bar chart with y=1 reference line. EAR row 暂为 TBD（待 migration validation）。
+### §1.2 Agentic workflow
+
+Agentic 负载 = LLM 通过 tool-call 自驱、多 turn 完成任务。与 chatbot 的本质差异：每个 turn 由上一个 turn 的
+tool-call 结果触发（无人类 think-time），prefill-dominated（input/output ≈ 75×）。
+
+**但它是一个 mixture，不是"全多轮"**（C1, `figs/workload_chars/c1_session_mixture.png`）：
+
+- **90.3%** 的 session 是单轮（mean 1.62 turns）；但多轮 session（9.7%）= **44.2% 的请求**、**66.9% 的
+  prefill 质量**。
+- Continuation hazard（Lindy）：turn1→2 仅 **10.2%**，turn5→6 87%，turn12→13 **94.3%** —— heaviness 在
+  cold-start 几乎不可预测（corr(turn1_input, n_turns) = **0.04**）。
+
+> **Routing 含义（贯穿 §3.2）**：heaviness 在 session 起点不可预测 → 必须 **reactive**（观测累计负载），不能
+> proactive 预判；单轮海洋与深尾的最优策略相反（前者 load-balance、后者 affinity-pin），且 turn-1 无法区分 →
+> 唯一可行的策略是"人人 load-balanced 起步、随 turn 累积变 sticky" —— 正是 LPWL 的 emergent 行为。
+
+### §1.3 Serving agents in the wild
+
+- **PD colocation（8C）**：每个 instance 对称，prefill 与 decode 在同一 GPU 上由 chunked-prefill +
+  continuous batching 交错。弹性 KV 池，无静态分区。
+- **PD disaggregation**：把 instance 静态分成 prefill 池（P）与 decode 池（D），物理隔离两个阶段
+  （DistServe / Splitwise）。
+- **KV cache storage hierarchy**：GPU HBM → CPU DRAM → 远端 pooled store（RDMA/SSD，如 Mooncake Store /
+  LMCache）。把被淘汰/跨实例的 KV 下沉到更慢但更大的层，用传输换重算。
+
+> 三者各被本文一节回应：PD-colo 在 §3.1 被"复活"为默认；PD-disagg 在 §3.1 被证否（agentic regime）；storage
+> hierarchy 在 §2.2 被定量地"限位"（GPU 命中远胜下层，且活跃 working set 本就装得下）。
 
 ---
 
-## §2 Background and Workload Characterization
+## §2 GPU memory hit is the key to serving agents
 
-### §2.1 Agentic Workload Primer
+### §2.0 正确的 metric：request latency / TPS / GPU utilization（不是 TTFT/TPOT）
 
-Agentic workload 与 chatbot 的三个本质差异：
+**为什么不是 per-request TTFT/TPOT**：agentic 的 turn 之间有反馈环，单 turn 的延迟会跨 turn **复利**成 session
+端到端时间与系统吞吐差距。只有 **request/session latency、tokens-per-second、GPU utilization** 能 capture 这件事。
 
-- **Multi-turn, programmatic continuation**：每个 turn 由上一个 turn 的 tool-call 结果触发，没有人类 think-time
-- **Prefill-dominated**：input/output token ratio **75x**，98% 计算在 prefill 阶段（chatbot 为 1-10x）
-- **Skewed sessions**（来自 Qwen3 production trace，n=1.3M session / 2.1M req / 7200s）：top 1% 贡献 **46.5%** input token，top 5% **66.5%**，top 10% **74.6%**，top 25% **87.5%**，top 50% **96.0%** —— 半数 session 几乎占满全部 input mass
+**Dispatch coupling（降级为本节论证）**。每个 turn 间有外部 gap `T_external`（chatbot 是人在读/想/打字；agentic
+是 tool 执行）。Little's Law：`L = Λ · N · (W_turn(L) + T_external)`。当 `W_turn ≳ T_external` 时
+`W_turn(L)` 经 KV 竞争耦合到并发 L，scheduler 的 ε 退步被反馈环放大成 wall-clock 数倍差。
 
-平均 session 长度 TBD turn、TBD 输入 token。Per-request KV footprint（Qwen3-Coder-30B-A3B, 98304 B/token）：p50 **1.8 GiB**, p90 **8.0 GiB**, p95 **9.6 GiB**, p99 **11.5 GiB**. 单 instance KV pool ≈ 0.4 × 96 GiB = **38.4 GiB**（剩 50% model params bf16 + 10% runtime activation），所以 p99 请求一个 instance 只能装 **3 个 concurrent decode**；改 PD-disagg 4P+4D 让系统 decode 容量直接减半（系统并发 24 → 12）。
+production trace 实测 `T_external` CDF（`figs/f3a_inter_turn_gap.png`）：
 
-### §2.2 KV Cache Reuse Topology
-
-Trace 上 KV reuse 的分解：
-
-| Class | Share |
-|---|---|
-| Intra-session | **93.2%** |
-| Cross-session | 5.7% |
-| Shared prefix | 1.1% |
-
-理论 APC 上界：any-session **80.3%**，intra-session-only **79.6%**，差距 <1pp。**cache 本质上是 session-local 的**；任何不保留 session affinity 的调度都丢掉绝大部分 reuse 机会。
-
-**Figure 2: Workload characterization (3 panels)** — 现有数据可复用：
-
-![F2a Reuse topology — intra 93.2% / cross 5.7% / shared 1.1%](figs/f2a_reuse_topology.png)
-
-![F2b Session input-token mass CDF — production trace top 1%/5%/10%/25%/50% = 46.5%/66.5%/74.6%/87.5%/96.0% (replay window overlaid for sanity)](figs/f2b_session_skew.png)
-
-![F2c Per-instance decode concurrency vs deployment (KV pool 38.4 GiB; p99 req fits only 3/inst; PD-disagg halves system decode capacity)](figs/f2c_kv_footprint_cdf.png)
-
-> 📝 Layout TBD：三张拼成 1×3 还是分散到 §2.1/§2.2/§2.4 各一张。
-
-### §2.3 Dispatch Coupling — Why Locality Dominates
-
-这是本文最依赖直觉的论证，单独成节。
-
-**直觉**。每个 turn 之间有一段**外部 gap** `T_external`（chatbot 是人在读+想+打字、agentic 是 tool 执行）。下一 turn 在 `T_external` 之后到达。Little's Law: `L = Λ · N · (W_turn(L) + T_external)`。系统能不能避免反馈环，取决于 `W_turn` 是否**远小于** `T_external`：
-- 如果 `W_turn ≪ T_external`：session 停留时间被 `T_external` 主导，scheduler 调度速度的小变化几乎不动 `L`，系统在**开环 regime**；
-- 如果 `W_turn ≳ T_external`：`W_turn(L)` 这一项被 KV 竞争耦合到 L，Little's Law 变成 L 的隐式方程，**闭环 regime**，scheduler 上的 ε 退步被反馈环放大成几倍的 L*。
-
-**Agentic 与 chatbot 不在二元区分上，而在 `T_external` 的分布上**。下面是 production trace 实测的 `T_external = next.start − prev.end` CDF（agentic = Qwen3-Coder, n=783k inter-turn gaps; chatbot = qwen3-max chat, n=42k gaps）：
-
-![F3a Inter-turn external gap CDF — agentic vs chatbot](figs/f3a_inter_turn_gap.png)
-
-| Metric | Agentic | Chatbot |
+| | Agentic | Chatbot |
 |---|---:|---:|
-| p25 | 0.69s | 4.85s |
-| **p50** | **1.6s** | **7.2s** |
-| p90 | 44s | 15s |
-| p99 | 738s | 43s |
-| gap < 1s | **39%** | 4% |
-| gap < 5s | 67% | 29% |
+| p50 | **1.6 s** | 7.2 s |
+| gap < 1 s | **39%** | 4% |
+| gap < 5 s | 67% | 29% |
+| p99 | 738 s | 43 s |
 
-两个分布形状完全不同：
-- **Chatbot 是 unimodal**，5–15s 紧密集中（人类交互节奏）；
-- **Agentic 是 bimodal**：39% 的 gap < 1s（autonomous tool-call mode，chatbot 仅 4%）+ 13% 的 gap > 30s（session paused/abandoned，chatbot 仅 2%）。
+agentic 有一段 chatbot 没有的 **sub-second tool-call mass（39% vs 4%）**，几乎天然 `W_turn ≫ T_external` → 闭环。
+**实测**：lmetric 跑 600s trace 用 49 min wall-clock = **8× amplification**。**结论：per-turn 延迟的小差被放大成
+端到端数量级差 → 必须用 request latency / TPS / GPU util 衡量。**
 
-**Agentic 的危险来自 sub-second tool-call mode** —— 这 39% 的 turn 几乎天然 `W_turn ≫ T_external`，dispatch coupling 必然激活；而 chatbot 没有这一段质量，要把 W_turn 推得很大才会进入闭环。
+- **TPS / GPU util** 的工具：`microbench/fresh_setup/bench_report.py`（TTFT/TPOT/E2E 全分位 + TPS +
+  per-worker GPU util）。PD stall 时 GPU ~0% util vs colo 34%（§3.1）即是 GPU-util 作为 metric 的直接体现。
 
-**实测 regime 对照**：
+> **🚧 待补**：5 baseline × ≥3 runs 的 wall-clock amplification sweep（replayer 已输出 `amplification` 字段），
+> 钉死本节实证 closure。优先级高。
 
-| Scheduler | TTFT p90 | Agentic frac(W_turn > T_ext) | Chatbot frac(W_turn > T_ext) |
-|---|---:|---:|---:|
-| unified | 7.3s | **73%** | 32% |
-| lmetric | 15.7s | **80%** | 88% |
+### §2.1 KV$ hit is common and critical
 
-unified 在 agentic 上把 73% 的 turn 推进闭环，在 chatbot 上只有 32%。lmetric 在 agentic 上 80%、**chatbot 上也到 88%** —— lmetric 的 W_turn 大到把 chatbot 自己也推进闭环，这是 lmetric 在两种 workload 都 underperform 的一个直接根因。
+Trace 上 KV reuse 的分解（`figs/f2a_reuse_topology.png`）：**intra-session 93.2% / cross-session 5.7% /
+shared-prefix 1.1%**。理论 APC 上界：intra-only **79.6%** vs any-session 80.3%，差 <1pp —— **cache 本质上是
+session-local 的**。
 
-**具体例子**：一个 coding agent 跑 20 turn 的任务，假设 `T_external` 是 sub-second 模式（tool-call 0.5s）。
+per-turn 视角（C2, `figs/workload_chars/c2_work_amortization.png`）：resident context 11k→56k+ token 增长而
+new-prefill 从 2.7k 坍缩到 ~200 token，per-turn reuse 爬到 **99.6%**，resident/new（"PD tax"）到 turn 12 ≈
+250×、turn 30 ≈ 450×。**绝大部分 prefill 工作是可被命中省掉的**；命中与否直接决定 TTFT。
 
-- 快策略：`W_turn` = 2s，每 turn 总 2.5s，session 共 50s，平均并发 10 session
-- 慢策略（线性估算）：`W_turn` = 3s，每 turn 3.5s，session 70s，应并发 14
-- 慢策略（实际）：14 并发让 KV pool 更紧 → `W_turn` 推到 4s → session 90s → 18 并发 → `W_turn` 5s …… 反馈环放大到撞墙或落到一个远更糟的不动点
+### §2.2 Hits on GPU is more important than the CPU
 
-**形式化**。记 `Λ` = session 到达率，`N` = 每 session turn 数，`W_turn(L)` = 单 turn 服务时间作为并发 L 的递增函数（并发越多、KV 竞争越激烈、`W_turn` 越大）。Little's Law:
+**命中层级的代价是实测的，不是断言的**（Qwen3-Coder-30B-A3B / H20）。TTFT(s, p50) 服务一段长 L 的复用前缀，
+来自每个 KV 层：
 
-```
-L = Λ · N · (W_turn(L) + T_external)
-```
+![四层命中延迟：GPU < CPU-local < remote-RDMA-store ≪ miss，差距随 context 拉大](v2/figs/exp_a_tier_latency.png)
 
-设策略变化让 `W_turn` 整体放大 `(1+ε)` 倍，小扰动分析得到不动点 L\* 的灵敏度：
 
-```
-dL*/dε = L* · W_turn(L*) / [W_turn(L*) + T_external − Λ · N · W_turn(L*) · W'_turn(L*)]
-```
+| prefix L | miss(recompute) | **remote RDMA store** | CPU-local(DRAM,PCIe) | GPU(HBM) | miss/RDMA | RDMA/CPU | CPU/GPU |
+|---:|---:|---:|---:|---:|---:|---:|---:|
+| 8k  | 0.588 | 0.151 | 0.076 | 0.053 | 3.9× | 2.0× | 1.5× |
+| 16k | 1.547 | 0.262 | 0.105 | 0.063 | 5.9× | 2.5× | 1.7× |
+| 32k | 4.604 | 0.680 | 0.158 | 0.080 | 6.8× | 4.3× | 2.0× |
+| **64k** | **15.23** | **0.97** | **0.27** | **0.11** | **15.8×** | **3.6×** | **2.4×** |
 
-注意两点：
-1. 分子 ∝ `W_turn / (W_turn + T_external)`：当 `T_external ≫ W_turn` 时灵敏度 → 0（开环）；当 `T_external → 0` 时灵敏度趋于其上界（闭环）。**所以 agentic 的 sub-second tool-call mass 把灵敏度推到上界，chatbot 的 5–15s mass 把灵敏度压低**。
-2. 分母 `... − Λ · N · W'_turn(L*)`：接近 KV 饱和时趋于 0，**任何调度退步在饱和附近都被无限放大** —— 这是 lmetric 在 600s trace 上跑出 8x wall-clock 的根因。
+- **GPU hit ~flat**（42→111 ms / 1k→64k）：命中即整段前缀在 HBM，只重算最后一个 token。
+- **CPU-local hit** transfer-bound（PCIe H2D 实测 ~54 GB/s）；CPU-hit ≈ GPU-hit + KV/PCIe + ~0.15s 开销。
+  （native KV offload，命中经 `vllm:external_prefix_cache_hits` 100% 验证。）
+- **remote RDMA-store hit** = Mooncake-Store 机制（实测：两 instance，B 用 `do_remote_prefill` 经 RDMA 从 A 拉取
+  缓存前缀而非重算；`mb2_kv_transfer.py` / `v2/.../run_rdma.sh`）。对 recompute 是大赢（**最高 16×**，与 blog 的
+  46× 同向），但付 **NIC 税**（有效 ~5–7 GB/s，cf. MB2 raw ~9.7 GB/s；multi-NIC pooling 可抬高），故比 CPU-local
+  慢 3.6×、比 GPU 慢 ~9×（64k），**代价差随 context 拉大**。
+- **结论 —— 层级严格且随 context 拉大：`GPU < CPU-local < remote-RDMA-store ≪ miss`**。global KV store 确实
+  有用（这也是该路线存在的理由），但每靠近 GPU 一层就再省 1.4–4× TTFT。**最值钱的复用是 GPU-resident 的那种。**
 
-**Figure 3: Dispatch coupling schematic** — `figs/f3_coupling_schematic.png` **🚧 TBD (CUSTOM DRAW)**
-> 需要新画一张示意图：左半 timeline 对比（chatbot：`system → T_external (5–15s) → system`；agentic：`system → T_external (sub-second to long-tail) → system`），右半反馈环 `W_turn → L → W_turn`，标注两个 regime 的判别条件 `W_turn ≷ T_external`。
+#### Evidence #1：GPU is sufficient to hold most KV requests
 
-### §2.4 Takeaway
+**realized APC 与 latency 在很小的 GPU 容量就饱和**（closed-loop 多轮负载，并发 4，扫 GPU KV 容量）：
 
-三个性质 —— intra-session locality dominant (§2.2)、long context + prefill-heavy (§2.1)、dispatch coupling (§2.3) —— 共同决定了 agentic workload 的调度必须以 **locality 为主导**，并能容忍 skew 带来的 instance 间负载不均。
+![容量 knee：APC 与 TTFT p90 在 3.6 GB（=活跃 working set）饱和，之后 dead-flat](v2/figs/exp_b_capacity_knee.png)
+
+| GPU KV (GB) | realized APC | TTFT p90 |
+|---:|---:|---:|
+| 1.2 | 7.4% | 13.00 s |
+| 2.4 | 36.3% | 4.62 s |
+| **3.6** | **80.3%** | **0.53 s** |
+| 9.7 | 72.9% | 0.65 s |
+| 14.5| 72.9% | 0.65 s |
+
+**Knee 出现在 3.6 GB = 恰好 = 活跃 working set（4 session × 0.91 GB）**：APC 饱和到上界、TTFT p90 从 13.0s 坍缩
+到 0.53s，之后 dead-flat。**超过 working set 的 HBM 买不到额外收益；为追长尾而建的 CPU/storage tier 同理 ≈ 0。**
+
+**Cluster-scale 校正（关键）**：working_set 分析（`analysis/working_set/`，`figs/working_set/`）显示"装下整个
+2h cluster 的全部 reuse 尾巴需 ~14 节点"，**这不构成 CPU offload 的动机** —— 那是用 1 个 replica 容量去装**整个
+cluster** 的 reuse；产出该 trace 的真实 cluster 远不止 14 节点（trace 是 cluster 聚合，见
+`project-trace-is-cluster-level`）。在 per-cluster 的 HBM 总量下，活跃 working set 本就 GPU-resident（live KV
+533–1157 GB ≪ 单节点 1528 GB）。**knee 位置随并发线性增长 = 随 cluster GPU 数增长**，而 cluster 本就提供了它。
+
+> ⚠ **Scope**：本小节"装得下"指的是**活跃 working set 产生的近期高价值复用**，不是"全部 reuse 尾巴"。冷
+> session 长 gap 后回来的深尾命中（既低价值/byte 又贵于 fetch）正是 storage-hierarchy 派追的东西；本文论点是
+> 在 agentic 下这条尾巴不值得为之建深层级。§5 正面回应该派。
+
+### §2.3 Takeaway
+
+正确的 metric（§2.0）+ 命中集中在 GPU 才便宜（§2.2）+ 活跃 working set 装得下 HBM（Ev#1）⇒ **GPU-hit-first**：
+设计目标是最大化活跃 working set 的 **GPU 常驻 + 命中**，而非建深 CPU/storage hierarchy。§3 给出三个推论。
 
 ---
 
-## §3 Why Existing Schedulers Don't Fit
+## §3 Optimizing agent serving with the GPU-Hit-First Principle
 
-三类现有调度各自撞上 §2 三个性质中的一个：
+### §3.1 Make PD-colocation great again
 
-### §3.1 Load-balanced routing 丢 locality
+静态 PD-disaggregation 对 chatbot 有效，对 agentic 结构性失败 —— colocation 才应是默认。
 
-Round-robin 和 load-aware routing（如 LMetric, OSDI'26）最大化 instance 利用率，但忽略 session affinity。**实测 APC 跌到 56.9%**（vs 上界 79.6%），23pp 的差距直接来自丢失的 intra-session cache hit。违反 §2.2。
+**端到端证据**（`microbench/fresh_setup/PD_DISAGG_RESULTS.md`，8×H20，trace replay）：**没有任何静态 P/D 比能赢
+8-way colocation（8C）**，且失败模式随比例移动：
 
-**为什么"cache-aware load routing"也不够 —— LMetric 的 cache 信号被乘性 score 稀释**。LMetric 的打分是
+| Metric | **8C** | 6P+2D | 4P+4D | 2P+6D |
+|---|---:|---:|---:|---:|
+| completion | **100%** | 100% | 100% | **9%** 💀 |
+| wall-clock | **2994 s** | 3419 | 4171 | 5762 |
+| prefix-cache hit | **19.4%** | 0% | 0% | 0% |
+| TTFT p50 | **7.0 s** | 41.0 | 56.4 | 23.6 |
+| E2E p90 | **83.3 s** | 91.8 | 157.1 | 499 |
 
-```
-P = pending_prefill_tokens + (input_length - cache_hit)
-score = P × num_requests
-```
+- **D-heavy（4P+4D）**：decode 池饱和 **97.5%**、prefill 池 ~30% —— 半个 cluster 的 KV 被困在错的一侧；agentic
+  请求大（p99 KV **11.5 GiB**），4D 让系统 decode 容量直接减半（24→12 并发，`figs/f2c_kv_footprint_cdf.png`、
+  `figs/f4b_pdsep_kv_wall.png`）。
+- **P-heavy（2P+6D）**：prefill 池 jam 99.7%，872 请求堆积，**91% 永不完成**。
+- **更聪明的路由救不了**（§6.x）：给 P 侧加 session-affinity 反而更差（4P+4D completion 100%→36%），GPU ~0%
+  util，cluster 卡在 KV-transfer 协调而非 compute —— 复现 producer hot-pinning。
 
-cache_hit 只在 `P` 里作减项；而 `score` 是**乘性**的。一个 session affinity 的 instance 会因为持续接到 session 而 `num_requests` 升高，乘积把 cache 收益吃掉。例：8000 输入 token、暖 instance cache_hit = 7500 vs 冷 instance cache_hit = 0、pending_prefill 都是 2000、num_requests 分别 5 vs 1，则 LMetric score 暖 = `2500 × 5 = 12500`、冷 = `10000 × 1 = 10000`，**LMetric 选冷**，丢掉 ~90% cache。结果：
+**为什么 colo 赢（正确论证，C2/C3 支撑）**：
 
-| 策略 | APC | vs load_only | 设计点 |
+- **时变 P:D 需求**：agentic 同时在 roofline 两侧有实质工作 —— compute-bound prefill（~30% 时间）+
+  memory-bound decode（**~70% 时间**，C3 token≠time 校正，`figs/workload_chars/c3_prefill_decode_balance.png`）。
+  colo 的弹性池吸收当下热的那一相；静态分区让 P-instance 带宽闲、D-instance 算力闲。
+- **resident KV 本地化**（C2）：下一 turn 的 prefix = [prevPrompt+prevAnswer] 横跨 P/D 两侧，disagg 必须
+  gather/transfer，colo 免费本地保留。
+- **transfer 不便宜且拓扑无关**（MB2，`figs/mb2_transfer_time_compare.png`）：Mooncake `batch_transfer_sync_write`
+  恒走 RDMA NIC（~9.7 GB/s），intra ≈ inter；PD-disagg 的 per-request transfer 税无法靠拓扑买回。
+- **phase-isolation 是 disagg 唯一的真赢面但被压倒**（MB1，`figs/mb1_interference.png`：32k prefill 让
+  per-stream TPOT 退化 52×，131k → 183×）—— 但被 D 侧容量天花板压倒（见上）。
+
+**边界（不 overclaim）**：crossover sweep（`analysis/crossover/`，`figs/crossover_pd_advantage.png`）给出 colo
+停止占优的 input 长度 —— colo 在 agentic 工作点赢，且我们知道边界在哪。
+
+### §3.2 Biased KV-cache-awareness in global routing
+
+GPU-hit-first 在路由层 = **把 cache-awareness 作为带偏置的独立路由路径**，而不是折叠进 load cost。
+
+**反例：load-balanced / 朴素 cache-aware-load 丢 locality**（`figs/f4a_apc_loss.png`）。LMetric（OSDI'26）打分
+`P = pending_prefill + (input − cache_hit)`，`score = P × num_requests` —— cache 只作 cost-model **减项**，而 score
+是**乘性**的，有 affinity 的 instance 因 num_requests 高被乘式吃掉 cache 收益：
+
+| 策略 | APC | vs load_only | cache 处理方式 |
 |---|---:|---:|---|
-| load_only | 53.9% | — | 纯负载 (`score = num_requests`) |
-| LMetric | 57.2% | **+3.3pp** | cache 作 cost-model 减项 |
-| sticky | 77.7% | **+23.8pp** | cache 作硬约束 |
-| unified | 78.7% | **+24.8pp** | cache 作硬+软偏好混合 |
+| load_only | 53.9% | — | 无 |
+| LMetric | 57.2% | **+3.3pp** | cost-model 减项（被稀释）|
+| sticky | 77.7% | **+23.8pp** | 硬约束 |
+| unified | 78.7% | **+24.8pp** | 硬+软混合 |
 
-**`load_only → LMetric` 的 +3.3pp 几乎可忽略；`LMetric → sticky` 的 +20.5pp 才是 cache 信号被正确处理的回报**。Cache awareness 不能只作为 cost-model 的一项被吞掉 —— 必须作为**独立路由路径**（sticky / unified hybrid）。这是 §3.1 比"丢 locality"更具体的失败模式。
+`load_only→LMetric` 的 +3.3pp 几乎可忽略；**+20.5pp 的回报来自把 cache 作独立路由路径**。
 
-### §3.2 静态 PD-disaggregation 撞 D 侧 KV 墙
+**本文方法：LPWL（least-prefill-work，parameter-free）**（`project-lpwl-policy`、`analysis/lpwl_5policy_600s.md`）：
+按 `new_uncached ≈ input − cache_hit` 路由 —— `new_uncached≈input`（冷/新 session）自动按负载分散，`new_uncached≈0`
+（暖 session）自动 stick。**零旋钮**，在 600s trace 上击败 tuned unified+A+B（TTFT p90 **−31%**），full w600 上打平。
+这正是 C1 mixture 要的形状：无需 classifier 自动切分单轮海洋与深尾。
 
-静态把 instance 分成 P pool 和 D pool 对 chatbot 有效，对 agentic 失败：agentic 请求平均 33.6k token，需要 **3.3GB** KV；4D 方案下 p90 请求占 D 侧 KV pool **69%**，p99 直接 **溢出 138%**。结果：**TTFT p50 暴涨 62-72x**，成功率从 99.5% 跌至 **52-68%**。违反 §2.1（prefill-dominant + 长 context）。
+**两条被否的杠杆（节省读者时间）**：
 
-### §3.3 Pure session-sticky 的真正失败：全员被 hot session 拖累
+- **real-time engine state 不是路由杠杆**（ES ablation，`project-es-ablation-sweep`）：只对一次性 placement 有用
+  （sticky −26%），对 per-req load-chasing 有害（load_only +27%）。
+- **derived-κ decode 项 net-negative**：decode-awareness 是错的杠杆。
 
-硬 session-instance 绑定恢复 locality（APC **77.2%**，达到上界 97%），但**绝对 worker latency** 全员被拖累 —— 是 pure sticky 的真正失败模式。
+**Pure sticky 的失败用绝对 per-worker latency 衡量**（`figs/f4c_per_worker_ttft.png`，不能用 max/median 比值）：
 
 | | median worker TTFT p90 | max worker | system e2e p90 |
 |---|---:|---:|---:|
-| `sticky` | **20.3 s** | 55.4 s | **34.6 s** |
-| `unified` (affinity + LMetric fallback) | **10.3 s** | 37.7 s | **18.0 s** |
-| `lmetric` | 14.0 s | 31.3 s | 24.8 s |
+| sticky | **20.3 s** | 55.4 s | **34.6 s** |
+| unified | **10.3 s** | 37.7 s | **18.0 s** |
 
-机制：production trace 上 top 1% session 占 46.5% input mass、top 5% 占 66.5%，hot session 数量远大于 instance 数（8）；sticky 的 hash 绑定让 **每个 worker 都自己承接一份 hot session**，median worker 也被拖慢到 20s 量级。unified 用 LMetric fallback 把 cold/new session 重路由到非 hot worker，保留 7/8 worker 的速度。系统 p90 由大多数请求决定，所以 unified 在 e2e p90 上 ~2x 快于 sticky。
+hot session 数 ≫ instance 数（8），sticky 的 hash 绑定让**每个 worker 都自接一份 hot session**，median 也被拖慢；
+biased 路由把 cold/new 重路由到非 hot worker，保 7/8 worker 速度 → e2e p90 ~2× 快。**这引出 §3.3：sticky 的残余
+hot-pin 需要 migration 解。**
 
-**§3.3 sub-finding**：hot pin failure 必须用 **per-worker absolute latency**（median + max）衡量，**不能用 normalized ratio**。`max/median` 在 unified 这样的"affinity + escape"方案上会反向惩罚 —— sticky 的 ratio 2.73 比 unified 的 3.67 低，但 sticky 的 median 也高（20.3s vs unified 10.3s），ratio 越低反而越糟。本文 paper 中所有 worker 平衡相关的比较一律用 (median, max) 双指标，不用单一比值。
+### §3.3 GPU KV-cache dedup with migration instead of replication
 
-违反 §2.4 的 skew 容忍要求。
+**视角**：多个 instance 各自缓存同一段 prefix = GPU 容量被 **replication** 浪费；GPU-hit-first 要求**全局只留一份 +
+把 session 搬到那一份**（migration/dedup），既保 GPU 命中又均衡负载、修复 §3.2 的残余 hot-pin。
 
-**Figure 4: Three baselines, three failure modes** — 拆成三个子图，分别放在 §3.1/§3.2/§3.3：
+- **Trigger**：host 的 `pending_prefill > T_hot` 且 session 在 `T_cool` 内未迁移过。
+- **Target**：用 **observable pending prefill tokens** 选最轻 instance，**不用** cost-model 预测（mooncake cost
+  model 误差 10–21×，by construction 绕过）。
+- **Mechanism**：当前 request 重定向到 target，session KV 经 Mooncake `kv_connector` 迁移；binding 更新；后续
+  turn 按 affinity 路由到新 host。迁移成本在该 session 剩余 turn 上摊销。
+- **Thrashing prevention**：per-session cooldown `T_cool`。
 
-§3.1 — APC 实测 vs 理论上界 79.6% (lmetric 56.9%, load_only 54.1%, sticky 77.2%, unified 79.4%)：
-![F4a APC loss](figs/f4a_apc_loss.png)
-
-§3.2 — D 侧 KV pool 占用 vs per-request KV footprint，4P+4D 和 6P+2D 在 agentic regime 都穿过 90% 内存墙：
-![F4b PD-sep KV memory wall](figs/f4b_pdsep_kv_wall.png)
-
-§3.3 — Per-worker TTFT p90 across 8 instances × 5 policies。sticky 的所有 worker 都被拖慢（median 20.3s），unified 把伤害集中在 e4 上、其他 worker 快（median 10.3s）：
-![F4c Per-worker TTFT p90 distribution](figs/f4c_per_worker_ttft.png)
-
-
-> 📝 可选支撑图 — Prefill-decode 干扰（同 GPU 8k prefill 让 TPOT 退化 66x），放 §3.3 支撑 sticky 的 interference 论证：
-![F4d PD interference](figs/f4d_pd_interference.png)
-
-### §3.4 Takeaway
-
-**问题不是任何单一 baseline 太弱，而是没有一个方案同时满足 §2 的三个性质**：保留 locality、尊重 D 侧 KV 容量、容忍 skew 带来的负载不均。EAR 是据我们所知第一个三件事同时做到的调度器。
+> **状态**：**substrate 已验证 net-positive**（`kv_both` connector 在 trace replay 上 TTFT p90 **−18.6%**，
+> DR-fix 后 **−36.6%** vs plain；之前认为是 blocker 的 transfer overhead 已不存在，4 次历史 revert 的主因消失）。
+> migration correctness smoke tests 通过。**policy 层 e2e（trigger + target 在反馈环里的真实收益）仍未直接验证**
+> —— 这是全文最弱的一环，独立风险是"决策错误 + cooldown thrashing"。affinity-only pillar（§3.2 LPWL）已独立成立，
+> 即便 migration 仍 marginal，paper 也有 strong-routing 主线。
 
 ---
 
-## §4 Design: EAR
+## §4 System Evaluation
 
-### §4.1 Architecture
+> **🚧 关键缺口**：目前证据散落在 mb1/mb2/mb5/crossover/lpwl/v2；§4 需要一个**集成系统**（colocation +
+> biased routing + dedup-migration，统一命名）跑端到端、用 §2.0 的新 metric（request latency / TPS / GPU util）
+> 评测，并把 §3.1/§3.2/§3.3 做成 ablation。
 
-EAR 是位于 N 个同质 instance 之上的 router。每个 instance 是对称的 PD-colocated，没有静态 P/D 分区。每个 session 在 router 内维护一个 **host binding** —— 当前持有该 session KV 状态的 instance。Binding 在常态下稳定，仅在 hotspot 触发时通过 migration 改变。
+### §4.1 Setup
+- **Trace**：真实 Qwen3-Coder agentic trace（cluster-level，见 `project-trace-is-cluster-level`）；日常迭代
+  `w600_r0.0015_st30`（~850 req / ~13 min / peak QPS ~1.6 / APC headroom ~76%）。
+- **Hardware**：8× H20 (96 GB)；vLLM 0.18.1 (V1, chunked-prefill) + Mooncake。
+- **Baselines**：① round-robin ② LMetric (OSDI'26) ③ kvcache-aware+load 线性混合 ④ sticky ⑤ static PD-disagg
+  (4P/4D) ⑥ **本文系统**（colo + LPWL + dedup-migration）。
+- **Metrics**：request latency (mean/p50/p90/p99)、**TPS**、**GPU util (median/max worker)**、APC、
+  wall-clock amplification（不单看 TTFT/TPOT，§2.0）。
 
-**Figure 5: EAR architecture and request flow** — `figs/f5_architecture.png` **🚧 TBD (CUSTOM DRAW)**
-> 组件图：router (含 session→host table) → N 个 symmetric instances；affinity 路径实线，migration path 虚线。适合 TikZ / draw.io。
+### §4.2 End-to-end
+- `figs/f6_e2e_latency_bars.png` / `f6_e2e_latency_full_grid.png`（现有 4–5 baseline；🚧 补 static PD-disagg
+  列 + 本文系统列）。
 
-### §4.2 Pillar 1: Affinity-Default Routing
+### §4.3 Ablation（GPU-hit-first 三推论各关一个）
+- −colocation（→ static PD-disagg，§3.1）
+- −biased routing（→ load-balance / LMetric，§3.2）
+- −dedup-migration（→ pure sticky，§3.3）
+- 🚧 migration-only / full 待 policy e2e 验证。
 
-- **Cold start**：新 session 到达时，router 用 load-balance（选 pending prefill tokens 最少的 instance）分配初始 host
-- **Warm path**：已建立 session 的后续每个 turn 一律路由到当前 host
-- **效果**：intra-session KV reuse 被构造性保留，APC 接近 §2.2 的上界 79.6%
+### §4.4 Microbench 支撑（已有，复用）
+- §2.2 四层命中：`v2/figs/exp_a_tier_latency.png`
+- §2.2 容量 knee：`v2/figs/exp_b_capacity_knee.png`
+- §3.1 PD-disagg 失败：`figs/f4b`、`PD_DISAGG_RESULTS`、`crossover_pd_advantage`
+- transfer cost：`figs/mb2_*`；phase interference：`figs/mb1_interference.png`
 
-### §4.3 Pillar 2: Hot-Triggered Session Migration 🚧 PARTIAL VALIDATION
-
-避免 Pillar 1 退化成 pure sticky 的关键 mechanism。
-
-> **状态（2026-05-27 更新）**：
-> - **Substrate 验证 PASS**（commit `ef9e010`）：`kv_both` connector 在 trace replay 上 net positive（TTFT p90 −18.6%），DR-fix 后再 −22%。之前认为是 migration blocker 的 transfer overhead 已不存在。
-> - **策略层 e2e 验证 PENDING**：trigger 阈值 + target selection 在 agentic 反馈环里的真实收益仍未直接测。之前 4 次 migration 尝试（`6b255fa`, `e991960/5772149`, `cc6e562`, `4c583f2`）被还原的主因（substrate overhead）已消失，但 trigger 决策错误 + cooldown thrashing 是独立风险，需新一轮 e2e 实验确认。
-
-#### §4.3.1 Trigger signal
-
-EAR 实时监控每个 instance 的 **pending prefill tokens**。新 request 到达且按 affinity 应该路由到 host H 时，router 先检查：
-- `H.pending_prefill > T_hot`？（hotspot 检测）
-- session 在过去 `T_cool` 秒内未发生过 migration？（thrashing prevention，§4.3.4）
-
-两个条件同时满足才考虑触发 migration。`T_hot` 和 `T_cool` 的取值见 §5.5 sensitivity。
-
-#### §4.3.2 Target selection
-
-候选集：所有 instance 中 (a) 剩余 KV 容量能装下 session 现有 context、(b) `pending_prefill` 严格小于 H 的。选 `pending_prefill` 最低者。
-
-**关键设计点**：我们用 **observable current load** 而不是 **predicted transfer time** 排序。文献和 colleague 数据均显示 mooncake cost model 的预测误差达 10-21x；而 pending prefill tokens 是 router 直接观察到的数值，accuracy by construction。
-
-若候选集为空（所有其他 instance 都装不下，或都比 H 更忙），EAR 保留当前 binding，继续在 H 上处理请求 —— **migration 是 opportunistic，不是 mandatory**。
-
-#### §4.3.3 Mechanism
-
-Migration 触发时：
-
-1. 当前 request 直接重定向到 target instance T
-2. session 累计的 KV 状态从 source H 通过 Mooncake `kv_connector` 传输到 T
-3. session 的 host binding 更新为 T；后续 turn 按 affinity 自动路由到 T
-
-KV transfer 发生在触发该 migration 的 request 的 critical path 上，但被该 session 剩余的 TBD turn 摊销。
-
-#### §4.3.4 Thrashing prevention
-
-每个 session 维护 `last_migration_timestamp`。在 cooldown `T_cool` 内被禁止再次 migrate。Cooldown 把 migration 行为限制在 O(session_lifetime / T_cool) 量级。
-
-### §4.4 Implementation
-
-基于 vLLM 0.18.1 + Mooncake (vanilla kv_connector)。EAR 是一个 router 进程，~TBD LoC。Session→host 表用 TBD（in-memory dict / Redis）维护。
+### §4.5 Sensitivity
+- 到达率 λ / skew(Zipf α) / KV pool size（不依赖 migration，可做）；`T_hot`/`T_cool`（依赖 migration，deferred）。
 
 ---
 
-## §5 Evaluation
+## §5 Related Work
 
-### §5.1 Setup
-
-- **Trace**: 真实 Qwen3-Coder agentic trace，TBD requests / TBD seconds / r=0.0015 st=0.3，peak QPS ~1.6，APC headroom ~76%
-- **Hardware**: TBD × H20 (96GB HBM)
-- **Engine**: vLLM 0.18.1 + Mooncake `kv_connector`
-- **Baselines** (6 个):
-  1. `load-balance` —— round-robin
-  2. `LMetric` —— OSDI'26 load-aware routing
-  3. `kvcache-aware + load-balance` —— linear combination of cache score and load score
-  4. `sticky` —— 硬 session-instance pinning
-  5. `static PD-disagg` —— 4P / 4D 静态分区
-  6. `EAR` —— 本文
-- **Metrics**: TTFT (mean/p50/p90/p99)、TPOT (同上)、E2E、APC、worker TTFT p90 (median + max)、wall-clock vs trace-time
-
-### §5.2 End-to-end Performance
-
-**Figure 6 (headline, p90 only)** — ✅ (PARTIAL，缺 PD-disagg 列)
-
-![F6 E2E latency bars — 4 policies, p90 only](figs/f6_e2e_latency_bars.png)
-
-**Figure 6 full (mean / p50 / p90 / p99 × TTFT / TPOT / E2E)** — ✅ 数据完备：
-
-![F6 full latency grid — 4 percentiles × 3 metrics](figs/f6_e2e_latency_full_grid.png)
-
-> **🚧 TBD (NEW DATA)**：两张图都缺 `static PD-disagg` 那一列；EAR 列也是 TBD（需 migration validation）。要再补同样格式但包含全 6 个 baseline 的版本。Headline 图用 p90 一行进 main paper，完整 grid 可进附录或 supplementary。
-
-| Scheduler | TTFT p50 | TTFT p90 | TPOT p90 | APC | Worker p90 (median / max) | Wall-clock factor |
-|---|---|---|---|---|---|---|
-| load-balance | TBD | TBD | TBD | TBD | TBD | TBD |
-| LMetric | TBD | TBD | TBD | 56.9% | 6.53 | ~8x |
-| kvcache+load | TBD | TBD | TBD | TBD | TBD | TBD |
-| sticky | TBD | 18.02s | TBD | 77.2% | 13.65 | TBD |
-| static PD-disagg | 62.8s | TBD | TBD | TBD | TBD | TBD |
-| **EAR** | TBD | **7.35s** | TBD | **79.4%** | TBD | TBD |
-
-(粗体数字来自现有 "unified" 原型测量。)
-
-### §5.3 Ablation 🚧 PARTIAL DEFER
-
-我们独立关闭两个 pillar:
-
-- **EAR (affinity only)**: 等价于 pure sticky；衡量 locality 单独贡献
-- **EAR (migration only)**: cold-balance + reactive migration，无 affinity；衡量 migration 能否独立成立
-- **EAR (full)**: 两个 pillar 都开
-
-**Figure 7: Ablation** — `figs/f7_ablation.png` **🚧 TBD — DEFERRED (BLOCKED ON MIGRATION VALIDATION)**
-> 完整 ablation 需要 migration-only / both / affinity-only 三个配置。Migration-only 和 both 都依赖 migration 重测。现阶段可先做 affinity-only vs load-balance 的两点对比（已有数据：unified 79.4% APC vs lmetric 56.9% APC）。
-
-预期结论：affinity-only 拿到 locality 但 interference 翻倍；migration-only 抓不住 locality；两者都必须。
-
-### §5.4 Dispatch Coupling Validation
-
-闭环 §2.3 的论证。对每个 baseline 测量：
-
-- 单 turn 平均服务时间 `W_turn`（x 轴）
-- Wall-clock / trace-time amplification（y 轴）
-
-**Figure 8: Wall-clock amplification vs per-turn service time** — `figs/f8_coupling_measured.png` **🚧 TBD (NEW DATA)**
-> 散点：x = 平均 per-turn `W_turn`（从 per-request metrics 算 TTFT + decode_time），y = amplification (`wall_clock / trace_span`，Phase 1 patch 已暴露)。每个 baseline 一个点。理论曲线 `L*/(1 − Λ·N·W'(L*))` 叠加（可选）。这是 §2.3 论证的实证 closure，**优先级最高**。
-
-预期：EAR 在 `W_turn` 最小且放大系数最低的角上。
-
-### §5.5 Sensitivity
-
-| 参数 | 范围 | 检验 |
-|---|---|---|
-| 到达率 λ | TBD | EAR 在低/高负载下是否稳定 dominate |
-| Skew 程度 (Zipf α) | TBD | sticky 与 EAR 的差距是否随 skew 拉开 |
-| KV pool size | TBD | static PD-disagg 撞墙边界 |
-| `T_hot` (migration threshold) | TBD | 触发太宽 → thrash，太严 → 错过 |
-| `T_cool` (cooldown) | TBD | 同上 |
-
-**Figure 9: Sensitivity heatmaps** — `figs/f9_sensitivity.png` **🚧 TBD (NEW DATA, PARTIAL DEFER)**
-> Arrival rate / skew / KV pool size 这三轴可现在做（不依赖 migration）；`T_hot` / `T_cool` 两轴依赖 migration validation，deferred。
-
-### §5.6 Migration Microbenchmark 🚧 FULL DEFER
-
-刻画 EAR 内部 migration 行为：
-
-- Migration 触发率（% of requests）
-- 平均 KV transfer 时间
-- Migration accuracy：迁移后 target instance 在接下来 TBD 个 turn 内保持非 hot 的比例
-- Thrashing rate：cooldown 窗口内多次迁移的 session 占比（应为 0）
-
-**Figure 10: Migration timeline** — `figs/f10_migration_timeline.png` **🚧 TBD — DEFERRED (BLOCKED ON MIGRATION VALIDATION)**
-> 时间轴上每个 instance 的 pending prefill tokens heatmap，migration 事件以箭头标出。完全依赖 migration 重测。
+- **Serving systems**：vLLM, Mooncake, SGLang, DistServe, Splitwise。本文基于 vLLM+Mooncake，与
+  DistServe/Splitwise 不同在于**不做静态 P/D 分区**（§3.1 给出 agentic regime 的证否 + crossover 边界）。
+- **Cache-aware routing**：LMCache, Production-Stack, LMetric (OSDI'26)。本文指出 cache 信号不能折叠进乘性 load
+  score（§3.2 LMetric 稀释分析），须作独立 biased 路由路径。
+- **KV storage hierarchy / offload（主要对手，正面回应）**：Mooncake Store, LMCache, AttentionStore 等把 KV 下沉
+  到 CPU/SSD/远端 pool。**本文不否认其对 recompute 的收益**（§2.2 实测 remote-RDMA 命中比 recompute 快达 16×，
+  与 Mooncake-Store blog 的 46× 同向）；但论证在 **agentic** 下 (i) 命中层级 GPU ≫ CPU ≫ RDMA-store（§2.2），
+  (ii) 活跃 working set 本就 GPU-resident（§2.2 Ev#1）⇒ 深层级的边际收益低，应优先 GPU 常驻而非建深 hierarchy。
+- **Stateful migration**：Pollux, Gandiva（RL training 的 migration-as-rebalancing）。本文把该思路迁到 LLM KV
+  cache 的 dedup（§3.3）。
 
 ---
 
-## §6 Discussion and Limitations
+## §6 Conclusion
 
-- **Extreme skew**: 若单个 session 自己就把任意 instance 撑成 hot，EAR 退化为 sticky。我们未在该 regime 做 stress test。
-- **Cost model accuracy**: EAR 用 observable load 绕过了预测误差问题。但未来若引入 predictive admission control，需要解决 mooncake cost model 10-21x 误差。
-- **Heterogeneous hardware / multi-model**: EAR 假设 instance 同质。混合模型 / 混合 GPU 池需要扩展 binding 模型。
-- **Per-instance batch tuning (future)**: 动态调整 `max_batched_tokens` 可能进一步降低 instance 内部 prefill-decode 干扰，留作 future work。
-
----
-
-## §7 Related Work
-
-- **LLM serving systems**: vLLM, Mooncake, SGLang, DistServe, Splitwise. EAR 基于 vLLM + Mooncake 实现，与 DistServe/Splitwise 不同之处在于不做静态 P/D 分区。
-- **Cache-aware routing**: LMCache, Production-Stack, LMetric (OSDI'26)。这些工作最小化 cross-instance cache miss，但不迁移状态。
-- **Stateful service migration**: Pollux, Gandiva (RL training)。EAR 借鉴 migration-as-rebalancing 思路，将其迁移到 LLM inference 的 KV cache 场景。
-
----
-
-## §8 Conclusion
-
-对 agentic LLM workload，locality 是主导调度杠杆。EAR 用 session-affinity routing 抓住它，用 hot-triggered session migration 保护它，单一方案在 TTFT、APC、worst-worker p90、wall-clock throughput 四个维度同时 dominate 五个 baseline。
+对 agentic LLM 负载，用户感受的 request latency / TPS / GPU util 由 **KV 命中是否在 GPU HBM 上** 主导。命中层级
+`GPU ≫ CPU > RDMA-store ≫ recompute` 且代价差随 context 拉大（§2.2），而活跃 working set 小到本就 GPU-resident
+（§2.2 Ev#1）。**GPU-hit-first** 原则由此统一三件事：复活 PD-colocation（§3.1）、在路由里做 biased
+KV-awareness（§3.2）、用 migration 而非 replication 做 GPU 去重（§3.3）。
 
 ---
 
 ## Work Plan
 
 ### ✅ Done
+- §1/§2 背景与 metric 论证；dispatch-coupling 数学；inter-turn gap CDF（`f3a`）
+- §2.1 reuse topology / C1–C3（`f2a`、`workload_chars/*`）
+- **§2.2 四层命中实测（`v2/exp_a_tier_latency`：GPU/CPU/RDMA/miss）**
+- **§2.2 Ev#1 容量 knee（`v2/exp_b_capacity_knee`）+ working_set + cluster-scale 校正**
+- §3.1 PD-disagg 证否（`PD_DISAGG_RESULTS`、crossover、MB1/MB2）
+- §3.2 LPWL（−31%）、LMetric 稀释、sticky hot-pin、ES ablation
+- §3.3 migration substrate net-positive + correctness smoke tests
 
-- [x] §1 anchor sentence + contribution bullets
-- [x] §2 outline + reuse existing characterization figures (`f2a`/`f2b`/`f2c`)
-- [x] §3.1/§3.2/§3.3 outline + reuse existing baseline failure figures (`f4a`/`f4b`/`f4c`/`f4d`)
-- [x] §4 design description (§4.3 待实证)
-- [x] §5.2 partial figure (`f6` 5/6 baselines)
-- [x] `replayer/replay.py` patched to emit `trace_span_s` + `amplification` in summary
+### 🟢 不依赖 migration，现在可做
+- §2.0 wall-clock amplification sweep（5 baseline × ≥3 runs）— **优先级最高**
+- §4 集成系统命名 + 端到端 baseline 矩阵（含 static PD-disagg 列）
+- §4.5 λ / skew / KV pool sensitivity
+- 草拟 §1–§3 正文（证据/图已齐）
 
-### 🟢 Can do without migration (paper writing now possible)
+### 🚧 Deferred（待 migration policy e2e）
+- §3.3 migration trigger + target 的反馈环收益验证
+- §4.3 full / migration-only ablation
+- §4.5 `T_hot` / `T_cool` sensitivity
 
-- [ ] Draft §1-§4 正文（数据全有，figures 已 copy 完）
-- [ ] §2.3 dispatch coupling 那一节的正文 draft（数学已经在 conversation 里推完）
-- [ ] §3 三个失败模式正文 draft
-- [ ] §5.4 wall-clock amplification 实测（5 baseline × ≥3 runs）— **优先级最高**，这是 §2.3 的实证 closure
-- [ ] §5.2 把 static PD-disagg 补进 `f6` 那张图（重跑或合并现有 PD-sep 数据）
-- [ ] §5.5 sensitivity 的 λ / skew / KV pool 三轴
-- [ ] §3 三张子图各自独立的 latex/markdown layout 决定
+### 🎨 待画
+- §1.3 storage-hierarchy 示意（GPU HBM → CPU DRAM → RDMA store）
+- §2.0 dispatch-coupling schematic（chatbot vs agentic timeline + 反馈环）
+- 集成系统 architecture 图
 
-### 🚧 Deferred (待 migration validation)
-
-- [ ] §4.3 migration mechanism e2e 验证：substrate 已通（commit `ef9e010`），缺 trigger + target selection 的策略层实验
-- [ ] §5.3 full ablation (migration-only + both 两个配置)
-- [ ] §5.5 `T_hot` / `T_cool` 两轴 sensitivity
-- [ ] §5.6 migration microbench 全部
-- [ ] §1 teaser 图 (`f1`) EAR 那一列
-- [ ] §5.2 表里 EAR 那一行
-- [ ] §4.3.1 / §4.3.4 的 `T_hot` 和 `T_cool` 取值
-
-### 🎨 Custom drawings (paper-writing 阶段)
-
-- [ ] `f3_coupling_schematic.png` —— chatbot vs agentic timeline + 反馈环
-- [ ] `f5_architecture.png` —— EAR 组件图
-
-### ❓ Open design decisions
-
-- [ ] §4.4 session→host 表的存储介质（in-memory dict vs Redis）
-- [ ] §5.1 instance 数量、trace 总长度的最终定稿
+### ❓ Open
+- 集成系统最终命名（GPU-hit-first 是原则；系统名待定）
+- §4 instance 数 / trace 总长定稿

README.md

@@ -0,0 +1,77 @@
+# agentic-kv
+
+Serving agentic LLM workloads by keeping the KV working set in GPU HBM
+(GPU-hit-first). Research outline: [`PAPER_OUTLINE.md`](PAPER_OUTLINE.md).
+Evidence + experiments: [`v2/`](v2/).
+
+---
+
+## ⚠️ Benchmarking methodology — read this first
+
+> **Replay agentic traces with `--dispatch-mode thinktime`, not the default
+> `tracets`.** It is the faithful, more realistic load — and the dispatch mode
+> materially changes the performance you measure.
+
+The replayer offers two ways to time each turn:
+
+| mode | turn-k dispatched at | what it models |
+|---|---|---|
+| `tracets` (default) | `max(prev_turn_finished, trace_ts)` | absolute production schedule |
+| **`thinktime` (use this)** | `prev_turn_finished + time_to_parent_chat` | real closed-loop agent pacing |
+
+**Why it matters.** `tracets` collapses the inter-turn think-time to ~0 whenever
+the system falls behind (it fires the next turn immediately because the trace
+timestamp is already in the past). That manufactures **artificial request
+bursts** — spiking instantaneous concurrency → KV-pool pressure → preemption →
+inflated tail latency and wasted throughput. `thinktime` keeps each turn's real
+gap (tool-exec + agent think), so the offered load is what a real agent produces.
+
+**Measured (w600 first-300s window, 8×H20, round-robin, 100% completion):**
+
+| metric (N=8) | `tracets` (Mode 1) | **`thinktime` (Mode 2)** | Δ |
+|---|---:|---:|---:|
+| E2E p90 | 102.8 s | **73.5 s** | **−28%** |
+| E2E p99 | 245 s | **227 s** | −7% |
+| TTFT p90 | 56.1 s | **39.7 s** | **−29%** |
+| system TPS | 111.8 | **119.3** | **+7%** |
+| wall-clock | 967 s | **787 s** | −19% |
+| TPOT p90 | 0.174 s | 0.188 s | ~flat |
+
+So under realistic capacity, `tracets` makes the system look **~30% worse on
+tail latency** than it actually is. Tell-tale: scaling 6→8 instances barely helped
+`tracets` (975→967 s — its bursts re-saturate regardless of capacity) but helped
+`thinktime` a lot (1125→787 s). Under heavy saturation (N=6) the two converge
+(E2E p90 ≈ 118–120 s), since there is no slack for bursts to harm. Decode (TPOT)
+is dispatch-independent everywhere.
+
+**Recommendation:** benchmark with `--dispatch-mode thinktime`; use `tracets`
+only as an explicit bursty stress case. Full ablation:
+[`v2/exp_c_dispatch_ablation/`](v2/exp_c_dispatch_ablation/).
+
+### How to use it
+
+```bash
+# 1. annotate a trace with the real per-turn gap (one-time; scans the raw trace)
+python scripts/add_time_to_parent.py traces/w600_r0.0015_st30.jsonl traces/w600_ttp.jsonl
+
+# 2. replay closed-loop with faithful think-time
+python -m replayer --trace traces/w600_ttp.jsonl --endpoint <eps> \
+    --model <model> --dispatch-mode thinktime
+```
+
+`time_to_parent_chat = this_turn.request_ready_time_ms − parent_turn.request_end_time_ms`,
+computed from the raw trace and stored per request; turn-1 has none (fires at its
+trace arrival). Traces without the field fall back to `tracets`.
+
+---
+
+## Project map
+
+- [`PAPER_OUTLINE.md`](PAPER_OUTLINE.md) — GPU-hit-first paper outline (the thesis).
+- [`v2/`](v2/) — evidence experiments:
+  - `exp_a_tier_latency/` — KV-hit cost by tier (GPU < CPU-local < remote-RDMA < miss).
+  - `exp_b_capacity_knee/` — realized APC / latency knee vs GPU capacity.
+  - `exp_c_dispatch_ablation/` — the replay-mode study above.
+- `replayer/` — trace replayer (`--dispatch-mode`, closed-loop think-time).
+- `scripts/add_time_to_parent.py` — trace annotation for `thinktime`.
+- `microbench/`, `analysis/` — PD-disagg, routing, workload characterization.

RESULTS_SUMMARY.md

@@ -39,52 +39,75 @@ Production trace = Qwen3-Coder agentic，1.3 M sessions / 2.1 M reqs / 7200 s。
 
 参考图：`figs/f4a_apc_loss.png`、`figs/f4b_pdsep_kv_wall.png`、`figs/f4c_per_worker_ttft.png`、`figs/f6_e2e_latency_bars.png`、`figs/f6_e2e_latency_full_grid.png`。
 
-## 4. PD-disagg 在 agentic 下输——cost vs benefit（§3.2）
+## 4. Static PD-disagg 为什么失败（§3.2）—— 容量问题，不是 cost-benefit 问题
 
-由两个独立 microbench 钉死（**全用 vanilla vLLM 0.18.1 + Mooncake 0.3.11，fresh venv，无 patch**）。
+⚠ **2026-05-27 纠正**：本节前一版本论证"PD-disagg 因为 transfer cost > phase isolation benefit 而失败"。**这个论证算错了**。正确的 phase-isolation benefit 是**每个 prefill 事件 × D 个 concurrent stream** 的总和（≈ `D × T_prefill`），不是单个 request 的 decode 时长。用正确公式，PD-disagg 在 phase-isolation 这一维上**赢 colo 一两个数量级**。Static PD-disagg 在 agentic 上失败的**真正根因是 D 侧 KV pool 容量**，不是这一维。
 
-### 4.1 MB2 — KV transfer cost
+### 4.1 真正的失败模式：D 侧 KV 容量天花板
 
-dash1 GPU 0+1（intra-node）和 dash1 ↔ dash2（inter-node, 200 Gbps RoCE）扫 9 个 size × 5 reps。
+| | 8C colo | 4P+4D PD-disagg |
+|---|---:|---:|
+| Per-D-instance KV pool（0.4 × 96 GiB） | 38 GiB | 38 GiB |
+| 系统总 decode 容量（D 实例数 × 单池） | 8 × 38 = **304 GiB** | 4 × 38 = **152 GiB** |
+| p99 单请求 KV = 11.5 GiB → 最多并发 decode | 24 | **12（减半）** |
 
-| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求（11.5 GiB）transfer 时间 |
+Colleague 4P+4D 实测：TTFT p50 0.91 s → **62.8 s（62×）**、success rate **99.5% → 52%**。失败模式：**D 池溢出 + 排队**，不是 transfer 延迟。
+
+参考图：`figs/f4b_pdsep_kv_wall.png`（pdf 版本是高质量 paper figure）。
+
+### 4.2 MB2 — KV transfer cost（per-request 一次性成本，**不**是 dominant cost）
+
+dash1 GPU 0+1（intra）和 dash1 ↔ dash2（inter, 200 Gbps RoCE）扫 9 个 size × 5 reps。
+
+| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求 11.5 GiB transfer |
 |---|---|---|
-| Intra-node | **9.7 GB/s** | p50 **1.9 s** · min 1.5 s · max 10 s |
-| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · min 1.3 s · max 9.2 s |
+| Intra-node | **9.7 GB/s** | p50 **1.9 s** · max 10 s |
+| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · max 9.2 s |
 
-**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC，包括 intra-node loopback**，不走 NVLink。200 Gbps NIC 是天花板，**PD-disagg 的 transfer cost 与拓扑无关**。
+**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC**，不走 NVLink。200 Gbps NIC 是天花板。**PD-disagg transfer cost 与拓扑无关**。
 
-参考图：`figs/mb2_transfer_time_compare.png`、`figs/mb2_transfer_bw_compare.png`、doc `analysis/mb2/README.md`。
+参考图：`figs/mb2_transfer_time_compare.png`、doc `analysis/mb2/README.md`。
 
-### 4.2 MB1 — Phase interference（chunked-prefill on, 默认 baseline）
+### 4.3 MB1 — Phase interference（PD-disagg 的潜在 benefit 上界）
 
-dash1 GPU 0 单 instance，D（concurrent decodes）× P（prefill size）扫描。
+dash1 GPU 0 单 instance（无 kv_connector），chunked-prefill 默认开启，D × P 扫描。D=8 结果：
 
-D=8（最 agentic-realistic）的结果：
-
-| Prefill | prefill_ttft | per-stream TPOT during | penalty |
+| Prefill | T_prefill | per-stream TPOT during | penalty |
 |---|---:|---:|---:|
 | 2k tok | 143 ms | 32 ms | 4× |
-| 8k | 583 ms | 114 ms | 15× |
-| 32k | 4.5 s | 388 ms | **52×** |
-| 65k | 15.6 s | 757 ms | **99×** |
-| 131k | 57 s | 1419 ms | **183×** |
+| 32k tok | 4.5 s | 388 ms | **52×** |
+| 131k tok | 57 s | 1419 ms | **183×** |
 
-baseline TPOT 7.7 ms。**Decode 在大 prefill 期间基本被 halted**。chunked-prefill 已经默认开启，PD-disagg 在它之上能额外提供的 phase isolation = **decode 在 prefill 期间被 halted 的那部分时间**。
+**Decode 在 prefill 期间被几乎完全 halted**，单 stream 损失 ≈ `T_prefill` 的时间。**每个 prefill event 总 decode 损失 ≈ `D × T_prefill`**。
 
 参考图：`figs/mb1_interference.png`、doc `analysis/mb1/README.md`。
 
-### 4.3 联合结论
+### 4.4 联合 cost-benefit（per-prefill event）
 
-| | Per-request |
-|---|---|
-| **Max PD-disagg benefit**（救回来的 decode 时间）| ≤ **decode 时长 = 50–200 ms**（agentic tool-call output）|
-| **PD-disagg cost**（MB2 transfer p50）| 80 MiB ≈ 8 ms · 3 GiB ≈ 320 ms · 11.5 GiB ≈ **1.9 s**（p99 实测最差 10 s）|
-| Cost / Benefit | **每个 KV ≥ 80 MiB 的请求都输**；trace 平均 KV 192 MiB → 已经输 |
+| Prefill (KV size) | T_prefill | Cost = T_transfer | Benefit = D × T_prefill (D=8) | Cost / Benefit |
+|---:|---:|---:|---:|---:|
+| 2k tok (192 MiB) | 0.14 s | 8 ms | 1.1 s | **0.7%** |
+| 33k tok (3 GiB, trace mean) | 4.5 s | 0.32 s | 36 s | **0.9%** |
+| 125k tok (12 GiB, ~p99) | 57 s | 1.9 s | 456 s | **0.4%** |
 
-**结论**：在 agentic 上 **PD-disaggregation 是结构性失败的**。Chunked-prefill 默认已经在 colocation 内做了 first-order phase isolation；PD-disagg 在此之上能额外补的（decode 短时段没被 prefill 挤）小于它新带来的（每个 routed 请求都付 KV transfer）。这个结论与拓扑无关（intra-node 和 inter-node 一样）。
+**PD-disagg 在 phase-isolation 这一维赢 100×–250×**。但**这不是 §3.2 该用的论证**，因为 §3.2 真正的 dominant failure 是 §4.1 的 D 池容量天花板（颠覆了上面的全部数学）。
 
-参考图：`figs/pd_cost_vs_benefit.png`（§3.2 headline）。
+**总结**：
+- D 侧 KV 容量天花板（§4.1）→ PD-disagg 在 agentic 上**结构性失败**。
+- MB1 + MB2 的合计 cost-benefit 在 phase isolation 维度上 PD-disagg 是赢的，**但这件事被容量天花板压倒**。
+- Paper §3.2 论证应该聚焦"D 池装不下"，MB1/MB2 数据用作 supporting context（per-request transfer charge 量化、phase isolation benefit 量化）而不是 main argument。
+
+> ✅ **2026-05-30 更新 — 干净栈三轴 ablation 证实本节、并加 regime 细化。**
+> 本节的容量论点（D 池容量天花板 / decode 减半）在修复 `e13391e` 污染后的 clean stack
+> 上**得到确认**：concurrency 轴 N=64 时 PD 倾覆，**producer APC 从 71% 崩到 1.4%、TPS −30%**，
+> 而 colo 线性 scale（Fig 3）。**细化**：PD 并非"在 agentic 上一律失败"——它在
+> *低负载 / decode-heavy / 低复用* 区间**赢** colo；真正失败的是 agentic corner（高复用 +
+> 短输出 + 大上下文 + 高并发）——静态 P:D split 无法同时给出复用所需的 producer 容量
+> *和* decode 容量，而 colo 的弹性池两者兼得。
+> **另注**：旧 MB5 文档（`PD_DISAGG_RESULTS.md` §6）"session-affinity 救不了 PD / PD 复用=0%"
+> 的结论是 `e13391e`（producer 每次 KV 传输后 evict prefix）的**污染产物，已撤回**；
+> 干净栈上 session 路由的 producer APC 与 colo 持平（71–82%）。
+> 图：[`figs/mb5_pd_ablation/`](figs/mb5_pd_ablation/)。
 
 ## 5. EAR 设计的实证状态（§4）
 
@@ -96,10 +119,12 @@ baseline TPOT 7.7 ms。**Decode 在大 prefill 期间基本被 halted**。chunke
 ## 6. 已经能写的 paper 主张（按 confidence 排序）
 
 1. **Agentic vs chatbot 在调度上是不同 regime**（dispatch coupling + sub-second tool-call mass）—— 实证完整
-2. **PD-disaggregation 在 agentic 下输**（cost > benefit，跨拓扑）—— **MB1 + MB2 实证完整**
-3. **三类现有调度 baseline 各自的失败模式** —— 实证完整
-4. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整
-5. **Hot-triggered migration 修复 sticky 的 hot pin** —— **design 完整、e2e 待验证**
+2. **三类现有调度 baseline 各自的失败模式**（load-balance / static PD-disagg / pure sticky）—— 实证完整
+3. **Static PD-disagg 在 agentic 下失败的 dominant 根因是 D 侧 KV 容量**（不是 phase-isolation cost-benefit）—— 实证完整（`f4b` + colleague 4P+4D 数据）
+4. **Mooncake transfer cost 拓扑无关**（intra ≈ inter，~9.7 GB/s NIC 上限）—— 实证完整（MB2）
+5. **Phase isolation interference 在 chunked-prefill on 下仍然显著**（per-stream TPOT during prefill 15×–2000× baseline）—— 实证完整（MB1）。**注意**：这条数据本身不直接论证 "PD-disagg 失败"，因为算正确账后 PD-disagg 反而在这一维上赢；它的用途是给 §3.2 提供 phase-isolation benefit 上界的量化。
+6. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整
+7. **Hot-triggered migration 修复 sticky 的 hot pin** —— design 完整、e2e 待验证
 
 ## 7. 待做
 

analysis/crossover/d1_i1024_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.1385939740576,
+      "amplification": 1.0013425037685975,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.08144469193556772,
+        "p50": 0.07862715201918036,
+        "p90": 0.08015060934703797,
+        "p99": 0.0875979653932154
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005001699398049616,
+        "p50": 0.004988961030788246,
+        "p90": 0.005045765990923557,
+        "p99": 0.005062779263327164
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.3968209869372152,
+        "p50": 0.393534954986535,
+        "p90": 0.39730903680901974,
+        "p99": 0.40925762055674536
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.1604231200181,
+      "amplification": 1.0014153316291596,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.09946277569807849,
+        "p50": 0.09600010397844017,
+        "p90": 0.10452785079833121,
+        "p99": 0.11205230774357905
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005007447102661814,
+        "p50": 0.004987124730611131,
+        "p90": 0.005003212126977151,
+        "p99": 0.005478902989961502
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.415208436744531,
+        "p50": 0.41056320699863136,
+        "p90": 0.4200975856045261,
+        "p99": 0.44871115096379066
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.2032543020323,
+      "amplification": 1.0015582278907484,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.10561635944505095,
+        "p50": 0.10468761203810573,
+        "p90": 0.11257308297790587,
+        "p99": 0.12065987563692024
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005328901365752947,
+        "p50": 0.005144592110318915,
+        "p90": 0.005990574603515958,
+        "p99": 0.006688486758013448
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.4416300980896939,
+        "p50": 0.42991435900330544,
+        "p90": 0.4854830394731835,
+        "p99": 0.5404117306252005
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d1_i16384_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 303.05708941399644,
+      "amplification": 1.0110793839624128,
+      "ttft": {
+        "count": 1167,
+        "mean": 1.674437926279444,
+        "p50": 1.5353219069947954,
+        "p90": 2.0787689138029237,
+        "p99": 3.039117059087727
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.035498316425319934,
+        "p50": 0.02951206674587743,
+        "p90": 0.085249871320677,
+        "p99": 0.15422643764865662
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 3.9111985703531236,
+        "p50": 3.392241114997887,
+        "p90": 7.760864628604043,
+        "p99": 11.30427318874542
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 1136,
+      "completion_rate": 0.9734361610968295,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 866.9596620649972,
+      "amplification": 2.8924089607535697,
+      "ttft": {
+        "count": 1136,
+        "mean": 65.09021699308856,
+        "p50": 66.22900710900285,
+        "p90": 112.5535424454938,
+        "p99": 124.55262411334482
+      },
+      "tpot": {
+        "count": 1136,
+        "mean": 0.005710658520121912,
+        "p50": 0.005725543936557461,
+        "p90": 0.005750613698356098,
+        "p99": 0.0058447879207267845
+      },
+      "e2e": {
+        "count": 1136,
+        "mean": 65.4504681098121,
+        "p50": 66.59053339700768,
+        "p90": 112.9150809329949,
+        "p99": 124.91415351489852
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 307.42712411200046,
+      "amplification": 1.0256589867656205,
+      "ttft": {
+        "count": 1167,
+        "mean": 3.6233640342625417,
+        "p50": 3.255483777000336,
+        "p90": 6.0935156565916255,
+        "p99": 7.349482456580735
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.006360297526341433,
+        "p50": 0.006324973206372104,
+        "p90": 0.007198417158741947,
+        "p99": 0.007942749238420567
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 4.024414356593621,
+        "p50": 3.6796783979953034,
+        "p90": 6.510242249601289,
+        "p99": 7.7530036393977895
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.6563838903170522,
+          "pd_advantage": 1.0,
+          "n_slo": 766
+        },
+        "4P+4D": {
+          "attainment": 0.0,
+          "pd_advantage": 0.0,
+          "n_slo": 0
+        },
+        "6P+2D": {
+          "attainment": 0.20565552699228792,
+          "pd_advantage": 0.3133159268929504,
+          "n_slo": 240
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.7095115681233933,
+          "pd_advantage": 1.0,
+          "n_slo": 828
+        },
+        "4P+4D": {
+          "attainment": 0.012853470437017995,
+          "pd_advantage": 0.018115942028985508,
+          "n_slo": 15
+        },
+        "6P+2D": {
+          "attainment": 0.7746358183376179,
+          "pd_advantage": 1.0917874396135265,
+          "n_slo": 904
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d1_i2048_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.1927249849541,
+      "amplification": 1.0015230992770423,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.1367582682356152,
+        "p50": 0.1337889520218596,
+        "p90": 0.13568343156948687,
+        "p99": 0.1417779766954481
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005039774583944486,
+        "p50": 0.005025791809968059,
+        "p90": 0.005088228446667984,
+        "p99": 0.0051380019550329985
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.45452996390628697,
+        "p50": 0.45110413199290633,
+        "p90": 0.4555703255813569,
+        "p99": 0.4688122991356063
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.2166367470054,
+      "amplification": 1.0016028753010615,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.1660798433322169,
+        "p50": 0.16653224604669958,
+        "p90": 0.17306958099361508,
+        "p99": 0.1832630861806683
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005046875948133808,
+        "p50": 0.0050270736812510425,
+        "p90": 0.0050664691831029595,
+        "p99": 0.005532995437730161
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.48433690918382055,
+        "p50": 0.4845748710213229,
+        "p90": 0.4916891472181305,
+        "p99": 0.5235905526624989
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.2855537990108,
+      "amplification": 1.001832800991369,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.17076528707394278,
+        "p50": 0.16959826194215566,
+        "p90": 0.17923957279417663,
+        "p99": 0.18792458378709853
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005377765550956012,
+        "p50": 0.005177637602808693,
+        "p90": 0.006078562941697855,
+        "p99": 0.007001848420689976
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.5098518045703324,
+        "p50": 0.4961305959150195,
+        "p90": 0.5569328068522736,
+        "p99": 0.6314893902023319
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d1_i32768_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 951,
+      "completion_rate": 0.8149100257069408,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 567.1572082909988,
+      "amplification": 1.8921879104611408,
+      "ttft": {
+        "count": 951,
+        "mean": 156.08624654169503,
+        "p50": 159.80228465000982,
+        "p90": 263.44082942500245,
+        "p99": 267.3189407904938
+      },
+      "tpot": {
+        "count": 951,
+        "mean": 0.5120599246480225,
+        "p50": 0.5265210192539223,
+        "p90": 0.5403215261268621,
+        "p99": 0.5436686666190138
+      },
+      "e2e": {
+        "count": 951,
+        "mean": 188.3470215559188,
+        "p50": 192.9057134770119,
+        "p90": 294.68771972299146,
+        "p99": 300.2205298934932
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 596,
+      "completion_rate": 0.5107112253641817,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 789.8321366070013,
+      "amplification": 2.63509093833936,
+      "ttft": {
+        "count": 596,
+        "mean": 240.99040966441748,
+        "p50": 259.1066791820049,
+        "p90": 379.7789027979961,
+        "p99": 381.22047349565617
+      },
+      "tpot": {
+        "count": 596,
+        "mean": 0.006412497327210905,
+        "p50": 0.006444117785865692,
+        "p90": 0.0064974606745333095,
+        "p99": 0.006660929446938789
+      },
+      "e2e": {
+        "count": 596,
+        "mean": 241.39516468475196,
+        "p50": 259.5132920899996,
+        "p90": 380.1822811405,
+        "p99": 381.620277955458
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 712,
+      "completion_rate": 0.6101113967437874,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 880.7267912379903,
+      "amplification": 2.938339780290167,
+      "ttft": {
+        "count": 712,
+        "mean": 175.69983718378106,
+        "p50": 172.43730882649834,
+        "p90": 250.6456608123961,
+        "p99": 594.014642834873
+      },
+      "tpot": {
+        "count": 712,
+        "mean": 0.007946936024875445,
+        "p50": 0.007568808706382119,
+        "p90": 0.010293392149296345,
+        "p99": 0.011275891952114068
+      },
+      "e2e": {
+        "count": 712,
+        "mean": 176.20114515147316,
+        "p50": 172.91078094949626,
+        "p90": 251.09661155799986,
+        "p99": 594.4257701966026
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        },
+        "4P+4D": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        },
+        "6P+2D": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        },
+        "4P+4D": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        },
+        "6P+2D": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d1_i4096_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.32634208898526,
+      "amplification": 1.0019688816195438,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.2628433253752162,
+        "p50": 0.2604120689211413,
+        "p90": 0.2637787788407877,
+        "p99": 0.2976114549371393
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005143964614959416,
+        "p50": 0.005124552334318795,
+        "p90": 0.005180289564166395,
+        "p99": 0.0052467077378080115
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.5871811069348818,
+        "p50": 0.5839827890740708,
+        "p90": 0.5883447280153632,
+        "p99": 0.6408826243295329
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.3759389320621,
+      "amplification": 1.0021343499332585,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.30861241175426407,
+        "p50": 0.3091614870354533,
+        "p90": 0.3227507559815422,
+        "p99": 0.3547600568598136
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005148936446188915,
+        "p50": 0.0051316127147791645,
+        "p90": 0.005184003428393413,
+        "p99": 0.00547781713853871
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.6332931331915089,
+        "p50": 0.6365591660141945,
+        "p90": 0.6472898541716858,
+        "p99": 0.7053035433730102
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.4041277950164,
+      "amplification": 1.0022283955081221,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.31527804771431056,
+        "p50": 0.31678270106203854,
+        "p90": 0.3284846659982577,
+        "p99": 0.3402980738645419
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.0055213440500518985,
+        "p50": 0.005325577683776381,
+        "p90": 0.006278165857056304,
+        "p99": 0.0070092806818761975
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.6634175265073216,
+        "p50": 0.6494167000055313,
+        "p90": 0.7135140666039661,
+        "p99": 0.7741130576422437
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d1_i8192_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.66455180699995,
+      "amplification": 1.0030972395602922,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.584800394000121,
+        "p50": 0.5817365200055065,
+        "p90": 0.5870239008043427,
+        "p99": 0.6088061455392736
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005585180932058873,
+        "p50": 0.005308644777941813,
+        "p90": 0.005355837698406836,
+        "p99": 0.014590345752427337
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.9369382756572071,
+        "p50": 0.9158835929993074,
+        "p90": 0.9229304600012256,
+        "p99": 1.5052119748995754
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.7607197400066,
+      "amplification": 1.003418081467173,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.7293765576349805,
+        "p50": 0.6887177099997643,
+        "p90": 0.8911576298065484,
+        "p99": 1.1879012519156076
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005308951736224922,
+        "p50": 0.0053098901269203495,
+        "p90": 0.005347346425488857,
+        "p99": 0.005420954272433716
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 1.064155324243521,
+        "p50": 1.0238807429996086,
+        "p90": 1.22566919879755,
+        "p99": 1.521622942700049
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.8507765819959,
+      "amplification": 1.003718535142407,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.7028102769056583,
+        "p50": 0.6950660040020011,
+        "p90": 0.7394948218076024,
+        "p99": 0.8658222487580485
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005782890183689697,
+        "p50": 0.00558671395230617,
+        "p90": 0.006629064533303286,
+        "p99": 0.007366264003757681
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 1.0674261210719522,
+        "p50": 1.0612060459970962,
+        "p90": 1.1425091800017981,
+        "p99": 1.3045775910036173
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o1024_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 1950,
+      "completion_rate": 0.5489864864864865,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 348.4953830049999,
+      "amplification": 1.1626534102882857,
+      "ttft": {
+        "count": 1950,
+        "mean": 0.8245253885559363,
+        "p50": 0.7988759850013594,
+        "p90": 1.5384256363991882,
+        "p99": 2.125257717882705
+      },
+      "tpot": {
+        "count": 1950,
+        "mean": 0.07444567704430907,
+        "p50": 0.08914986800879843,
+        "p90": 0.09127071481447005,
+        "p99": 0.09233926755616054
+      },
+      "e2e": {
+        "count": 1950,
+        "mean": 77.04034660455785,
+        "p50": 92.29079191399796,
+        "p90": 94.47993849659906,
+        "p99": 95.72745552870882
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 2754,
+      "completion_rate": 0.7753378378378378,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 744.3694221920014,
+      "amplification": 2.4833719166185753,
+      "ttft": {
+        "count": 2754,
+        "mean": 4.5006646864741695,
+        "p50": 2.329877773499902,
+        "p90": 10.863291689799375,
+        "p99": 21.572781211729307
+      },
+      "tpot": {
+        "count": 2754,
+        "mean": 0.046089308743682535,
+        "p50": 0.04704797274047173,
+        "p90": 0.04792202048768524,
+        "p99": 0.059544689888886115
+      },
+      "e2e": {
+        "count": 2754,
+        "mean": 51.68415100813728,
+        "p50": 50.91020956099965,
+        "p90": 59.22212425199977,
+        "p99": 68.42946432940865
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 1928,
+      "completion_rate": 0.5427927927927928,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 821.7403777359868,
+      "amplification": 2.7414975897474716,
+      "ttft": {
+        "count": 1928,
+        "mean": 39.07585104131927,
+        "p50": 42.6814165695032,
+        "p90": 63.28579387369681,
+        "p99": 73.84470144698193
+      },
+      "tpot": {
+        "count": 1928,
+        "mean": 0.04064862157373329,
+        "p50": 0.0398688508558163,
+        "p90": 0.04277554483881431,
+        "p99": 0.0695667276081441
+      },
+      "e2e": {
+        "count": 1928,
+        "mean": 80.66499786808757,
+        "p50": 84.05398454950046,
+        "p90": 105.42814423799427,
+        "p99": 113.52591980495636
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.10191441441441441,
+          "pd_advantage": 1.0,
+          "n_slo": 362
+        },
+        "4P+4D": {
+          "attainment": 0.3502252252252252,
+          "pd_advantage": 3.43646408839779,
+          "n_slo": 1244
+        },
+        "6P+2D": {
+          "attainment": 0.06447072072072071,
+          "pd_advantage": 0.6325966850828729,
+          "n_slo": 229
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o128_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 300.61600343801547,
+      "amplification": 1.0029177964157683,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.139721406387908,
+        "p50": 0.13759525847854093,
+        "p90": 0.14188919204752892,
+        "p99": 0.17052793529117438
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.006496044329480822,
+        "p50": 0.006430442302898453,
+        "p90": 0.007889476366154117,
+        "p99": 0.009282101713493095
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 0.9651016481538636,
+        "p50": 0.9548319909954444,
+        "p90": 1.1440699087572284,
+        "p99": 1.3324073514551853
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 300.7410873459885,
+      "amplification": 1.0033351024670651,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.18467717330343286,
+        "p50": 0.1806962049449794,
+        "p90": 0.1929074571467936,
+        "p99": 0.2592483600333798
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.006980159784640828,
+        "p50": 0.006945047342387123,
+        "p90": 0.008164690015837552,
+        "p99": 0.009163911323187739
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 1.0715517728354285,
+        "p50": 1.063076548918616,
+        "p90": 1.2275727374362762,
+        "p99": 1.4021264237130526
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 301.1243950970238,
+      "amplification": 1.004613897210595,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.20078434001432569,
+        "p50": 0.19958186394069344,
+        "p90": 0.213167638995219,
+        "p99": 0.23225504373782313
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.010393430778126047,
+        "p50": 0.010396533917478015,
+        "p90": 0.012065167099743436,
+        "p99": 0.01323438493101344
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 1.5212831554822883,
+        "p50": 1.5218885459471494,
+        "p90": 1.7440477049094625,
+        "p99": 1.9191367691196497
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o2048_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 1050,
+      "completion_rate": 0.2956081081081081,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 414.92026689198974,
+      "amplification": 1.3842607013613617,
+      "ttft": {
+        "count": 1050,
+        "mean": 0.8393797435154972,
+        "p50": 0.8442945880015031,
+        "p90": 1.4299255040969001,
+        "p99": 2.1686189359705894
+      },
+      "tpot": {
+        "count": 1050,
+        "mean": 0.08418485046407832,
+        "p50": 0.08937374470713316,
+        "p90": 0.09085719287396188,
+        "p99": 0.09190563933350096
+      },
+      "e2e": {
+        "count": 1050,
+        "mean": 173.22149556513375,
+        "p50": 183.9983977669981,
+        "p90": 187.03014375649946,
+        "p99": 189.36978534828302
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 1320,
+      "completion_rate": 0.3716216216216216,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 802.1317602040072,
+      "amplification": 2.6760791447511205,
+      "ttft": {
+        "count": 1320,
+        "mean": 19.92388417242484,
+        "p50": 8.059029546493548,
+        "p90": 64.71132232169766,
+        "p99": 87.07121913193099
+      },
+      "tpot": {
+        "count": 1320,
+        "mean": 0.05007250707090057,
+        "p50": 0.0492711045202749,
+        "p90": 0.07142825628519621,
+        "p99": 0.08267229383702501
+      },
+      "e2e": {
+        "count": 1320,
+        "mean": 122.44309290737698,
+        "p50": 118.35192646600626,
+        "p90": 167.89536100080443,
+        "p99": 187.19021692074878
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 1041,
+      "completion_rate": 0.29307432432432434,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 879.8981197069952,
+      "amplification": 2.935523968100636,
+      "ttft": {
+        "count": 1041,
+        "mean": 86.06142147925078,
+        "p50": 92.45557322700915,
+        "p90": 153.47880471601093,
+        "p99": 159.16995789420034
+      },
+      "tpot": {
+        "count": 1041,
+        "mean": 0.04275836784091199,
+        "p50": 0.038143618996089604,
+        "p90": 0.06297858072350163,
+        "p99": 0.07100282797010407
+      },
+      "e2e": {
+        "count": 1041,
+        "mean": 173.5925440399473,
+        "p50": 199.81516771799943,
+        "p90": 230.71809448600106,
+        "p99": 234.9438766648003
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.009009009009009009,
+          "pd_advantage": 1.0,
+          "n_slo": 32
+        },
+        "4P+4D": {
+          "attainment": 0.11486486486486487,
+          "pd_advantage": 12.75,
+          "n_slo": 408
+        },
+        "6P+2D": {
+          "attainment": 0.04954954954954955,
+          "pd_advantage": 5.5,
+          "n_slo": 176
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o256_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 301.78380814404227,
+      "amplification": 1.0068138369093473,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.14506312318632342,
+        "p50": 0.14291562204016373,
+        "p90": 0.1491016250802204,
+        "p99": 0.1832571337954128
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.0089611879488046,
+        "p50": 0.008925844482420122,
+        "p90": 0.010698060123752072,
+        "p99": 0.011754058378410762
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 2.430673889741114,
+        "p50": 2.419925262918696,
+        "p90": 2.874893436266575,
+        "p99": 3.1480205075151746
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 3551,
+      "completion_rate": 0.9997184684684685,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 744.8868488909211,
+      "amplification": 2.4850981601942745,
+      "ttft": {
+        "count": 3551,
+        "mean": 0.20500909556101396,
+        "p50": 0.20017725799698383,
+        "p90": 0.21613375598099083,
+        "p99": 0.29023518750909716
+      },
+      "tpot": {
+        "count": 3551,
+        "mean": 0.010307210080394247,
+        "p50": 0.010299421709450874,
+        "p90": 0.011793581666607482,
+        "p99": 0.012639003840969035
+      },
+      "e2e": {
+        "count": 3551,
+        "mean": 2.8339171693501317,
+        "p50": 2.8360355379991233,
+        "p90": 3.2198793930001557,
+        "p99": 3.438588996999897
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 303.4300506779691,
+      "amplification": 1.0123060459587976,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.2333820717117756,
+        "p50": 0.23283391550648957,
+        "p90": 0.24394672318594532,
+        "p99": 0.2734717815916518
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.016453822599812207,
+        "p50": 0.016576926972415737,
+        "p90": 0.017214638463623238,
+        "p99": 0.01769411424845092
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 4.430368151474328,
+        "p50": 4.463736370031256,
+        "p90": 4.628522685484495,
+        "p99": 4.75812664218829
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        },
+        "4P+4D": {
+          "attainment": 0.9997184684684685,
+          "pd_advantage": 0.9997184684684685,
+          "n_slo": 3551
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o4096_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 545,
+      "completion_rate": 0.15343468468468469,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 505.4139580530027,
+      "amplification": 1.6861665622961597,
+      "ttft": {
+        "count": 545,
+        "mean": 4.137136383750316,
+        "p50": 0.5702517530007754,
+        "p90": 1.6976309113961179,
+        "p99": 56.1814190310361
+      },
+      "tpot": {
+        "count": 545,
+        "mean": 0.08390926873267023,
+        "p50": 0.08933810225128375,
+        "p90": 0.09449654152820693,
+        "p99": 0.10591519025133134
+      },
+      "e2e": {
+        "count": 545,
+        "mean": 347.7765141811284,
+        "p50": 366.3568219640001,
+        "p90": 388.1241692415934,
+        "p99": 435.2848098113155
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 771,
+      "completion_rate": 0.2170608108108108,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 850.1328976760124,
+      "amplification": 2.8362209684341706,
+      "ttft": {
+        "count": 771,
+        "mean": 63.210636654403324,
+        "p50": 1.1097561890055658,
+        "p90": 179.64081536799495,
+        "p99": 186.67013871119852
+      },
+      "tpot": {
+        "count": 771,
+        "mean": 0.05484690890217986,
+        "p50": 0.04795774376874187,
+        "p90": 0.08067800507863548,
+        "p99": 0.09099416300415113
+      },
+      "e2e": {
+        "count": 771,
+        "mean": 287.81779562259663,
+        "p50": 312.2160719559906,
+        "p90": 372.0004520520015,
+        "p99": 379.3279856524052
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 627,
+      "completion_rate": 0.17652027027027026,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 867.8833550199925,
+      "amplification": 2.8954402028102515,
+      "ttft": {
+        "count": 627,
+        "mean": 179.58769048342904,
+        "p50": 238.1998468660022,
+        "p90": 378.29023678940143,
+        "p99": 385.40577973942356
+      },
+      "tpot": {
+        "count": 627,
+        "mean": 0.04188420961205498,
+        "p50": 0.03654626756630041,
+        "p90": 0.06031132874202571,
+        "p99": 0.06738955674930582
+      },
+      "e2e": {
+        "count": 627,
+        "mean": 351.1066520824709,
+        "p50": 387.2650127700035,
+        "p90": 507.9008203571953,
+        "p99": 570.6463984230224
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        },
+        "4P+4D": {
+          "attainment": 0.05855855855855856,
+          "pd_advantage": NaN,
+          "n_slo": 208
+        },
+        "6P+2D": {
+          "attainment": 0.036036036036036036,
+          "pd_advantage": NaN,
+          "n_slo": 128
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o512_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 304.97662343096454,
+      "amplification": 1.0174657357946404,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.16242270423816507,
+        "p50": 0.15836620499612764,
+        "p90": 0.17363918052287775,
+        "p99": 0.24847999344696287
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.013691483182659388,
+        "p50": 0.013865661168213087,
+        "p90": 0.015242900696529327,
+        "p99": 0.01626683903372128
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 7.160272567887971,
+        "p50": 7.247483663493767,
+        "p90": 7.962273431208451,
+        "p99": 8.480892732607899
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 3551,
+      "completion_rate": 0.9997184684684685,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 744.9637431279989,
+      "amplification": 2.4853546954349977,
+      "ttft": {
+        "count": 3551,
+        "mean": 0.2446379999283825,
+        "p50": 0.24067969399038702,
+        "p90": 0.2630795220611617,
+        "p99": 0.3426029055262916
+      },
+      "tpot": {
+        "count": 3551,
+        "mean": 0.016916919575073883,
+        "p50": 0.016998299133030737,
+        "p90": 0.01775875886689886,
+        "p99": 0.018166751548973206
+      },
+      "e2e": {
+        "count": 3551,
+        "mean": 8.89104466149889,
+        "p50": 8.933106195996515,
+        "p90": 9.333998591057025,
+        "p99": 9.536390463996213
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 3551,
+      "completion_rate": 0.9997184684684685,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 312.45468625507783,
+      "amplification": 1.0424141157985172,
+      "ttft": {
+        "count": 3551,
+        "mean": 0.36231219612768273,
+        "p50": 0.3462264990666881,
+        "p90": 0.4059687410481274,
+        "p99": 0.9837204645154998
+      },
+      "tpot": {
+        "count": 3551,
+        "mean": 0.03268022218101953,
+        "p50": 0.03333031399418835,
+        "p90": 0.03557429400772957,
+        "p99": 0.038558618127279086
+      },
+      "e2e": {
+        "count": 3551,
+        "mean": 17.068018403084057,
+        "p50": 17.392655145958997,
+        "p90": 18.56302695896011,
+        "p99": 20.159411454980727
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        },
+        "4P+4D": {
+          "attainment": 0.9997184684684685,
+          "pd_advantage": 0.9997184684684685,
+          "n_slo": 3551
+        },
+        "6P+2D": {
+          "attainment": 0.9997184684684685,
+          "pd_advantage": 0.9997184684684685,
+          "n_slo": 3551
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o64_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 300.21845804993063,
+      "amplification": 1.001591501940313,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.1369056966029782,
+        "p50": 0.13516780693316832,
+        "p90": 0.13835511771030723,
+        "p99": 0.15317223175661632
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.005430781936309906,
+        "p50": 0.005039893150780468,
+        "p90": 0.007063257358303028,
+        "p99": 0.0077793481296283135
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 0.4793568905444592,
+        "p50": 0.45412574551301077,
+        "p90": 0.5810393166844734,
+        "p99": 0.6301419002050533
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 3551,
+      "completion_rate": 0.9997184684684685,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 735.1059510430787,
+      "amplification": 2.452467041410379,
+      "ttft": {
+        "count": 3551,
+        "mean": 0.17374673821534295,
+        "p50": 0.17102365801110864,
+        "p90": 0.18273873499128968,
+        "p99": 0.24688138795318082
+      },
+      "tpot": {
+        "count": 3551,
+        "mean": 0.005485491774206834,
+        "p50": 0.0053864502226046865,
+        "p90": 0.006091995366168992,
+        "p99": 0.007109403222178419
+      },
+      "e2e": {
+        "count": 3551,
+        "mean": 0.5196563635758822,
+        "p50": 0.5100997349945828,
+        "p90": 0.5655082209268585,
+        "p99": 0.6639982180204242
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 300.38101644301787,
+      "amplification": 1.0021338307036503,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.18293367427152893,
+        "p50": 0.1822461549891159,
+        "p90": 0.1938482352765277,
+        "p99": 0.21272844232735222
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.007192309629699456,
+        "p50": 0.007143509595484902,
+        "p90": 0.008732455453672816,
+        "p99": 0.009842920153335268
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 0.636424056327808,
+        "p50": 0.6324848984950222,
+        "p90": 0.7393875011475757,
+        "p99": 0.8261980937235056
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        },
+        "4P+4D": {
+          "attainment": 0.9997184684684685,
+          "pd_advantage": 0.9997184684684685,
+          "n_slo": 3551
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        }
+      }
+    }
+  ]
+}

analysis/lpwl_5policy_600s.md

@@ -0,0 +1,131 @@
+# LPWL vs 4 baselines — parameter-free routing for agentic workloads
+
+Date: 2026-05-29. Hardware: dash1, 8×H20, Qwen3-Coder-30B-A3B, TP=1,
+max_model_len=200000, fresh vLLM per arm (cold APC), `--policy` via
+`scripts/b3_isolated_policy.sh`. Analyzer: `scripts/bench_report.py`.
+
+## Motivation
+
+unified+A+B carries too many knobs (`overload_factor`, `lmetric_decode_weight`,
+the 0.5 `cache_ratio` gate). Goal: a policy derived from the agentic *pattern*
+with no tuned constants, that does not overfit.
+
+## LPWL (Least-Prefill-Work-Left)
+
+`scripts/cache_aware_proxy.py:pick_instance_leastwork`, `--policy leastwork`:
+
+```
+score_i = pending_prefill_tokens_i + max(0, input_len − cache_hit_i)   → argmin
+```
+
+Tie-break: fewest `num_requests`, then round-robin. **Zero hyperparameters.**
+
+Why this shape (straight from the workload):
+- Decode is cheap (I/O ~217×) ⇒ the only load worth modeling is outstanding
+  *prefill* token-work. No decode weight; dropping LMetric's `×num_requests`
+  also makes an idle-but-decoding host score `input` (its true marginal cost),
+  not 0 — fixing the empty-batch degeneracy for free.
+- Cache-awareness *is* the affinity mechanism: a returning session's owner has
+  `new_uncached ≈ 0`, so it sticks unless its prefill backlog exceeds the cache
+  saving (`input`). The stick-vs-spill crossover is computed from real
+  token-work — no `overload_factor`, no `cache_ratio` gate.
+- Session skew degrades gracefully: a heavy session inflates its owner's
+  `pending_prefill`, auto-diverting *other* sessions while the heavy one stays
+  put (no cold re-prefill).
+
+## Results — 600s trace (`w600_r0.0015_st30_first600s.jsonl`, 807 reqs)
+
+This is the colder regime (theoretical APC ceiling ≈ 70% vs 80% for full w600).
+
+| policy | knobs | TTFT mean | TTFT p90 | E2E mean | E2E p90 | E2E p99 | TPOT p90 | APC | req-bal |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| **LPWL** | **0** | **3398** | **7983** | **8116** | **19014** | 87024 | 26 | 0.648 | **1.55×** |
+| unified+A+B | 3 | 3876 | 11562 | 8199 | 22569 | **74266** | **25** | 0.661 | 1.56× |
+| unified default | 2 | 5066 | 16389 | 10481 | 28427 | 96361 | 34 | 0.689 | 2.28× |
+| LMetric | 0 | 4809 | 14037 | 10051 | 26726 | 97442 | 32 | 0.507 | 2.11× |
+| sticky | 0 | 5758 | 20356 | 10815 | 34734 | 82732 | 28 | **0.696** | 3.86× |
+
+(latencies ms; req-bal = max:min per-worker request count; this batch predates
+the GPU-capture harness change so per-worker GPU util reads N/A.)
+
+### Findings
+
+1. **LPWL is overall best with zero knobs:** TTFT mean −12% / p90 −31%, E2E
+   mean ~tie / p90 −16% vs the tuned unified+A+B; best request balance; TPOT
+   tied-best. Only loss is E2E p99 (+17%) from heavy-class decode concentration.
+2. **The baselines bracket the problem and explain why LPWL works:** sticky has
+   the highest APC (0.696) but worst latency (hot-pin, 3.86× imbalance); LMetric
+   has the worst APC (0.507) because `×num_requests` swallows the cache signal.
+   LPWL drops exactly that factor, so locality re-emerges (APC 0.648, beside the
+   explicit-affinity policies) while balance stays tight — the sweet spot, no
+   gate, no tuning.
+3. **Anti-overfit, demonstrated:** unified+A+B was tuned (of=1.3, lmw=0.01) on
+   the *full* w600; on the colder 600s regime the parameter-free policy beats it
+   by 31% TTFT p90. The tuning did not transfer; LPWL did.
+
+### Per-class TTFT (ms, mean / p50 / p90 / p99) — LPWL dominates except the floor
+
+| class | LPWL p90 | A+B p90 | LPWL p99 | A+B p99 |
+|---|---:|---:|---:|---:|
+| WARM<5k | 319 | 324 | 1032 | 2092 |
+| MED5-20k | 1618 | 1952 | 3013 | 33189 |
+| HEAVY20-50k | 4851 | 6198 | 14599 | 29044 |
+| HEAVY+>50k | 28942 | 33777 | 52651 | 50778 |
+
+LPWL's only weak class is the workload-inherent HEAVY+>50k floor (≈tied across
+all policies). Elsewhere it avoids the mid-class tails the unified gate creates
+when it pins a mid request behind a 50k-token turn on a barely-warm owner.
+
+## Full-w600 cross-check (1214 reqs, `outputs/lpwl_vs_ab_live/`)
+
+On the warmer full trace, LPWL vs unified+A+B is a wash: LPWL wins TTFT p90
+(−14%) but loses TPOT (+38%) and per-worker balance. Combined claim across both
+regimes: **LPWL ∈ [tied, clearly-better] vs a tuned baseline, at zero knobs.**
+
+## Ablation: derived-κ decode term (`leastwork_kappa`) — NET-NEGATIVE
+
+Tested the proposed knob-free fix for LPWL's E2E-p99: `--policy leastwork_kappa`,
+`score = (pending_prefill + new_uncached) × (1 + κ·ongoing_decode_tokens)`, with
+κ = 2.5e-6 *derived* from hardware (KV ~100 KB/tok ÷ HBM 4 TB/s ÷ TPOT 10 ms on
+H20+Qwen3-30B-A3B), not trace-tuned. Same 600s trace, fresh vLLM, cold APC.
+
+| metric | leastwork | leastwork_kappa | Δ |
+|---|---:|---:|---:|
+| TTFT p90 | 7983 | 9390 | +18% (worse) |
+| TTFT p99 | 44891 | 42370 | −6% |
+| E2E p90 | 19014 | 21674 | +14% (worse) |
+| E2E p99 | 87024 | 90155 | +4% (did NOT fix) |
+| APC | 0.648 | 0.647 | tie |
+| req-balance | 1.55× | 1.97× | worse |
+
+**Verdict: decode-awareness is the wrong lever for agentic.** The κ term is
+correct physics aimed at a negligible effect (decode is cheap, output p50≈80),
+so it mostly bounces heavy requests off their cache-owner → cold re-prefill
+elsewhere → new hotspots (balance degrades 1.55×→1.97×). It does NOT fix E2E-p99
+because that tail is the **structural HEAVY+>50k floor** (per-class p99 ≈51–52k
+for *all* policies), not decode interference — i.e. not routing-fixable. This is
+a negative result that *justifies* LPWL's omission of any decode term. The policy
+is kept in-tree as a documented ablation; do not revive without a decode-heavy
+regime. (First run on the GPU-capturing harness: per-worker GPU util mean 42–83%,
+1.95× spread — it even shows the κ-induced imbalance.)
+
+## Caveats / open work
+
+- n=1 per arm. The 600s −31% TTFT p90 is corroborated by mean/p50/per-class, but
+  repeat to bound run-to-run noise (no 3× repeats yet, by request — quick single
+  set first).
+- E2E-p99 deep tail is the one consistent LPWL weak spot (heavy-session decode
+  concentration). Proposed knob-free fix: add `+ κ·ongoing_decode` with
+  `κ = measured(TPOT/token) / prefill_throughput` (a derived hardware ratio, not
+  a tuned scalar). Not yet implemented.
+
+## Repro
+
+```bash
+# 5-policy, 600s trace (≈18 min/arm, ~90 min total)
+OUTROOT=.../outputs/policy5_600s \
+    bash microbench/connector_tax/cache_sweep/run_5policy_600s.sh
+# unified report
+.venv/bin/python scripts/bench_report.py --root .../outputs/policy5_600s \
+    leastwork unified_ab unified_def lmetric sticky
+```

analysis/mb1/README.md

@@ -15,20 +15,28 @@ bottom; the **Summary** block is what gets cited.
 | Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** |
 | Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** |
 | Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** |
-| Maximum PD-disagg benefit per agentic decode | **≤ 50–200 ms** (= decode duration) |
 
-**§3.2 headline (cost vs benefit, this run + MB2)**:
+**What MB1 actually measures**:
 
-> Under chunked-prefill, every ongoing decode stream is essentially
-> **halted while a prefill chunk is in flight** — per-stream effective
-> TPOT during the burst is 15× to 2000× baseline, scaling with prefill
-> size. PD-disagg can recover this stall, but the recovery is bounded by
-> the **decode duration** of the request being protected. For agentic,
-> decode is 50–200 ms (tool-call output). MB2 shows PD-disagg pays
-> 300 ms – 10 s of KV-transfer cost per request to do that recovery. The
-> cost exceeds the benefit ceiling for any per-request KV ≥ ~80 MiB
-> (~830 tokens) — well below all agentic operating points. The benefit
-> never beats the cost in this workload.
+> During a prefill burst, every ongoing decode stream is essentially
+> halted (per-stream effective TPOT is 15×–2000× baseline, scaling with
+> prefill size). The **total decode time lost per prefill event is
+> `D × T_prefill`** (D concurrent decodes each lose ~T_prefill of useful
+> work). For the trace mean (P ≈ 33k tokens, T_prefill ≈ 4.5 s) at D=8
+> that's **~36 seconds of decode-equivalent work lost per request**.
+> This is the **upper bound on what PD-disaggregation's phase isolation
+> could recover** on the decode side.
+
+**⚠ Correction (2026-05-27)**: an earlier version of this README framed
+the §3.2 PD-disagg argument as "phase-isolation benefit is capped at
+the decode duration of the new request (~50–200 ms), so MB2 transfer
+cost dominates". That framing was wrong. The correct accounting is
+benefit-per-prefill-event = D × T_prefill (aggregate decode time saved
+across all stalled streams), which is **much larger than per-request
+transfer cost**. The actual reason static PD-disagg fails in agentic
+is **D-side KV pool capacity** (`figs/f4b_pdsep_kv_wall.png`), not a
+cost-vs-benefit imbalance on phase isolation. See `RESULTS_SUMMARY.md`
+section 4 for the corrected framing.
 
 ## Setup
 
@@ -107,25 +115,30 @@ the cleanest "average over the whole burst window" number).
   halts decode). This is the entire prefill duration's worth of decode
   time that could in principle be recovered.
 
-Two big caveats for **agentic** application:
+**Connecting to the §3.2 PD-disagg argument** (corrected):
 
-1. **Decode is short** (~50–200 ms for tool-call output). The actual
-   recoverable benefit per request is bounded by the decode duration,
-   not by `prefill_ttft`. If a decode lasts 100 ms and a 5-second prefill
-   collides with it, PD-disagg can save at most 100 ms — not 5 s.
-2. **PD-disagg pays KV-transfer cost** (MB2: 300 ms – 10 s per request
-   for agentic sizes). For any KV ≥ ~80 MiB the cost already exceeds the
-   ~100 ms benefit ceiling. Cost > benefit across the whole agentic
-   distribution.
+PD-disagg's promised phase-isolation benefit is **per prefill event**,
+not per request. When a new prefill arrives, it stalls every concurrent
+decode stream on the same GPU. The aggregate decode time lost across
+those D streams is `D × T_prefill`. PD-disagg moving prefill off-decode-GPU
+recovers all of it.
 
-## §3.2 cost-vs-benefit figure
+Plugging numbers per prefill event:
 
-`figs/pd_cost_vs_benefit.png` overlays MB1 benefit ceiling (50–200 ms
-band, capped by decode duration) on top of MB2 transfer cost curve. The
-cost curve crosses the benefit ceiling somewhere around **80 MiB / 830
-tokens** of KV — well below the trace mean (192 MiB / 2k tok ≈ trace
-mean per request KV, and we know agentic averages 33k tokens, p99
-125k). For anything bigger PD-disagg pays more than it can recover.
+| Prefill size | T_prefill | PD-disagg cost (MB2 T_transfer) | PD-disagg benefit (D=8 × T_prefill) | Ratio |
+|---:|---:|---:|---:|---:|
+| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s | 0.7 % |
+| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s | 0.9 % |
+| 125k tok (~p99) | 57 s | 1.9 s | 456 s | 0.4 % |
+
+On the **phase-isolation axis alone**, PD-disagg wins by 100×–250×.
+The reason static PD-disagg nonetheless **fails in agentic** is a
+*different* failure mode: the D-side KV pool cannot fit p90+ requests
+(p99 = 11.5 GiB; D-instance pool ≈ 38 GiB; 4P+4D halves system-wide
+decode capacity → TTFT p50 62×, success rate 99.5% → 52% in colleague's
+4P+4D experiment). The structural problem is **capacity** (see
+`figs/f4b_pdsep_kv_wall.png`), not transfer-cost vs phase-isolation
+trade-off.
 
 ## Reproduction
 
@@ -174,4 +187,7 @@ ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop
 
 3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on
 dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`.
-Figures: `figs/mb1_interference.png`, `figs/pd_cost_vs_benefit.png`.
+Figure: `figs/mb1_interference.png`. The figure
+`figs/pd_cost_vs_benefit.png` from the original commit `029821c` was
+based on the wrong "benefit ≤ decode duration" accounting; **deleted in
+the correction commit**.

analysis/mb2/README.md

@@ -24,12 +24,25 @@ get cheaper by co-locating P and D on the same node — the ~9.7 GB/s
 ceiling applies regardless. Halving the transfer cost cannot be bought
 back by topology.
 
-**Headline for the paper §3.2**: at the agentic tail, **pure KV transfer
-takes 1.5 – 10 s**. A median agentic decode is **50 – 200 ms** of tool-call
-output. So **PD-disaggregation adds 8 – 100 × decode-time of transfer on
-top of every routed request**. Phase isolation (the thing PD-disagg
-trades transfer cost for) can only win back at most one decode duration
-— for agentic that's negligible. The arithmetic is one-sided.
+**What MB2 actually measures**: the **per-request charge** that
+PD-disagg pays for every routed request — `T_transfer ≈ KV_size / 9.7
+GB/s`. For agentic this is **8 ms (192 MiB / trace lower) – 1.9 s
+(11.5 GiB / p99)**.
+
+**⚠ Correction (2026-05-27)**: an earlier version of this README
+framed §3.2 as "transfer cost (1.5–10 s) >> decode duration (50–200 ms),
+so PD-disagg loses on cost-vs-benefit." That accounting was wrong:
+PD-disagg's phase-isolation benefit is **per-prefill-event** and equals
+`D × T_prefill` (aggregate across stalled decode streams), not the
+single-request decode duration. With trace-mean `T_prefill = 4.5 s` and
+D = 8, the benefit is ~36 s — far larger than the ~0.32 s transfer
+cost. PD-disagg's phase-isolation axis is a *win*, not a loss.
+
+The actual reason static PD-disagg fails in agentic is **D-side KV
+capacity** (`figs/f4b_pdsep_kv_wall.png`), not a cost-vs-benefit
+imbalance. See `RESULTS_SUMMARY.md` section 4 for the corrected
+framing. MB2 still serves as the source of the per-request transfer
+cost number used in that analysis.
 
 ---
 
@@ -137,43 +150,44 @@ analysis; not done yet.
   treats them as additional samples (same sizes); the per-size
   aggregates use all of them.
 
-## Implications for §3.2 PD-disagg cost argument
+## Implications for §3.2 PD-disagg argument
 
 For each PD-disagg-routed request, transfer wall-time is:
 
 ```
-T_transfer(KV_size) = max(  pure_transfer(KV_size),  rx_overhead  )
-                    ≈ KV_size / 9.7 GB/s   for KV_size <= 3 GiB
+T_transfer(KV_size) ≈ KV_size / 9.7 GB/s   for KV_size ≤ 3 GiB
                     ≈ 0.3 – 10 s            for KV_size in [3, 12] GiB
 ```
 
-Agentic decode wall-time is typically 50 – 200 ms (tool-call output of
-a few tens of tokens at ~50 tok/s). So the **transfer/decode ratio**
-under intra-node best-case Mooncake is:
+This is the **per-request transfer charge** of PD-disagg. It's a
+real cost, but in the context of phase-isolation accounting it is
+*small* compared to the benefit:
 
-| KV size | T_transfer @9.7 GB/s | typical decode | T_transfer / T_decode |
-|---|---:|---:|---:|
-| 192 MiB (2k tok)   |   20 ms | 100 ms | 0.2× |
-| 768 MiB (8k tok)   |   84 ms | 100 ms | 0.8× |
-| 3 GiB (33k tok ≈ trace mean) |  321 ms | 100 ms | **3.2×** |
-| 6 GiB (~p90)       | 1900 ms | 100 ms | **19×**  |
-| 12 GiB (~p99)      | 2800 ms | 100 ms | **28×** (median) – **100×** (p99 variance) |
+| Prefill | T_prefill (MB1) | T_transfer (MB2) | Phase-isolation benefit at D=8 = D × T_prefill |
+|---:|---:|---:|---:|
+| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s |
+| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s |
+| 125k tok (~p99) | 57 s | 1.9 s | 456 s |
 
-PD-disagg's promised payoff is *eliminating prefill–decode interference
-on the decode instance*. The maximum benefit it can buy is bounded
-above by the **decode duration itself** (you cannot recover more time
-than the decode existed). For agentic that's 50 – 200 ms. The cost is
-the table column above — 0.3 – 10 s of transfer per routed request.
+On the phase-isolation axis alone, PD-disagg recovers two orders of
+magnitude more decode time than it pays in transfer. **It is NOT this
+axis that defeats static PD-disagg in agentic** — see colleague's
+4P+4D experiment (TTFT p50 62×, success rate 99.5% → 52%) which is
+driven by **D-side KV-pool overflow** on long-context requests
+(`figs/f4b_pdsep_kv_wall.png`), not by transfer latency.
 
-**Cost > Benefit by 5× to 100× across the agentic distribution.** Below
-~3 GiB the ratio is small (≤1×); above 3 GiB the ratio explodes; above
-6 GiB even individual draws can take 10 s for a single transfer.
+What MB2 contributes to the paper is therefore:
+- The **per-request transfer cost number** (used as the cost input
+  to the cost-benefit accounting above).
+- The empirical observation that **Mooncake's transfer cost is
+  topology-independent** — intra-node and inter-node both go through
+  the RDMA NIC and hit the same 9.7 GB/s ceiling. PD-disagg's
+  transfer cost does not get cheaper by co-locating P and D.
 
-This data alone is not the whole §3.2 argument — we still need to
-account for D-side KV capacity (`f4b`, separate axis), cache reuse loss,
-and static-partition mismatch (MB3 / MB4 / MB5). But it nails one
-of the two key cost axes with measured numbers from vanilla mooncake,
-not the dash0 patched build.
+The dominant §3.2 failure mode of static PD-disagg in agentic is
+**capacity**, not transfer cost. MB3 / MB4 / MB5 will quantify the
+remaining axes (D-pool occupancy, cache reuse degradation under PD
+routing, static-partition mismatch).
 
 ## Open questions / next runs
 

analysis/mb5_pd_ablation/README.md

@@ -0,0 +1,107 @@
+# PD-disagg vs colocation — controlled reuse & concurrency axes (v2)
+
+Self-contained results for the **controlled-variable** redo of the MB5 PD-vs-colo
+ablation. Supersedes the confounded first cut (held input fixed and sliced the
+prefix, so "more reuse" was entangled with "less prefill"). All arms route through
+the proxy at fair **APC parity** (session-routed producers reach the same prefix-cache
+hit rate as colo), so PD loses on *structure*, not on broken cache.
+
+- **Config arms:** `colo` = 8×kv_both (8C-proxy, session-affinity); PD = `6P+2D / 4P+4D / 2P+6D`.
+- **Driver:** closed-loop N (`REPLAY_MAX_INFLIGHT`) + fixed think-time; `gen_synthetic_trace.py --mode regular`.
+- **PD-arm wall-cap:** collapsed PD arms drain pathologically slowly, so PD arms run with a
+  wall-deadline (`REPLAY_MAX_DURATION`; un-run turns counted as failures → honest completion%);
+  **colo is uncapped** so the reference is always fully measured.
+- **Hardware:** run on **dash2** (8×H20). dash0's RDMA NICs were faulted for Mooncake during
+  this work (could not init the transfer engine; needs an admin reset — no sudo); dash2's NICs
+  are healthy. cpfs/venv/data are shared across the boxes.
+
+---
+
+## 1. Reuse / APC axis — fixed real prefill, vary cached prefix
+
+N=8. Hold the **real new-prefill work per turn constant** (`--delta-len`) and grow the cached
+prefix → reuse = prefix/(prefix+delta). Three shapes isolate output vs delta:
+
+| | delta (real prefill/turn) | output | role |
+|---|---|---|---|
+| **A** | 2048 | 256 | original |
+| **C** | 2048 | 128 | A vs C = pure **output** 256→128 |
+| **B** | 1024 | 128 | C vs B = pure **delta** 2048→1024 |
+
+**PD-best advantage** = colo E2E-p90 / best-PD E2E-p90 (>1 ⇒ PD wins):
+
+| reuse% | A d2048/o256 | C d2048/o128 | B d1024/o128 |
+|---|---|---|---|
+| 20 | 1.34 | 1.41 | — |
+| 50 | 1.36 | 1.37 | — |
+| 67 | **1.47** | **1.49** | **1.27** |
+| 80 | 1.31 | 1.23 | 1.25 |
+| 90 | 1.15 | 1.01 | — |
+| 95 | 0.87 | 0.89 | 0.89 |
+
+![reuse 3-way](../../figs/mb5_pd_ablation/reuse_compare_ABC.png)
+
+**Findings:**
+1. **Output length is ~negligible.** A and C (same delta) track each other across the whole
+   range — halving output barely moves PD's advantage.
+2. **Delta (real prefill/turn) is the dominant shape factor.** B (delta=1024) sits clearly
+   below A/C at mid reuse (67%: 1.27 vs ~1.48). More real prefill per turn → bigger PD win,
+   because PD-disagg's benefit is isolating prefill from decode — more prefill to isolate.
+3. **Crossover to colo at reuse ~90–95% is robust** across all three shapes: PD always loses
+   the high-reuse / large-resident-context corner (it must KV-transfer the whole resident
+   context every turn for a few hundred new tokens; colo keeps it local).
+
+*Caveat:* the clean, uncapped, 100%-completion comparison region is reuse **20–80%** (carries
+findings 1–2). At reuse 90/95% the PD arms collapse and C's points are capped-completion, while
+A/B are full-drain — comparable in direction, not in exact PD completion%.
+
+Data: `fig1_reuse_fixed.json` (A), `fig1_reuse_d2048_o128.json` (C), `fig1_reuse_d1024_o128.json` (B).
+
+---
+
+## 2. Concurrency axis — agentic corner, sweep N
+
+in=32768 (prefix 32256 + delta 512, **reuse 0.984**), out=128; closed-loop N ∈ {8,16,32,48,64,96,128};
+PD arms capped 600s, colo uncapped.
+
+| N | **colo** completion · E2E-mean · TPS | best PD-arm completion |
+|---|---|---|
+| 8 | **256/256** · 2.4s · 326 | 6P+2D 256/256 |
+| 16 | **512/512** · 3.5s · 462 | 6P+2D 439/512 (86%) |
+| 32 | **1024/1024** · 13.3s · 190 | all PD **<27%** |
+| 48 | **1536/1536** · 24.9s · 168 | all PD <32% |
+| 64 | **2048/2048** · 38.4s · 166 | all PD <31% |
+| 96 | **3072/3072** · 60.0s · 171 | PD **2–7%** |
+| 128 | **4096/4096** · 80.8s · 181 | 4P+4D 6%, 2P+6D <1% |
+
+![concurrency](../../figs/mb5_pd_ablation/fig3_concurrency_axis.png)
+
+**Finding:** **colo completes 100% of requests at every concurrency level** — it degrades
+*gracefully* (latency rises 2.4s→81s, nothing dropped). **Every static PD split collapses, and
+progressively earlier as N rises**: PD is viable only at N≤8–16; by N≥32 it drops 70–99% of
+requests while its prefix-cache hit-rate craters to ~0%. colo's elastic pool absorbs the
+time-varying P/D demand; the static partition + per-turn 32k KV-transfer cannot. (Latency
+percentiles count successes only, so they *understate* PD — read them with the completion column.)
+
+Data: `fig3_conc32k.json`.
+
+*Caveat:* N=128 6P+2D is missing (one transient vLLM/Mooncake startup flake at the end); does
+not change the picture (all PD arms are already collapsed by N=128). The SLO auto-stop in the
+driver is a no-op (a stdout-capture bug), so the full grid ran — more points, not fewer.
+
+---
+
+## 3. Reproduce
+
+```bash
+# on a box with healthy Mooncake/RDMA NICs (dash2), cpfs mounted:
+R=/home/admin/cpfs/wjh/agentic-kv-fresh
+# reuse axis (three shapes): DELTA/OL pick the shape; tag carries _o${OL}
+ssh dash2 "cd $R && DELTA=2048 OL=256 bash microbench/fresh_setup/run_reuse_fixed.sh"
+ssh dash2 "cd $R && DELTA=2048 OL=128 bash microbench/fresh_setup/run_reuse_fixed.sh"
+ssh dash2 "cd $R && DELTA=1024 OL=128 bash microbench/fresh_setup/run_reuse_fixed.sh"
+# concurrency axis (capped):
+ssh dash2 "cd $R && NLIST='8 16 32 48 64 96 128' CONC_PD_MAXDUR=600 bash microbench/fresh_setup/run_conc.sh"
+# render (reads the *.json in this dir):
+python microbench/fresh_setup/plot_pd_crossover.py
+```

analysis/migration_trigger_validation/README.md

@@ -0,0 +1,147 @@
+# Migration Trigger Validation (unified_v4) — 2026-05-30
+
+Hardware: dash2, 8×H20, Qwen3-Coder-30B-A3B, TP=1, kv_both + DR-fix substrate.
+Trace: `w600_r0.0015_st30_first600s.jsonl` (807 reqs, 600s span).
+Policy: `unified_v4` = unified hybrid routing + pending-prefill-queue-triggered
+session migration (commit `3a6bf5d` on `kzlin-dev` branch).
+
+## Research Question
+
+Does Pillar 2 (hot-triggered session migration) provide measurable benefit on
+top of Pillar 1 (affinity-default routing)?
+
+## Experiment Design
+
+| Arm | Policy | Substrate | Trace QPS |
+|---|---|---|---|
+| unified_1x | unified (affinity-only) | kv_both + DR-fix | ~1.3 (original) |
+| unified_v4_1x | unified_v4 (affinity + migration) | kv_both + DR-fix | ~1.3 |
+| unified_v4_2x | unified_v4 (affinity + migration) | kv_both + DR-fix | ~2.7 (2× compressed) |
+
+The 2× trace was generated by halving inter-request intervals:
+`ts_new = ts_min + (ts_orig - ts_min) / 2`.
+
+## Results
+
+### 1x QPS: unified vs unified_v4
+
+| Metric | unified | unified_v4 | Delta |
+|---|---:|---:|---:|
+| OK/total | 807/807 | 807/807 | — |
+| TTFT mean | 3.990s | 4.142s | +3.8% |
+| TTFT p50 | 0.719s | 0.711s | −1.0% |
+| TTFT p90 | 11.499s | 12.293s | +6.9% |
+| TPOT p90 | 0.024s | 0.022s | −9.3% |
+| E2E p50 | 2.265s | 2.293s | +1.2% |
+| E2E p90 | 24.507s | 23.955s | −2.3% |
+| **Migrations** | **0** | **0** | — |
+
+**Conclusion**: At 1x QPS (~1.3 req/s, ~0.16 req/instance/s), the migration
+trigger NEVER fires. The two arms produce statistically identical results.
+
+### 2x QPS: unified_v4 under higher load
+
+| Metric | unified_v4 @ 2x |
+|---|---:|
+| OK/total | 807/807 |
+| TTFT mean | 5.227s |
+| TTFT p90 | 15.000s |
+| E2E p90 | 39.401s |
+| **Migrations** | **4/807 (0.5%)** |
+
+4 migrated requests (all verified via `v3_decode_target_url` in breakdown):
+
+| Session | Input | new_local | src_pp | fleet_median | proj_prefill | Target |
+|---|---:|---:|---:|---:|---:|---|
+| 1313181 | 22,686 | 22,686 | 13,360 | 6,680 | 5.1s | inst_5 |
+| 1310590 | 32,440 | 14,520 | 57,051 | 12,630 | 10.2s | inst_4 |
+| 1373431 | 126,340 | 126,340 | 73,385 | 33,294 | 28.5s | inst_4 |
+| 1313181 | 60,004 | 17,508 | 19,503 | 3,806 | 5.3s | inst_5 |
+
+## Root Cause Analysis: Why Zero Migrations at 1x
+
+The unified_v4 trigger requires BOTH arms to fire simultaneously:
+- **Absolute SLO arm**: `proj_prefill_s(src) > 2.5s` — fires for 41% of eligible requests
+- **Relative arm**: `src_pending_prefill > fleet_median × 1.5` — NEVER fires at 1x
+
+The relative arm fails because `pending_prefill_tokens` (the proxy's shadow
+counter) is 0 for **95% of routing decisions** at 1x QPS:
+
+| QPS | src_pp > 0 (% of eligible) | Migrations |
+|---:|---:|---:|
+| 1.3 (1×) | 5% (8/241) | 0 |
+| 2.7 (2×) | 24% (62/257) | 4 |
+
+**Mechanism**: `pending_prefill_tokens` reflects previously-dispatched requests
+that haven't finished their prefill yet. At 0.16 req/instance/s, each instance
+completes its prefill before the next request arrives — the counter is almost
+always 0 at decision time. Only under genuine queueing pressure (2× and above)
+does the counter become non-zero and the relative arm can fire.
+
+The high TTFT at 1x (~11.5s p90) comes from **compute-bound large prefills**
+(single 60k+ token requests inherently need ~9s), NOT from queue depth.
+
+## Interpretation for Paper
+
+1. **The migration mechanism is functionally correct.** At 2x it fires on the
+   right signal (src genuinely overloaded relative to fleet) and selects valid
+   targets (cooler instances with load gap).
+
+2. **At benchmark scale (8 instances, ~1 QPS), migration is not needed.** The
+   affinity-default routing (Pillar 1) already achieves APC ~79% and the
+   remaining hot-pin issue is mild (max-worker/median-worker ≈ 3.7×). The
+   "dispatch coupling" feedback loop is present but not yet at the catastrophic
+   amplification regime.
+
+3. **Migration becomes relevant under scale-out + higher load.** With more
+   instances (16–32), session skew concentrates more load per hot instance
+   while cold instances sit idle — exactly the condition where `src_pp >
+   fleet_median × 1.5` naturally fires. The 1x→2x progression (0%→0.5%
+   migration rate) shows the correct scaling direction.
+
+4. **Paper §3.3 framing**: Migration is a **scale-out insurance mechanism**
+   that gracefully degrades to no-op under low load. Its value is NOT
+   demonstrable at 8-instance single-node benchmark; the argument must rely on
+   (a) the mechanism's correctness (this experiment), (b) the substrate's
+   net-positive property (commit `ef9e010`), and (c) scale-out projection
+   (future: 16+ GPU, multi-node).
+
+## Next Steps
+
+- [ ] **Scale-out validation** (16 GPU, 2 nodes): With more instances and the
+  same trace, more sessions compete per-instance → higher pending_prefill →
+  migration triggers naturally. This is the strongest evidence path.
+- [ ] **3–4× QPS on 8 instances**: Push to saturation to measure migration's
+  effect in the catastrophic regime. Risk: may exceed serving capacity (errors).
+- [ ] **Threshold sensitivity**: Ablate `v4_rel_hi` (1.5→1.2→1.0) and
+  `v4_ttft_slo_s` (2.5→1.5→1.0) to characterize the trigger landscape.
+
+## Reproduction
+
+```bash
+# On dash2 (local /tmp, does NOT modify shared NAS):
+# 1x QPS
+bash /tmp/migration_exp/run_migration_ab.sh   # interleaved unified vs unified_v4
+
+# 2x QPS
+python3 -c "
+import json
+trace_in = '/home/admin/cpfs/wjh/agentic-kv/traces/w600_r0.0015_st30_first600s.jsonl'
+rows = [json.loads(l) for l in open(trace_in)]
+ts_min = min(r['timestamp'] for r in rows)
+for r in rows: r['timestamp'] = ts_min + (r['timestamp'] - ts_min) / 2.0
+with open('/tmp/migration_exp/trace_2x.jsonl', 'w') as f:
+    for r in rows: f.write(json.dumps(r) + '\n')
+"
+bash /tmp/migration_exp/run_2x.sh
+```
+
+## Data Locations (dash2 /tmp, ephemeral)
+
+| Path | Content |
+|---|---|
+| `/tmp/migration_exp/outputs/unified_run1/` | Baseline arm (1x) |
+| `/tmp/migration_exp/outputs/unified_v4_run1/` | Migration arm (1x) |
+| `/tmp/migration_exp/outputs/unified_v4_2x/` | Migration arm (2x) |
+| `/tmp/migration_exp/outputs/*/breakdown.json` | Per-request routing decisions with v4_* fields |
+| `/tmp/migration_exp/outputs/*/metrics.jsonl` | Per-request latency metrics |

analysis/migration_trigger_validation/results.json

@@ -0,0 +1,110 @@
+{
+  "experiment": "migration_trigger_validation",
+  "date": "2026-05-30",
+  "hardware": "dash2, 8xH20, Qwen3-Coder-30B-A3B, TP=1",
+  "substrate": "kv_both + DR-fix (delay_free_blocks + VLLM_EVICT_SENT_BLOCKS gate)",
+  "arms": {
+    "unified_1x": {
+      "metrics": {
+        "ok": 807,
+        "total": 807,
+        "ttft_mean": 3.99,
+        "ttft_p50": 0.719,
+        "ttft_p90": 11.499,
+        "ttft_p99": 45.982,
+        "tpot_p90": 0.0239,
+        "e2e_p50": 2.265,
+        "e2e_p90": 24.507,
+        "e2e_p99": 71.233
+      },
+      "migrations": 0
+    },
+    "unified_v4_1x": {
+      "metrics": {
+        "ok": 807,
+        "total": 807,
+        "ttft_mean": 4.142,
+        "ttft_p50": 0.711,
+        "ttft_p90": 12.293,
+        "ttft_p99": 46.148,
+        "tpot_p90": 0.0217,
+        "e2e_p50": 2.293,
+        "e2e_p90": 23.955,
+        "e2e_p99": 75.915
+      },
+      "trigger_summary": {
+        "trace": "w600_r0.0015_st30_first600s.jsonl",
+        "qps_factor": 1,
+        "total_requests": 807,
+        "migrations_triggered": 13,
+        "size_floor_filtered": 552,
+        "eligible_requests": 255,
+        "slo_arm_true": 100,
+        "src_pp_nonzero": 22
+      }
+    },
+    "unified_v4_2x": {
+      "metrics": {
+        "ok": 807,
+        "total": 807,
+        "ttft_mean": 5.227,
+        "ttft_p50": 0.942,
+        "ttft_p90": 15.0,
+        "ttft_p99": 59.227,
+        "tpot_p90": 0.1087,
+        "e2e_p50": 5.035,
+        "e2e_p90": 39.401,
+        "e2e_p99": 163.032
+      },
+      "trigger_summary": {
+        "trace": "trace_2x.jsonl (timestamps / 2)",
+        "qps_factor": 2,
+        "total_requests": 807,
+        "migrations_triggered": 4,
+        "size_floor_filtered": 550,
+        "eligible_requests": 257,
+        "slo_arm_true": 133,
+        "src_pp_nonzero": 62,
+        "pending_prefill_p90_when_nonzero": 68131,
+        "migrated_details": [
+          {
+            "session_id": "1313181",
+            "input_length": 22686,
+            "new_local": 22686,
+            "src_pending_prefill": 13360,
+            "fleet_median_pp": 6680.0,
+            "proj_prefill_s": 5.15,
+            "target_idx": 5
+          },
+          {
+            "session_id": "1310590",
+            "input_length": 32440,
+            "new_local": 14520,
+            "src_pending_prefill": 57051,
+            "fleet_median_pp": 12630.5,
+            "proj_prefill_s": 10.22,
+            "target_idx": 4
+          },
+          {
+            "session_id": "1373431",
+            "input_length": 126340,
+            "new_local": 126340,
+            "src_pending_prefill": 73385,
+            "fleet_median_pp": 33294.5,
+            "proj_prefill_s": 28.53,
+            "target_idx": 4
+          },
+          {
+            "session_id": "1313181",
+            "input_length": 60004,
+            "new_local": 17508,
+            "src_pending_prefill": 19503,
+            "fleet_median_pp": 3806.5,
+            "proj_prefill_s": 5.29,
+            "target_idx": 5
+          }
+        ]
+      }
+    }
+  }
+}

analysis/pd_separation_analysis.md

@@ -23,6 +23,22 @@ Per-request breakdown shows **87.7% of TTFT** is spent waiting for KV cache memo
 > Earlier cross-machine comparison (commit `1e86285`) was invalidated — baseline used warm instances. See REPORT.md §3.5.
 | **Delta** | **-45%** | **-36%** | **-44%** | **+30pp** |
 
+> ✅⚠️ **2026-05-30 — confirmed + refined by the clean MB5 ablation; one caveat.**
+> A producer-side contamination (`e13391e`: evicts a producer's prefix-cache on every
+> KV transfer) was found in the *agentic-kv-fresh* MB5 stack and gated off; everything
+> was re-run clean.
+> - **Confirmed:** this doc's central thesis — PD's failure is a **decode-side KV memory
+>   wall**, not transfer/prefill cost — is reproduced on the clean stack (concurrency
+>   axis: at N=64 the split collapses, APC 71%→1.4%, TPS −30%; colo scales). Fig 3.
+> - **Refined:** "PD separation is net negative" is **regime-dependent**, not universal.
+>   Clean ablation shows PD *wins* at low load / decode-heavy / low-reuse and loses the
+>   **agentic corner** (high reuse + short output + large context + high concurrency).
+> - **Caveat (cross-check):** if this study's runs used the fork vLLM that contains
+>   `e13391e`, any **producer prefix-cache / APC reuse** figures here (e.g. §5.3.1) may be
+>   understated. The decode-memory-wall result is *not* reuse-dependent and is unaffected.
+>   On the clean stack, session-routed producers reach APC parity with colo (71–82%).
+> Figures: [`figs/mb5_pd_ablation/`](../figs/mb5_pd_ablation/).
+
 ---
 
 ## 1. Workload Characterization

analysis/v2/PD_DISAGG_LMETRIC.md

@@ -0,0 +1,165 @@
+# PD-colo vs PD-disagg on the real agentic trace — LMetric (cache-aware) clean-stack anchor
+
+**Figure:** [`figs/v2/fig4_lmetric_pd_vs_colo.png`](../../figs/v2/fig4_lmetric_pd_vs_colo.png)
+**Data:**   [`analysis/v2/fig4l_lmetric.json`](fig4l_lmetric.json)
+**Date:**   2026-05-31 · Hardware: dash1, 8×H20 · Model: Qwen3-Coder-30B-A3B-Instruct
+· vLLM 0.18.1 (V1, chunked-prefill on, `e13391e` eviction gated **off**)
+· Mooncake 0.3.11 · Routing: cache-aware proxy with **`--policy lmetric`**
+· Replayer per-request timeout 600 s.
+
+## TL;DR
+
+On the production agentic trace with the *right* routing baseline (LMetric, cache-aware),
+**PD-colo (8× kv_both) keeps 100 % completion on both traces** and matches the daily-bench
+expectation (~17 min for the high-load first600s, ~50 min for the full trace, with E2E p50
+~3 s and TTFT p50 ~1 s — **3.5–7× better than the original §3 round-robin baseline at the
+same wall-clock**). Every static **PD-disagg ratio fails** (14–65 % completion), and the
+failure mode rotates predictably with the split — **no static partition has a working
+operating point on this workload**. LMetric improves colo dramatically; it does *not*
+rescue PD-disagg, confirming the bottleneck is structural (D-pool admission capacity +
+multi-turn KV accumulation), not routing. A follow-up linear-policy run with PD-disagg
+**wall-capped at 2× the colo wall** (see end of doc) hits the **identical** success-rate
+ceiling — confirming the cap is structural, not policy-driven.
+
+## Setup
+
+- Trace: `w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions, agentic multi-turn,
+  contexts up to ~112 k tokens; "first600s" variant = same heavy sessions compressed
+  into 600 s → 807 reqs at 3.2× higher arrival rate).
+- 8 instances on 8 GPUs.
+- `--mode baseline` for colo (plain vLLM); `--mode pdsep --pd-ratio P:D` for the three PD
+  splits, all with Mooncake KV transfer.
+- Cache-aware proxy with LMetric scoring (`P_tokens × num_requests`) + session affinity
+  for multi-turn (the colleague's canonical baseline).
+
+## Results
+
+### first600s (1.35 req/s, high-load stress)
+
+| arm | success | E2E mean / p50 / p90 / p99 | TTFT p90 | TPOT p99 | TPS | wall |
+|---|---|---|---|---|---|---|
+| **colo (8C)** | **807/807 = 100 %** | 11.1 / 3.27 / 28.6 / 95.9 s | 14.5 s | 388 ms | 226 | 17.0 min |
+| pd6 (6:2) | 474/807 = **58.7 %** | 83.2 / 6.75 / 382 / 542 s | 380 s | 19 ms | 40 | 55 min |
+| pd4 (4:4) | 348/807 = **43.1 %** | 203 / 215 / 477 / 575 s | 475 s | 25 ms | 15 | 114 min |
+| pd2 (2:6) | 180/807 = **22.3 %** | 380 / 536 / 579 / 602 s | 577 s | 18 ms | 34 | 321 min* |
+
+### Full trace (0.42 req/s, original §3 anchor load)
+
+| arm | success | E2E mean / p50 / p90 / p99 | TTFT p90 | TPOT p99 | TPS | wall |
+|---|---|---|---|---|---|---|
+| **colo (8C)** | **1214/1214 = 100 %** | 10.9 / 3.13 / 29.6 / 93.7 s | 16.9 s | 254 ms | 125 | 49.9 min |
+| pd6 (6:2) | 793/1214 = **65.3 %** | 61.9 / 3.70 / 307 / 477 s | 300 s | 18 ms | 46 | 94 min |
+| pd4 (4:4) | 533/1214 = **43.9 %** | 131 / 8.22 / 468 / 531 s | 467 s | 21 ms | 13 | 231 min |
+| pd2 (2:6) | 169/1214 = **13.9 %** | 195 / 6.82 / 552 / 593 s | 549 s | 13 ms | 1 | 563 min |
+
+\* The pd2 wall-clock is dominated by per-request timeouts (`request_timeout=600 s`)
+draining concurrently behind the multi-turn session causality.
+
+## Five clean findings
+
+1. **LMetric+colo is the right baseline.** Full-trace colo wall **49.9 min ≈ the original
+   §3 RR's 49.9 min**, but E2E p50 **3.13 s vs §3's 10.8 s (3.5×)** and TTFT p50
+   **1.02 s vs §3's 7.0 s (7×)**. Same throughput envelope, far better latency — by virtue
+   of cache-aware routing concentrating each session's turns onto one instance for
+   prefix-cache reuse. The original §3 RR was an *unfairly weak* colo baseline.
+
+2. **Every static PD-disagg ratio fails on the agentic workload.** Completion drops to
+   14–65 %, on *both* traces. The drop is not a high-load artifact (it holds at the
+   original §3 arrival rate of 0.42 req/s); it is structural.
+
+3. **Failure mode rotates predictably with the P:D split:**
+   - **pd2 (2 producers)** → prefill-bound → 78–86 % TTFT timeouts.
+   - **pd6 (2 decode)** → decode-admission-bound → 35–41 % TTFT timeouts.
+   - **pd4 (4P+4D)** → both bottlenecks hit → 57 % TTFT timeouts.
+   - **No static ratio works.** Colo's elastic 8-GPU pool absorbs whichever phase is
+     hot at the moment.
+
+4. **Decode isolation works, but doesn't matter under failure.** TPOT p99 on every PD
+   arm is **13–25 ms** — an order of magnitude better than colo's 254–388 ms — but the
+   win applies only to the 14–65 % of requests that get admitted. The other 35–86 %
+   time out before ever seeing a first token, so the TPOT win is invisible to the end user.
+
+5. **The §3 RR "100 % PD completion" was a measurement artifact.** Original §3
+   (contaminated stack, RR routing) reported 100 % completion for pd6/pd4. LMetric on
+   the clean stack shows 44–65 %. Most plausible mechanism: `e13391e`'s eviction of
+   producer KV on every transfer acted as a **relief valve**, reducing producer-pool
+   pressure and letting more requests squeeze under the 600 s timeout — at the (uncosted)
+   price of cross-turn re-prefill. With the eviction off, producers retain prefix
+   correctly → cache works on PD too → but the cache itself contends for producer
+   pool capacity, and the decode-pool admission ceiling tips earlier. **PD-disagg is
+   worse on agentic than §3 advertised, not better.**
+
+## Linear-policy + wall-cap follow-up (2026-06-01) — the success ceiling is policy-invariant
+
+To check whether the LMetric routing was secretly handicapping PD-disagg, we re-ran
+first600s with the **default `--policy linear`** (cache-aware load score + sticky
+session affinity — the original baseline of the cache_aware_proxy stack) and
+**wall-capped each PD-disagg arm at 2 × the colo wall** (kill bench.sh + cleanup
+GPUs once cap is exceeded, record `records_at_cap`).
+
+| arm | linear success | linear wall | linear @-cap? | lmetric success | lmetric wall |
+|---|---|---|---|---|---|
+| **colo** | 807/807 = **100 %** | 964 s | — | 807/807 = **100 %** | 1021 s |
+| **pd6 (6:2)** | **472/807 = 58 %** | 2280 s | ⊗ cap (706 dispatched) | 474/807 = 59 % | 3325 s |
+| **pd4 (4:4)** | **349/807 = 43 %** | 2281 s | ⊗ cap (577 dispatched) | 348/807 = 43 % | 6850 s |
+| **pd2 (2:6)** | **176/807 = 22 %** | 2280 s | ⊗ cap (521 dispatched) | 180/807 = 22 % | 19275 s |
+
+→ Figure: [`figs/v2/fig4_linear_vs_lmetric.png`](../../figs/v2/fig4_linear_vs_lmetric.png) ·
+data: [`fig4r_linear.json`](fig4r_linear.json)
+
+**Three clean conclusions from the wall-cap experiment:**
+
+1. **The success-rate ceiling is structural, not a routing artifact.** Linear and
+   LMetric — two very different scoring policies (one session-sticky cache-aware,
+   the other non-sticky pure load) — converge on **identical success rates**
+   (58 / 43 / 22 %) for every PD-disagg ratio. Routing has *zero* effect on the
+   completion ceiling. The bottleneck is the static P:D split itself.
+
+2. **LMetric's longer wall was wall *wasted on requests that will never succeed*.**
+   When the cap is enforced, linear hits the same ceiling in 2280 s as LMetric did
+   in 3300–19000 s — the extra wall just slowly times out the unreachable
+   requests at 600 s each.
+
+3. **The wall-cap is the right way to bench PD-disagg.** Reporting "completion %"
+   without a wall budget is misleading (the bench eventually completes if you wait
+   forever, but only by counting timeouts as failures over hours). The honest
+   metric is **success-in-2×-colo-wall**, which gives the same answer for both
+   routings and matches what an end user would observe on a real SLO.
+
+This **strengthens** the §5 D-pool capacity-ceiling thesis: even with
+session-affinity routing serving every request to a warm prefix cache (which
+*should* maximise PD's throughput), the static D-pool can't admit more than
+~22 / 43 / 58 % of the agentic trace within 2× the colo budget. Colo wins not
+because routing is smarter, but because its **elastic pool** absorbs whichever
+phase is hot — there's no cap to hit.
+
+---
+
+## Reproduce
+
+```bash
+# On dash1, from the main repo /home/admin/cpfs/wjh/agentic-kv:
+for TR in w600_r0.0015_st30.jsonl w600_r0.0015_st30_first600s.jsonl; do
+  TRACE=traces/$TR bash scripts/bench.sh --tag fig4l_lmetric_colo_${TR%.*} \
+    --mode baseline --policy lmetric
+  for r in 6:2 4:4 2:6; do
+    TRACE=traces/$TR bash scripts/bench.sh --tag fig4l_lmetric_${r/:/p}_${TR%.*} \
+      --mode pdsep --pd-ratio $r --policy lmetric
+  done
+done
+
+python microbench/fresh_setup/plot_fig4l_lmetric.py
+python microbench/fresh_setup/plot_fig4_linear_vs_lmetric.py
+```
+
+For the linear + 2× wall-cap variant, run colo first to get `wall_clock_s`,
+compute `CAP=2*wall`, then launch each PD-disagg arm in the background and
+`SIGTERM` it (so bench.sh's cleanup trap fires) once `date +%s` minus the
+arm's start time exceeds `CAP`. The capped runs lack `metrics.summary.json`
+(replayer was killed before it could write); compute the summary directly from
+`metrics.jsonl` (see the inline collector used to build
+`analysis/v2/fig4r_linear.json`).
+
+Source `bench.sh` cleans GPUs before each arm and writes `metrics.jsonl` +
+`metrics.summary.json` per tag. Aggregation script: see the inline JSON dump used
+to build `analysis/v2/fig4l_lmetric.json`.

analysis/v2/fig4l_lmetric.json

@@ -0,0 +1 @@
+[{"tag": "fig4l_lmetric_colo_first600s", "arm": "colo", "trace": "first600s", "n": 807, "req": 807, "e2e": {"count": 807.0, "mean": 11.066699584425269, "p50": 3.27055042097345, "p90": 28.745733462180937, "p99": 97.40008939541167}, "ttft": {"count": 807.0, "mean": 5.119651803458883, "p50": 1.2114678020589054, "p90": 14.777630288852365, "p99": 50.68302261995841}, "tpot": {"count": 807.0, "mean": 0.03004899278845205, "p50": 0.009643197803618922, "p90": 0.042092699501536976, "p99": 0.3919741264067197}, "wall": 1020.5351374909515, "tps": 226.12940164644368}, {"tag": "fig4l_lmetric_colo_full", "arm": "colo", "trace": "full", "n": 1214, "req": 1214, "e2e": {"count": 1214.0, "mean": 10.928977524270508, "p50": 3.1279119075043127, "p90": 30.011970606888667, "p99": 94.77313101590481}, "ttft": {"count": 1214.0, "mean": 5.533819193267678, "p50": 1.017395684029907, "p90": 17.36427243486981, "p99": 51.49416554694993}, "tpot": {"count": 1214.0, "mean": 0.02049970290344434, "p50": 0.009544484575988867, "p90": 0.032480608771520716, "p99": 0.26057810739537074}, "wall": 2993.276069591986, "tps": 125.38402448497122}, {"tag": "fig4l_lmetric_pd2_first600s", "arm": "2P+6D", "trace": "first600s", "n": 180, "req": 807, "e2e": {"count": 180.0, "mean": 380.2505690135715, "p50": 535.6594606440049, "p90": 579.5011055286858, "p99": 601.5567972306756}, "ttft": {"count": 180.0, "mean": 378.7133691522933, "p50": 534.4269686369807, "p90": 577.3534130641376, "p99": 596.404559875431}, "tpot": {"count": 180.0, "mean": 0.007975266077679418, "p50": 0.007166497974743372, "p90": 0.012511071875514153, "p99": 0.017508981961061446}, "wall": 19275.367093455978, "tps": 1.8895100582735462}, {"tag": "fig4l_lmetric_pd2_full", "arm": "2P+6D", "trace": "full", "n": 169, "req": 1214, "e2e": {"count": 169.0, "mean": 194.88523891245458, "p50": 6.817620265996084, "p90": 552.1569225640735, "p99": 595.3934216396092}, "ttft": {"count": 169.0, "mean": 193.4153314989016, "p50": 5.60239192598965, "p90": 549.3611521873856, "p99": 582.4436428000824}, "tpot": {"count": 169.0, "mean": 0.007747395842651413, "p50": 0.007691574401794991, "p90": 0.011201243427351017, "p99": 0.013311375577245894}, "wall": 33770.57413210906, "tps": 0.9869539045920406}, {"tag": "fig4l_lmetric_pd4_first600s", "arm": "4P+4D", "trace": "first600s", "n": 348, "req": 807, "e2e": {"count": 348.0, "mean": 202.63302869595395, "p50": 214.03008900902933, "p90": 477.40967412578175, "p99": 576.6393926549597}, "ttft": {"count": 348.0, "mean": 199.96385804087797, "p50": 213.50966987549327, "p90": 475.7766476540827, "p99": 559.6153268160638}, "tpot": {"count": 348.0, "mean": 0.008873619369764751, "p50": 0.007645836479973812, "p90": 0.013845969236959285, "p99": 0.02567216653158788}, "wall": 6850.181333696004, "tps": 15.00296050477674}, {"tag": "fig4l_lmetric_pd4_full", "arm": "4P+4D", "trace": "full", "n": 533, "req": 1214, "e2e": {"count": 533.0, "mean": 130.94711188977982, "p50": 8.219856544979848, "p90": 473.44134307731883, "p99": 533.2597587251009}, "ttft": {"count": 533.0, "mean": 127.83193208824007, "p50": 4.8246813879814, "p90": 467.54664219671395, "p99": 528.8304683346115}, "tpot": {"count": 533.0, "mean": 0.008886429490232585, "p50": 0.007981476340708988, "p90": 0.013570741891233497, "p99": 0.023050950961825044}, "wall": 13884.384965199977, "tps": 12.621372890425038}, {"tag": "fig4l_lmetric_pd6_first600s", "arm": "6P+2D", "trace": "first600s", "n": 474, "req": 807, "e2e": {"count": 474.0, "mean": 83.15809065495806, "p50": 6.7270191764691845, "p90": 391.6558471220078, "p99": 544.7372293809171}, "ttft": {"count": 474.0, "mean": 80.70155321074382, "p50": 4.1273433425230905, "p90": 390.00296151017517, "p99": 539.0574236416071}, "tpot": {"count": 474.0, "mean": 0.008519881756330928, "p50": 0.00803907146806204, "p90": 0.012583933303093976, "p99": 0.018606097790947705}, "wall": 3325.2749515309697, "tps": 39.705588838364164}, {"tag": "fig4l_lmetric_pd6_full", "arm": "6P+2D", "trace": "full", "n": 793, "req": 1214, "e2e": {"count": 793.0, "mean": 61.907526705667, "p50": 3.69814173609484, "p90": 308.2633092067672, "p99": 477.48038318102715}, "ttft": {"count": 793.0, "mean": 59.25069201986225, "p50": 1.402295546955429, "p90": 302.5604081378088, "p99": 475.3738951798529}, "tpot": {"count": 793.0, "mean": 0.009137289999448822, "p50": 0.008635683270933276, "p90": 0.013065757584108427, "p99": 0.01816783740464599}, "wall": 5662.029295974993, "tps": 39.24494000021532}]

analysis/v2/fig4r_linear.json

@@ -0,0 +1 @@
+[{"tag": "fig4r_linear_colo_first600s", "arm": "colo", "trace": "first600s", "policy": "linear", "n": 807, "req": 807, "dispatched": 807, "e2e": {"count": 807.0, "mean": 8.436370009274967, "p50": 2.5224755640374497, "p90": 22.65510415879542, "p99": 75.54369598095519}, "ttft": {"count": 807.0, "mean": 4.2332503390957195, "p50": 0.8872958200518042, "p90": 11.684667797433207, "p99": 44.98891795879462}, "tpot": {"count": 807.0, "mean": 0.020958194728517718, "p50": 0.00851320761584622, "p90": 0.026440129078245465, "p99": 0.30344440533287176}, "wall": 963.6191155100241, "tps": 239.4857016486815, "capped": false}, {"tag": "fig4r_linear_pd2_first600s", "arm": "2P+6D", "trace": "first600s", "policy": "linear", "n": 176, "req": 807, "dispatched": 521, "e2e": {"count": 176, "mean": 378.5561210460834, "p50": 536.7719694490079, "p90": 583.832092280034, "p99": 601.3415494390065}, "ttft": {"count": 176, "mean": 377.12570991374446, "p50": 536.1157373189926, "p90": 580.3465002350276, "p99": 598.0943597999867}, "tpot": {"count": 176, "mean": 0.007864906140929698, "p50": 0.007212154543958604, "p90": 0.011962352272927423, "p99": 0.017870794738764347}, "wall": 2280, "tps": 14.419736842105262, "capped": true}, {"tag": "fig4r_linear_pd4_first600s", "arm": "4P+4D", "trace": "first600s", "policy": "linear", "n": 349, "req": 807, "dispatched": 577, "e2e": {"count": 349, "mean": 264.8537863784421, "p50": 306.6853819829412, "p90": 488.64622142596636, "p99": 596.5830293919425}, "ttft": {"count": 349, "mean": 262.3163347712099, "p50": 299.75751709297765, "p90": 485.475125996978, "p99": 596.4081599479541}, "tpot": {"count": 349, "mean": 0.010442244895290958, "p50": 0.008213572105774598, "p90": 0.019443845545703716, "p99": 0.028178529054794}, "wall": 2281, "tps": 38.306882946076286, "capped": true}, {"tag": "fig4r_linear_pd6_first600s", "arm": "6P+2D", "trace": "first600s", "policy": "linear", "n": 472, "req": 807, "dispatched": 706, "e2e": {"count": 472, "mean": 118.632779156234, "p50": 12.702161715948023, "p90": 458.1609142010566, "p99": 526.5488834320568}, "ttft": {"count": 472, "mean": 115.80202843308507, "p50": 9.745031949947588, "p90": 455.81679951993283, "p99": 516.5850186559837}, "tpot": {"count": 472, "mean": 0.00950947083585719, "p50": 0.008435572332624966, "p90": 0.015233499645638644, "p99": 0.023447183093280886}, "wall": 2280, "tps": 61.69210526315789, "capped": true}]

analysis/working_set/README.md

@@ -0,0 +1,67 @@
+# KV-cache Working-Set Sizing — GLM-5.1-FP8 · TP=8 · 1× B300 node
+
+工具：`scripts/working_set_analysis.py`（可配置 GPU 型号 / 并行度 TP·PP·EP / 模型 config.json /
+KV dtype / 权重大小）。图：`figs/working_set/glm5_fp8_tp8_b300.png`。
+
+## 复现
+
+```bash
+.venv/bin/python scripts/working_set_analysis.py \
+  /home/gahow/phd/kvcache-simulator/bailian-traces/glm_coder_blksz_512_040915-040917.jsonl \
+  --model-config /home/gahow/phd/kvcache-simulator/models/GLM-5/config.json \
+  --gpu B300 --tp 8 --ep 8 --kv-dtype-bytes 1 --weight-gb 744 --min-ts 0 \
+  --out figs/working_set/glm5_fp8_tp8_b300.png
+```
+
+## 方法
+
+`hash_ids` 是全局内容寻址 block id（同内容=同 id，复用=同 id 再现）。vLLM prefix cache 是
+block 级，所以**集群级 KV footprint = 任一时刻必须常驻的 distinct block 数**，与 placement 无关
+（affinity 只搬运 block，不改总量）。三种 working set：
+- `W_all` 永不淘汰（真上界）
+- `W_oracle` 每 block 只在 `[首次, 末次复用]` 常驻（Belady 完美预知 → 满 APC 上界的最小 HBM）
+- `W_denning(T)` 滑窗 T 内被访问的 distinct block（现实 TTL-LRU）
+
+KV/token：MLA → `L·(kv_lora_rank+qk_rope_head_dim)·dtype`；GQA → `2·L·kv_heads·head_dim·dtype`
+（与 `kvcache-simulator/src/config.rs::kv_block_bytes` 一致）。
+
+## 配置
+
+| 项 | 值 |
+|---|---|
+| 模型 | GLM-5.1-FP8（MLA, L=78, kv_lora=512+rope=64） |
+| KV/token · KV/block(512) | **43.9 KiB** · **23.0 MB**（≈ Qwen3 GQA 96 KiB 的一半） |
+| 硬件 | 8× B300 (288 GB) = 2304 GB HBM/replica |
+| 预算 | FP8 权重 744 GB + act 32 GB → **KV pool = 1528 GB/node** |
+| trace | dash0 glm_coder，475k req，**1.25h active @ 106 QPS**，~40k tok/req（剔除 77 条负 ts 暖机） |
+| APC 上界 | **80.4%** |
+
+## 结果
+
+| 保留窗口 T | peak footprint | = 节点 (GPU) | APC@T |
+|---:|---:|---:|---:|
+| 2s（在飞下限）| 533 GB | 0.3 (3) | 1.7% |
+| 10s | 2,068 GB | 1.4 (11) | 15% |
+| 30s | 4,906 GB | 3.2 (26) | 42% |
+| 60s | 7,698 GB | 5.0 (40) | 56% |
+| 300s | 21,960 GB | 14.4 (115) | 74% |
+| **oracle（满 80.4%）** | **21,399 GB** | **14.0 (112)** | 80.4% |
+| retain-forever | 167,018 GB | 109 (874) | — |
+
+## 结论
+
+1. **Serving：1 节点绰绰有余。** 在飞 KV（τ≈2-5s）仅 533–1157 GB ≪ 单节点 1528 GB。
+   MLA + B300 大 HBM 让 live footprint 微不足道——跑起来根本不缺显存。
+2. **缓存全部复用（80.4%）：1 节点差 ~14×。** oracle 下限 21.4 TB = 14 节点（112 GPU），
+   真实 LRU ~2× → ~28 节点。单节点（1528 GB）只能 hold ~10s 窗口 → cache 侧 APC 仅 ~10-15%。
+   要 ~56% 需 5 节点，~74% 需 ~14 节点。
+3. **瓶颈在长尾，不在 live。** 把 APC 50%→80% 装进 GPU HBM 要 5→14 节点，极不经济
+   → offload/migration 到 CPU DRAM（每节点 ~1.5 TB）是定量动机。与 Qwen 结论方向一致。
+
+## 注意
+
+- footprint 是 TTL-LRU（最浪费）+ shared-cache 下限：真实 capacity-LRU 同容量下 APC 更高，
+  但分区/affinity 不均衡又抬高需求；oracle / retain-forever 给出下/上界。
+- GLM trace mean ~40k tok/req，是 Qwen trace（11k）的 ~3.5×（tokenizer + 抽取不同），
+  **绝对 GB 不可跨模型横比**，方法与定性结论可比。
+- EP 不改变 KV 总量（只影响 expert 权重分布），`--ep` 仅作标注。

analysis/workload_chars/README.md

@@ -0,0 +1,81 @@
+# Agentic workload characterization C1–C3 (full 051315 production trace)
+
+Date 2026-05-29. Source: `trace-glm5.1-formatted/051315-051317.jsonl` on dash1
+(release file, 2,114,220 requests / 1,307,276 sessions / 2h, type=100% `coder`).
+This release file **is the full cluster-level production trace** — session skew
+reproduces 46.5/66.5/74.6/87.5/96.0 exactly. Compute: `compute_chars.py`
+(2-pass, ~65s, `~/ali-trace/.venv` python). Numbers: `chars.json`.
+
+> ⚠️ **Cluster-level, not per-instance.** This is one cluster's aggregate stream.
+> Concurrent-session counts have NO denominator of "8 instances" — do not compare
+> them to a single deployment's instance count.
+
+These three are NOT in the existing 13 analyzer figures (which are single-variable
+marginals on the older 041x traces). C1–C3 are joint/temporal and argument-bearing.
+
+## C1 — the workload is a MIXTURE, not "multi-turn agentic" (`c1_session_mixture.png`)
+
+- **90.3%** of sessions are single-turn; mean 1.62 turns, p99=18, max=3091.
+- But multi-turn sessions (9.7%) = **44.2% of requests** and **66.9% of input
+  (prefill) mass**. Single-turn = **60.2% of output (decode) mass**.
+- Continuation hazard P(reach k+1 | reached k): turn1→2 only **10.2%**, but
+  turn2→3 50.6%, turn5→6 87%, turn12→13 **94.3%** (Lindy / Pareto).
+- Predictability of heaviness at cold-start is near-zero:
+  corr(turn1_input, session_mass)=0.15, corr(turn1_input, n_turns)=**0.04**.
+
+**Routing:** heaviness is unpredictable at session start → proactive placement
+cannot pre-empt hot-pin → a REACTIVE mechanism (observable-load routing /
+migration) is required. But once a session has shown depth, it almost surely
+continues → "observed accumulated load" is the signal that works (not turn-1
+features, not cost-model prediction). The single/multi optimal strategies are
+opposite (load-balance the 90% one-shot sea vs affinity-pin the deep tail) and
+you can't tell them apart at turn 1 → the only viable policy starts everyone
+load-balanced and becomes sticky as turns accrue. This is exactly LPWL's
+emergent behavior (`new_uncached≈input`→by-load; `new_uncached≈0`→sticks), so
+C1 explains *why* a cache-aware-load score is the right shape — it auto-segments
+the mixture with no classifier.
+
+## C2 — marginal work collapses while resident state explodes (`c2_work_amortization.png`)
+
+Per turn: resident context grows 11k→56k+ tokens while new prefill collapses
+2.7k→~200 tokens; per-turn reuse climbs 83%→**99.6%**; resident/new ratio
+("the PD tax") grows to ~250× by turn 12, ~450× by turn 30.
+
+**PD-colocation:** the dominant cost is keeping ~50k+ resident KV available for
+the next turn's tiny delta. Disaggregation physically splits a turn's prefill-KV
+(P) and decode-KV (D), and the next turn's prefix = [prevPrompt + prevAnswer]
+spans both → must be gathered/transferred; colocation keeps it local for free.
+**Routing:** route on delta (`input − cache_hit`), never total input — C2 is the
+trace-level justification for LPWL's score function.
+
+## C3 — prefill/decode BALANCE (honest reframe) (`c3_prefill_decode_balance.png`)
+
+- Token mass: 98.7% input / **1.3% output**; of input, 60% reused-prefix, 40%
+  new-prefill (28.6B new-prefill tokens vs 0.94B decode tokens).
+- **But tokens ≠ time.** Under a per-request latency model (prefill@7k tok/s,
+  TPOT 10ms), aggregate decode-time share ≈ **70% (robust 68–71% across
+  constants)** — each decode token costs ~70–140× a prefill token. So this is
+  NOT a "decode is negligible" workload.
+- Per-request the bottleneck FLIPS within a session: turn-1 (and the 90%
+  single-turn) is prefill-bound; turns ≥3 are strongly decode-bound.
+
+**PD-colocation (correct argument):** the workload has *substantial* work on both
+sides of the roofline — compute-bound prefill (~30% of time) and memory-bound
+decode (~70%). Colocation interleaves them on one GPU (chunked prefill +
+continuous batching) so compute and HBM bandwidth are both used; static
+disaggregation strands P-instances bandwidth-idle and D-instances compute-idle.
+The earlier "decode is 1.3% so nothing to isolate" instinct was WRONG (token vs
+time confusion) — C3b is the correction.
+
+**Caveat:** C3b's 70% is a per-request-latency-weighted estimate; batched decode
+throughput will shift it. Ground-truth needs `-raw.jsonl` (`usage.cached_tokens`
+for exact reuse; `backend_first_response_time_ms` / `total_cost_time_ms` for real
+prefill vs decode wall time). Sampling that 522GB file is the next step.
+
+## Goal mapping
+
+| | argue PD-colocation | guide routing |
+|---|---|---|
+| C1 mixture + hazard | both segments favor colo (diff reasons) | reactive + auto-segment ⇒ LPWL shape |
+| C2 resident/delta | the PD tax (transfer/split resident KV) | route on delta, not total |
+| C3 prefill/decode | roofline complementarity (interleave) | per-req bottleneck flips within session |

analysis/workload_chars/chars.json

@@ -0,0 +1,964 @@
+{
+  "mixture": {
+    "single_sessions": 1179990,
+    "multi_sessions": 127286,
+    "req_single_pct": 55.81207253738968,
+    "req_multi_pct": 44.187927462610325,
+    "in_single_pct": 33.12487590117447,
+    "in_multi_pct": 66.87512409882554,
+    "out_single_pct": 60.24502960903973,
+    "out_multi_pct": 39.75497039096027
+  },
+  "turns": {
+    "mean": 1.6172713336739908,
+    "p99": 18.0,
+    "max": 3091,
+    "single_turn_pct": 90.26326498765371
+  },
+  "hazard": {
+    "1": 0.102101621998721,
+    "2": 0.5062146469376287,
+    "3": 0.7351961756478754,
+    "4": 0.8113739305485657,
+    "5": 0.8723731546954472,
+    "6": 0.8669264241631353,
+    "7": 0.9093235352011023,
+    "8": 0.9240204920989971,
+    "9": 0.901725753553022,
+    "10": 0.9346178826585841,
+    "11": 0.9260597637248089,
+    "12": 0.9427685226874781,
+    "13": 0.91950119395065,
+    "14": 0.936865189289012,
+    "15": 0.9382160896883085,
+    "16": 0.9308646838684262,
+    "17": 0.9371561574269995,
+    "18": 0.9312862196131557,
+    "19": 0.9333279456925813,
+    "20": 0.9351459000779289,
+    "21": 0.9399074074074074,
+    "22": 0.9404984730568416,
+    "23": 0.9473132921336546,
+    "24": 0.9193940734188413,
+    "25": 0.9497294046903187,
+    "26": 0.9323793845764214,
+    "27": 0.9483906016569333,
+    "28": 0.9368466275239868,
+    "29": 0.9472638336900031
+  },
+  "token_mass": {
+    "total_input": 71116829368,
+    "total_output": 940765734,
+    "out_in_ratio_pct": 1.3228454394837104,
+    "new_prefill": 28616906067,
+    "reused_prefix": 42499923301,
+    "new_prefill_pct_of_input": 40.23928839532401
+  },
+  "decode_time_fraction": {
+    "optimistic_for_prefill": 0.6812079219496285,
+    "mid": 0.6970810590484581,
+    "pessimistic": 0.711448473592609
+  },
+  "per_turn": {
+    "turn": [
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11,
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29,
+      30,
+      31,
+      32,
+      33,
+      34,
+      35,
+      36,
+      37,
+      38,
+      39,
+      40,
+      41,
+      42,
+      43,
+      44,
+      45,
+      46,
+      47,
+      48,
+      49,
+      50,
+      51,
+      52,
+      53,
+      54,
+      55,
+      56,
+      57,
+      58,
+      59,
+      60,
+      61,
+      62,
+      63,
+      64,
+      65,
+      66,
+      67,
+      68,
+      69,
+      70,
+      71,
+      72,
+      73,
+      74,
+      75,
+      76,
+      77,
+      78,
+      79,
+      80,
+      81,
+      82,
+      83,
+      84,
+      85,
+      86,
+      87,
+      88,
+      89,
+      90,
+      91,
+      92,
+      93,
+      94,
+      95,
+      96,
+      97,
+      98,
+      99,
+      100,
+      101,
+      102,
+      103,
+      104,
+      105,
+      106,
+      107,
+      108,
+      109,
+      110,
+      111,
+      112,
+      113,
+      114,
+      115,
+      116,
+      117,
+      118,
+      119,
+      120,
+      121,
+      122,
+      123,
+      124,
+      125,
+      126,
+      127,
+      128,
+      129,
+      130,
+      131,
+      132,
+      133,
+      134,
+      135,
+      136,
+      137,
+      138,
+      139,
+      140,
+      141,
+      142,
+      143,
+      144,
+      145,
+      146,
+      147,
+      148
+    ],
+    "med_resident_input": [
+      11035.0,
+      19505.0,
+      28059.0,
+      35089.0,
+      41215.0,
+      44750.0,
+      47419.5,
+      49874.0,
+      51905.0,
+      53068.0,
+      54782.0,
+      56414.0,
+      58229.0,
+      59123.5,
+      60434.5,
+      61320.0,
+      62243.0,
+      63411.0,
+      64510.5,
+      65423.0,
+      66942.5,
+      67965.0,
+      68826.0,
+      70165.5,
+      70052.0,
+      70936.0,
+      71547.0,
+      72648.0,
+      73406.0,
+      73844.0,
+      73604.0,
+      74937.5,
+      74778.0,
+      75460.0,
+      75029.0,
+      74978.0,
+      75933.0,
+      76590.0,
+      74695.0,
+      76813.0,
+      77079.5,
+      78310.0,
+      77848.0,
+      77549.0,
+      78203.0,
+      79102.0,
+      79202.0,
+      78821.0,
+      79868.0,
+      80229.5,
+      80912.0,
+      81620.0,
+      81612.5,
+      81836.5,
+      82506.0,
+      82948.0,
+      82633.0,
+      84107.5,
+      84176.0,
+      84441.0,
+      84101.0,
+      85192.0,
+      84127.0,
+      84783.5,
+      85087.0,
+      85771.5,
+      86110.0,
+      85374.5,
+      87137.0,
+      87677.0,
+      88587.0,
+      88656.0,
+      88882.0,
+      89284.0,
+      91512.0,
+      89850.0,
+      90596.0,
+      91244.0,
+      92102.0,
+      93431.0,
+      92333.5,
+      96682.0,
+      94999.0,
+      95226.5,
+      95173.0,
+      95910.0,
+      96528.0,
+      96508.0,
+      97270.0,
+      97301.0,
+      97076.5,
+      97105.0,
+      98032.0,
+      97962.5,
+      97968.5,
+      98310.0,
+      97061.0,
+      97631.0,
+      100126.0,
+      97765.0,
+      101076.0,
+      98198.5,
+      98678.0,
+      98307.0,
+      99174.0,
+      99882.0,
+      99974.0,
+      99757.0,
+      100065.5,
+      99943.0,
+      100612.0,
+      101138.0,
+      106738.0,
+      99621.0,
+      101980.0,
+      102252.0,
+      103018.0,
+      101238.0,
+      102005.0,
+      101897.0,
+      103576.0,
+      102159.5,
+      102695.5,
+      100590.5,
+      103236.0,
+      101812.0,
+      103074.0,
+      99966.0,
+      102183.5,
+      101882.0,
+      102572.5,
+      105622.5,
+      106066.0,
+      103974.0,
+      105443.5,
+      104716.0,
+      105041.0,
+      106628.0,
+      108320.0,
+      108022.5,
+      107621.5,
+      107664.0,
+      107913.0,
+      108630.0,
+      108382.0,
+      107216.5,
+      105731.0,
+      103986.0
+    ],
+    "med_new_prefill": [
+      11035.0,
+      2920.0,
+      1249.0,
+      767.0,
+      628.0,
+      485.0,
+      400.0,
+      359.0,
+      314.0,
+      274.0,
+      263.0,
+      258.0,
+      244.0,
+      231.0,
+      227.0,
+      222.0,
+      201.0,
+      200.0,
+      198.0,
+      189.0,
+      182.5,
+      184.0,
+      179.0,
+      188.0,
+      173.0,
+      180.0,
+      164.0,
+      167.0,
+      159.5,
+      168.0,
+      156.0,
+      174.0,
+      156.0,
+      159.0,
+      166.0,
+      165.0,
+      153.0,
+      158.0,
+      182.0,
+      149.0,
+      184.0,
+      172.0,
+      149.0,
+      167.0,
+      163.0,
+      152.0,
+      153.0,
+      171.0,
+      151.0,
+      146.0,
+      162.0,
+      153.0,
+      156.0,
+      164.0,
+      148.0,
+      143.0,
+      143.0,
+      149.0,
+      170.5,
+      159.0,
+      144.0,
+      168.0,
+      148.0,
+      144.5,
+      142.5,
+      146.5,
+      147.0,
+      157.0,
+      168.0,
+      153.0,
+      155.0,
+      127.5,
+      145.0,
+      143.0,
+      146.0,
+      123.0,
+      139.0,
+      137.0,
+      115.0,
+      139.5,
+      117.0,
+      154.0,
+      111.0,
+      124.0,
+      118.0,
+      90.0,
+      104.0,
+      116.0,
+      112.0,
+      76.5,
+      110.0,
+      101.0,
+      123.0,
+      114.0,
+      86.0,
+      92.0,
+      108.0,
+      85.0,
+      146.0,
+      77.5,
+      101.0,
+      102.0,
+      85.0,
+      77.0,
+      114.0,
+      66.0,
+      105.0,
+      90.0,
+      89.0,
+      100.0,
+      108.5,
+      100.0,
+      169.0,
+      89.0,
+      106.5,
+      78.0,
+      75.0,
+      90.0,
+      77.0,
+      88.0,
+      102.0,
+      83.5,
+      123.5,
+      116.5,
+      108.0,
+      119.0,
+      82.0,
+      80.0,
+      105.0,
+      90.0,
+      91.0,
+      113.0,
+      122.0,
+      102.0,
+      101.5,
+      64.0,
+      78.0,
+      52.5,
+      98.5,
+      72.0,
+      87.0,
+      102.0,
+      97.0,
+      123.0,
+      80.0,
+      132.5,
+      86.5,
+      111.0
+    ],
+    "med_output": [
+      63.0,
+      67.0,
+      111.0,
+      142.0,
+      158.0,
+      162.0,
+      164.0,
+      164.0,
+      159.0,
+      160.0,
+      159.0,
+      161.0,
+      160.0,
+      158.0,
+      154.0,
+      154.0,
+      154.0,
+      149.0,
+      146.0,
+      147.0,
+      142.0,
+      144.0,
+      143.0,
+      142.0,
+      140.0,
+      136.0,
+      137.0,
+      139.0,
+      136.0,
+      133.0,
+      130.0,
+      131.0,
+      125.0,
+      123.0,
+      122.0,
+      122.0,
+      118.0,
+      122.0,
+      114.0,
+      112.0,
+      115.0,
+      111.0,
+      109.0,
+      112.0,
+      109.0,
+      107.0,
+      111.0,
+      105.0,
+      108.0,
+      107.0,
+      100.0,
+      100.0,
+      95.0,
+      105.0,
+      103.0,
+      102.0,
+      100.0,
+      100.0,
+      98.0,
+      98.0,
+      101.0,
+      99.0,
+      101.0,
+      102.0,
+      97.0,
+      91.0,
+      100.0,
+      97.0,
+      94.0,
+      98.5,
+      92.5,
+      97.0,
+      102.0,
+      92.0,
+      95.0,
+      91.0,
+      91.0,
+      92.0,
+      85.0,
+      98.0,
+      96.0,
+      99.0,
+      94.0,
+      96.0,
+      90.0,
+      85.0,
+      99.0,
+      86.0,
+      99.0,
+      93.0,
+      92.0,
+      93.0,
+      87.0,
+      83.0,
+      87.5,
+      82.0,
+      80.0,
+      90.0,
+      92.0,
+      80.0,
+      77.0,
+      82.0,
+      87.0,
+      74.0,
+      83.0,
+      79.0,
+      84.0,
+      80.5,
+      79.0,
+      76.0,
+      78.5,
+      71.5,
+      81.0,
+      87.0,
+      82.0,
+      85.0,
+      87.0,
+      75.0,
+      75.0,
+      82.0,
+      86.0,
+      76.5,
+      77.5,
+      70.0,
+      78.0,
+      85.0,
+      77.0,
+      67.0,
+      76.5,
+      107.0,
+      92.0,
+      80.5,
+      85.0,
+      83.0,
+      77.0,
+      70.0,
+      84.0,
+      69.0,
+      97.0,
+      72.0,
+      81.0,
+      87.0,
+      89.0,
+      102.0,
+      83.0,
+      82.5,
+      91.0,
+      79.5
+    ],
+    "resident_over_new": [
+      1.0,
+      6.679794520547945,
+      22.46517213771017,
+      45.748370273794,
+      65.62898089171975,
+      92.26804123711341,
+      118.54875,
+      138.92479108635098,
+      165.30254777070064,
+      193.67883211678833,
+      208.29657794676805,
+      218.65891472868216,
+      238.64344262295083,
+      255.94588744588745,
+      266.23127753303964,
+      276.2162162162162,
+      309.6666666666667,
+      317.055,
+      325.81060606060606,
+      346.15343915343914,
+      366.8082191780822,
+      369.375,
+      384.5027932960894,
+      373.22074468085106,
+      404.9248554913295,
+      394.0888888888889,
+      436.2621951219512,
+      435.0179640718563,
+      460.2257053291536,
+      439.54761904761904,
+      471.8205128205128,
+      430.67528735632186,
+      479.34615384615387,
+      474.59119496855345,
+      451.98192771084337,
+      454.41212121212124,
+      496.29411764705884,
+      484.746835443038,
+      410.4120879120879,
+      515.5234899328859,
+      418.9103260869565,
+      455.2906976744186,
+      522.4697986577181,
+      464.36526946107784,
+      479.7730061349693,
+      520.4078947368421,
+      517.6601307189543,
+      460.94152046783626,
+      528.9271523178808,
+      549.5171232876712,
+      499.4567901234568,
+      533.4640522875817,
+      523.1570512820513,
+      499.0030487804878,
+      557.472972972973,
+      580.0559440559441,
+      577.8531468531469,
+      564.4798657718121,
+      493.7008797653959,
+      531.0754716981132,
+      584.0347222222222,
+      507.0952380952381,
+      568.4256756756756,
+      586.7370242214533,
+      597.1017543859649,
+      585.4709897610921,
+      585.7823129251701,
+      543.7866242038217,
+      518.672619047619,
+      573.0522875816994,
+      571.5290322580645,
+      695.3411764705883,
+      612.9793103448276,
+      624.3636363636364,
+      626.7945205479452,
+      730.4878048780488,
+      651.7697841726618,
+      666.014598540146,
+      800.8869565217391,
+      669.7562724014336,
+      789.1752136752136,
+      627.8051948051948,
+      855.8468468468468,
+      767.9556451612904,
+      806.5508474576271,
+      1065.6666666666667,
+      928.1538461538462,
+      831.9655172413793,
+      868.4821428571429,
+      1271.9084967320262,
+      882.5136363636364,
+      961.4356435643564,
+      797.0081300813008,
+      859.3201754385965,
+      1139.1686046511627,
+      1068.5869565217392,
+      898.7129629629629,
+      1148.6,
+      685.7945205479452,
+      1261.483870967742,
+      1000.7524752475248,
+      962.7303921568628,
+      1160.9176470588236,
+      1276.7142857142858,
+      869.9473684210526,
+      1513.3636363636363,
+      952.1333333333333,
+      1108.411111111111,
+      1124.3314606741574,
+      999.43,
+      927.2995391705069,
+      1011.38,
+      631.5857988165681,
+      1119.3370786516855,
+      957.5586854460093,
+      1310.923076923077,
+      1373.5733333333333,
+      1124.8666666666666,
+      1324.7402597402597,
+      1157.9204545454545,
+      1015.4509803921569,
+      1223.4670658682635,
+      831.5425101214574,
+      863.4377682403433,
+      955.8888888888889,
+      855.563025210084,
+      1257.0,
+      1249.575,
+      973.1761904761905,
+      1132.0222222222221,
+      1127.1703296703297,
+      934.712389380531,
+      869.3934426229508,
+      1019.3529411764706,
+      1038.8522167487686,
+      1636.1875,
+      1346.679487179487,
+      2031.009523809524,
+      1099.6954314720813,
+      1500.3125,
+      1237.028735632184,
+      1055.5294117647059,
+      1112.5051546391753,
+      883.170731707317,
+      1354.775,
+      809.1811320754717,
+      1222.3236994219653,
+      936.8108108108108
+    ],
+    "reuse_pct": [
+      0.0,
+      85.02947962061009,
+      95.5486653123775,
+      97.81412978426287,
+      98.47628290670872,
+      98.91620111731844,
+      99.1564651672835,
+      99.28018606889361,
+      99.39504864656584,
+      99.48368131453984,
+      99.5199153006462,
+      99.54266671393626,
+      99.5809648113483,
+      99.60929241333818,
+      99.62438673274372,
+      99.63796477495107,
+      99.67707212055974,
+      99.68459730961506,
+      99.69307322063851,
+      99.71111077144124,
+      99.72737797363409,
+      99.72927241962775,
+      99.73992386598088,
+      99.73206205328829,
+      99.75304059841261,
+      99.74625014097215,
+      99.7707800466826,
+      99.77012443563484,
+      99.78271530937526,
+      99.77249336438979,
+      99.78805499701103,
+      99.76780650542119,
+      99.79138249217684,
+      99.78929234031276,
+      99.77875221580989,
+      99.77993544773133,
+      99.7985065781676,
+      99.7937067502285,
+      99.75634245933462,
+      99.80602241808027,
+      99.76128542608606,
+      99.78036010726599,
+      99.80860137704244,
+      99.78465228436214,
+      99.79156809841055,
+      99.8078430381027,
+      99.80682306002375,
+      99.7830527397521,
+      99.81093804777883,
+      99.81802204924622,
+      99.79978247973106,
+      99.8125459446214,
+      99.8088528105376,
+      99.79960042279423,
+      99.82061910648923,
+      99.82760283551141,
+      99.82694565125313,
+      99.822845762863,
+      99.79744820376354,
+      99.81170284577398,
+      99.82877730348034,
+      99.80279838482487,
+      99.82407550489143,
+      99.82956589430727,
+      99.8325243574224,
+      99.82919734410615,
+      99.82928811984671,
+      99.81610434028896,
+      99.80720015607606,
+      99.82549585410084,
+      99.8250307607211,
+      99.85618570655116,
+      99.83686235683265,
+      99.83983692486895,
+      99.84045808200017,
+      99.86310517529216,
+      99.84657159256479,
+      99.84985314102846,
+      99.87513843347593,
+      99.85069195449047,
+      99.87328542728262,
+      99.84071492108149,
+      99.88315666480699,
+      99.8697841462198,
+      99.87601525642776,
+      99.90616202690022,
+      99.89225924084204,
+      99.87980271065611,
+      99.88485658476407,
+      99.9213779920042,
+      99.88668730331234,
+      99.89598887801864,
+      99.87453076546434,
+      99.88362893964528,
+      99.91221668189266,
+      99.90641847217984,
+      99.88872976787793,
+      99.91293748911718,
+      99.8541837285021,
+      99.92072827699074,
+      99.90007519094543,
+      99.89612875960428,
+      99.91386124566772,
+      99.92167393980083,
+      99.88505051727267,
+      99.93392202799302,
+      99.89497269290015,
+      99.90978076726445,
+      99.91105825684177,
+      99.89994296749147,
+      99.89215998091679,
+      99.90112519527774,
+      99.84166838426802,
+      99.91066140673152,
+      99.89556775838399,
+      99.92371787348903,
+      99.9271971888408,
+      99.91110057488295,
+      99.92451350423998,
+      99.91363828179436,
+      99.90152158801267,
+      99.91826506590185,
+      99.87974156608614,
+      99.8841838941053,
+      99.89538533069859,
+      99.883117903587,
+      99.92044550517105,
+      99.91997279074886,
+      99.89724368415645,
+      99.91166251153295,
+      99.91128226376466,
+      99.89301521929514,
+      99.88497727829841,
+      99.90189855156096,
+      99.9037399175862,
+      99.93888231024867,
+      99.92574328119497,
+      99.95076340173313,
+      99.90906573116692,
+      99.9333472193293,
+      99.91916113415999,
+      99.90526081141329,
+      99.91011277603255,
+      99.88677161005248,
+      99.92618700522226,
+      99.8764182751722,
+      99.91818861071965,
+      99.89325486123131
+    ]
+  }
+}

analysis/workload_chars/compute_chars.py

@@ -0,0 +1,180 @@
+import json, sys, math, statistics as st
+from collections import defaultdict, Counter
+import matplotlib; matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+
+PATH="/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl"
+OUT="/tmp/wlc_out"; import os; os.makedirs(OUT, exist_ok=True)
+BLOCK=512
+# --- transparent cost model for C3 (clearly-labeled estimate; raw-timing validation pending) ---
+PREFILL_TOK_S=7000.0     # MB1: 32k->4.5s ~7100 tok/s effective on H20 / 30B-A3B
+TPOT_S=0.010             # ~10ms/token decode (crossover unloaded ~5ms, loaded ~25ms)
+
+def pct(v,p):
+    if not v: return float('nan')
+    s=sorted(v);k=(len(s)-1)*p;f=int(k)
+    return s[f] if f+1>=len(s) else s[f]+(s[f+1]-s[f])*(k-f)
+
+# ---------- Pass A: structure (scalars only) ----------
+parents={}; recs={}; childcount=Counter()
+for line in open(PATH):
+    if not line.strip(): continue
+    d=json.loads(line); cid=d["chat_id"]; pid=d["parent_chat_id"]
+    parents[cid]=pid
+    recs[cid]=(float(d["timestamp"]),int(d["input_length"]),int(d["output_length"]),int(d["turn"]))
+    if pid!="-1": childcount[pid]+=1
+print(f"[A] records={len(recs)}", file=sys.stderr)
+
+root_of={}
+def root(cid):
+    path=[];c=cid
+    while True:
+        if c in root_of:r=root_of[c];break
+        p=parents.get(c,"-1")
+        if p=="-1" or p not in recs:r=c;break
+        path.append(c);c=p
+    for x in path:root_of[x]=r
+    root_of[cid]=r;return r
+sessions=defaultdict(list)
+for cid in recs: sessions[root(cid)].append(cid)
+seq={r:sorted(m,key=lambda c:(recs[c][3],recs[c][0])) for r,m in sessions.items()}
+print(f"[A] sessions={len(seq)}", file=sys.stderr)
+
+# ---------- C1: mixture + turn tail + hazard ----------
+sr=mr=sm=mm=so=mo=0
+turns_per=[]
+for r,s in seq.items():
+    multi=len(s)>1; turns_per.append(len(s))
+    for c in s:
+        _,inl,outl,_=recs[c]
+        if multi: mr+=1;mm+=inl;mo+=outl
+        else: sr+=1;sm+=inl;so+=outl
+tot_r=sr+mr; tot_in=sm+mm; tot_out=so+mo
+cnt_turn=Counter()
+for r,s in seq.items():
+    for c in s: cnt_turn[recs[c][3]]+=1
+hazard={k: (cnt_turn[k+1]/cnt_turn[k] if cnt_turn[k] else 0) for k in range(1,30)}
+
+# ---------- C2/C3: per-turn resident vs new-prefill (scalar) + hash_ids reuse ----------
+by_in=defaultdict(list); by_new=defaultdict(list); by_out=defaultdict(list)
+by_reuse_hash=defaultdict(list)  # hash-block prefix stability: reused/parent_blocks
+store={}  # cid -> (blockset, in, out) for chats with pending children
+tot_new_prefill=0; tot_reused=0
+for line in open(PATH):
+    if not line.strip(): continue
+    d=json.loads(line); cid=d["chat_id"]; pid=d["parent_chat_id"]
+    inl=int(d["input_length"]); outl=int(d["output_length"]); turn=int(d["turn"])
+    blocks=set(d["hash_ids"])
+    if pid in store:
+        pblk,pin,pout=store[pid]
+        new_prefill=max(0, inl - pin - pout)              # actual recompute (accounts for cached answer)
+        reused_blk=len(blocks & pblk)
+        by_reuse_hash[turn].append(reused_blk/len(pblk) if pblk else 0)
+        childcount[pid]-=1
+        if childcount[pid]<=0: del store[pid]
+        tot_reused += (inl-new_prefill)
+    else:
+        new_prefill=inl                                    # session start: all new (intra-session)
+    tot_new_prefill+=new_prefill
+    by_in[turn].append(inl); by_new[turn].append(new_prefill); by_out[turn].append(outl)
+    if childcount[cid]>0: store[cid]=(blocks,inl,outl)
+print(f"[B] done; store residual={len(store)}", file=sys.stderr)
+
+TURNS=[t for t in sorted(by_in) if len(by_in[t])>=50]
+med_in=[pct(by_in[t],.5) for t in TURNS]
+med_new=[max(pct(by_new[t],.5),1) for t in TURNS]
+med_out=[pct(by_out[t],.5) for t in TURNS]
+ratio=[med_in[i]/med_new[i] for i in range(len(TURNS))]
+reuse_pct=[(1-med_new[i]/med_in[i])*100 for i in range(len(TURNS))]
+# C3 time per turn (cost model)
+t_pref=[med_new[i]/PREFILL_TOK_S for i in range(len(TURNS))]
+t_dec=[med_out[i]*TPOT_S for i in range(len(TURNS))]
+
+# aggregate decode/prefill time fraction over a RANGE of constants
+def agg_time(prate,tpot):
+    tp=tot_new_prefill/prate; td=tot_out*tpot; return td/(tp+td)
+frac_lo=agg_time(13000,0.005); frac_mid=agg_time(7000,0.010); frac_hi=agg_time(3000,0.025)
+
+chars={
+ "mixture":{"single_sessions":sr if False else sum(1 for s in seq.values() if len(s)==1),
+   "multi_sessions":sum(1 for s in seq.values() if len(s)>1),
+   "req_single_pct":sr/tot_r*100,"req_multi_pct":mr/tot_r*100,
+   "in_single_pct":sm/tot_in*100,"in_multi_pct":mm/tot_in*100,
+   "out_single_pct":so/tot_out*100,"out_multi_pct":mo/tot_out*100},
+ "turns":{"mean":st.mean(turns_per),"p99":pct(turns_per,.99),"max":max(turns_per),
+   "single_turn_pct":sum(1 for x in turns_per if x==1)/len(turns_per)*100},
+ "hazard":hazard,
+ "token_mass":{"total_input":tot_in,"total_output":tot_out,"out_in_ratio_pct":tot_out/tot_in*100,
+   "new_prefill":tot_new_prefill,"reused_prefix":tot_reused,
+   "new_prefill_pct_of_input":tot_new_prefill/tot_in*100},
+ "decode_time_fraction":{"optimistic_for_prefill":frac_lo,"mid":frac_mid,"pessimistic":frac_hi},
+ "per_turn":{"turn":TURNS,"med_resident_input":med_in,"med_new_prefill":med_new,
+   "med_output":med_out,"resident_over_new":ratio,"reuse_pct":reuse_pct},
+}
+json.dump(chars, open(f"{OUT}/chars.json","w"), indent=2)
+
+# ================= FIGURES =================
+plt.rcParams.update({"figure.dpi":140,"font.size":10,"axes.grid":True,"grid.alpha":.3})
+
+# ---- C1 ----
+fig,ax=plt.subplots(1,3,figsize=(15,4.2))
+cats=["% sessions","% requests","% input\ntokens","% output\ntokens"];
+singv=[chars["mixture"]["single_sessions"]/len(seq)*100, chars["mixture"]["req_single_pct"],
+       chars["mixture"]["in_single_pct"], chars["mixture"]["out_single_pct"]]
+multv=[100-x for x in singv]
+x=np.arange(len(cats))
+ax[0].bar(x,singv,label="single-turn",color="#7fb3d5")
+ax[0].bar(x,multv,bottom=singv,label="multi-turn",color="#e74c3c")
+for i in range(len(cats)):
+    ax[0].text(i,singv[i]/2,f"{singv[i]:.0f}",ha="center",va="center",fontsize=9)
+    ax[0].text(i,singv[i]+multv[i]/2,f"{multv[i]:.0f}",ha="center",va="center",color="white",fontsize=9)
+ax[0].set_xticks(x);ax[0].set_xticklabels(cats);ax[0].set_ylabel("%");ax[0].set_ylim(0,100)
+ax[0].set_title("C1a Mixture: 90% sessions single-turn,\nbut multi-turn carries 2/3 prefill mass");ax[0].legend(loc="center right")
+# turn CCDF log-log
+tc=sorted(turns_per); n=len(tc); xs=sorted(set(tc))
+ccdf=[sum(1 for v in tc if v>=xx)/n for xx in xs]
+ax[1].loglog(xs,ccdf,marker=".",ms=3,color="#34495e")
+ax[1].set_xlabel("turns per session (k)");ax[1].set_ylabel("P(turns >= k)")
+ax[1].set_title(f"C1b Heavy-tailed session length\n(p99={chars['turns']['p99']:.0f}, max={chars['turns']['max']})")
+# hazard
+hk=list(range(1,20)); hv=[hazard[k]*100 for k in hk]
+ax[2].plot(hk,hv,marker="o",color="#16a085")
+ax[2].set_xlabel("reached turn k");ax[2].set_ylabel("P(continue to k+1) %");ax[2].set_ylim(0,100)
+ax[2].set_title("C1c Continuation hazard rises 10%->94%\n(unpredictable at start, Lindy after)")
+fig.tight_layout(); fig.savefig(f"{OUT}/c1_session_mixture.png"); plt.close(fig)
+
+# ---- C2 ----
+fig,ax=plt.subplots(1,3,figsize=(15,4.2))
+ax[0].semilogy(TURNS,med_in,marker="o",label="resident context (input)",color="#e74c3c")
+ax[0].semilogy(TURNS,med_new,marker="s",label="new prefill this turn",color="#2980b9")
+ax[0].set_xlabel("turn");ax[0].set_ylabel("tokens (median, log)");ax[0].legend()
+ax[0].set_xlim(1,30)
+ax[0].set_title("C2a Resident state explodes,\nmarginal work collapses")
+ax[1].plot(TURNS,ratio,marker="o",color="#8e44ad")
+ax[1].set_xlabel("turn");ax[1].set_ylabel("resident / new-prefill");ax[1].set_xlim(1,30)
+ax[1].set_title("C2b The PD tax = resident/delta\n(grows to ~250x by deep turns)")
+ax[2].plot(TURNS,reuse_pct,marker="o",color="#27ae60")
+ax[2].set_xlabel("turn");ax[2].set_ylabel("per-turn reuse %");ax[2].set_ylim(50,100);ax[2].set_xlim(1,30)
+ax[2].set_title("C2c Per-turn reuse climbs to 99.6%\n(deep turns are near-pure cache hits)")
+fig.tight_layout(); fig.savefig(f"{OUT}/c2_work_amortization.png"); plt.close(fig)
+
+# ---- C3 ----
+fig,ax=plt.subplots(1,2,figsize=(11,4.4))
+# token mass decomposition
+vals=[tot_reused/1e9, tot_new_prefill/1e9, tot_out/1e9]
+labs=[f"reused prefix\n{tot_reused/tot_in*100:.0f}% of input",
+      f"new prefill\n{tot_new_prefill/tot_in*100:.0f}% of input",
+      f"decode output\n{tot_out/tot_in*100:.1f}% of input"]
+ax[0].bar(range(3),vals,color=["#95a5a6","#2980b9","#e67e22"])
+ax[0].set_xticks(range(3));ax[0].set_xticklabels(labs,fontsize=8.5)
+ax[0].set_ylabel("tokens (billions)")
+ax[0].set_title("C3a Token mass: prefill-dominated\n(but tokens != time, see C3b)")
+# per-turn prefill vs decode TIME (cost model)
+ax[1].semilogy(TURNS,t_pref,marker="o",label="prefill time (new tok / 7k·s⁻¹)",color="#2980b9")
+ax[1].semilogy(TURNS,t_dec,marker="s",label="decode time (out·10ms)",color="#e67e22")
+ax[1].set_xlabel("turn");ax[1].set_ylabel("seconds (median, log)");ax[1].legend(fontsize=8);ax[1].set_xlim(1,30)
+ax[1].set_title(f"C3b Prefill→decode bottleneck flips within a session\n(agg decode-time share ≈ {frac_mid*100:.0f}%, range {frac_lo*100:.0f}–{frac_hi*100:.0f}%)")
+fig.tight_layout(); fig.savefig(f"{OUT}/c3_prefill_decode_balance.png"); plt.close(fig)
+print("FIGURES + chars.json written to", OUT)
+print(json.dumps({k:chars[k] for k in ["mixture","turns","token_mass","decode_time_fraction"]}, indent=2))

figs/mb5/CORRECTION.md

@@ -0,0 +1,19 @@
+# ⚠️ Correction notice for figs/mb5/ (2026-05-30)
+
+These figures back `microbench/fresh_setup/PD_DISAGG_RESULTS.md`. A producer-side
+contamination was later found in the stack that produced them: commit **`e13391e`**
+(deployed over the "fresh" pip vLLM by `scripts/deploy_vllm_patches.sh`) evicts a
+producer's prefix-cache blocks on every KV transfer, so a disaggregated producer
+could never keep a session's prefix warm. It is now gated behind
+`VLLM_EVICT_SENT_BLOCKS` (default off) and everything was re-run clean.
+
+| figure | section | status |
+|---|---|---|
+| `mb5_producer_hotspot.png` | §6.3 session-affinity hot-pinning | 🛑 **RETRACTED** — pure `e13391e` artifact. On the clean stack, session-routed producers reach APC parity with colo (71–82%); there is no 0%-util stall / hot-pin pathology. |
+| `mb5_latency_compare.png` | §3 round-robin headline | ✅ stands — RR's ~0% prefix-hit is a *routing* artifact (consecutive turns → different producers), not the eviction bug; reproduced clean. |
+| `mb5_kv_timeline.png`, `mb5_role_split.png`, `mb5_peak_utilization.png` | §5 per-role KV pool occupancy | ✅ D-pool capacity-ceiling mechanism stands (decode pegs while prefill strands). P-pool occupancy may read slightly low under eviction; the qualitative split is unaffected. |
+| `mb5_summary.csv` | aggregate | mixed — §3/§5 rows valid; any session-affinity rows superseded. |
+
+**Superseded by the corrected three-axis ablation:** [`../mb5_pd_ablation/`](../mb5_pd_ablation/)
+(reuse / shape / concurrency), data in [`../../analysis/mb5_pd_ablation/`](../../analysis/mb5_pd_ablation/).
+Raw §6 data `analysis/mb5/session_prod.json` is contaminated; `analysis/mb5/rr_prod.json` (round-robin) stands.

figs/mb5/mb5_summary.csv

@@ -0,0 +1,5 @@
+config,rep,n_requests,n_success,wall_clock_s,peak_pool_frac,steady_pool_frac,p_pool_peak_frac,p_pool_steady_frac,d_pool_peak_frac,d_pool_steady_frac,peak_waiting,latency_p50_s,latency_p90_s,latency_p99_s,ttft_p50_s,ttft_p90_s,ttft_p99_s,prefix_cache_hit_ratio
+8C,1,1214,1214,2994.218414353032,0.7174957362137578,0.3439702956225128,,,,,29,10.82550932947197,83.34998885790122,194.10265863158946,6.967104309005663,53.12018221841427,114.12611859919207,0.1937163528742694
+6P+2D,1,1214,1214,3419.065942236979,0.7726478112563957,0.42145750426378625,0.743272692817889,0.3082291074474133,0.9959636156907333,0.7434906196702672,128,44.48975181748392,91.82252187062406,147.70196208347772,40.95952733900049,86.68752026481089,142.84028979733685,0.0
+4P+4D,1,1214,1214,4170.666486939997,0.6997939169982945,0.45876918703808983,0.6438459351904491,0.28540363843092664,0.9753411028993746,0.5977686185332576,152,59.52004547297838,157.08703426021387,224.03997302683115,56.419772224500775,153.07864206891392,219.73412787001706,0.0
+2P+6D,1,1214,109,5761.816568834998,0.9698692438885731,0.9435119386014781,0.9969869243888573,0.9198408186469585,0.9620238772029562,0.9494504453287853,872,26.293884326005355,499.3484142678091,577.7122636228032,23.580788671970367,498.0334587502061,576.5306194114453,0.0

microbench/connector_tax/cache_sweep/MIGRATION_TRANSFER_COST.md

@@ -0,0 +1,178 @@
+# Why KV-transfer is slow during migration under real load
+
+**Question.** EAR's unified+A+B routing beats migration (v3) on agentic
+workloads. We wanted to know whether *layerwise* KV transfer would shrink
+migration's overhead enough to make it viable. Investigating that led to a
+sharper question: **in a real (loaded) cluster, when we migrate, the KV
+transfer is already slow — the effective bandwidth is far below the
+~10 GB/s wire rate. Why?**
+
+This doc answers that with instrumented measurements.
+
+**TL;DR.** Migration fires precisely when instances are *busy* (that's the
+trigger). But on a busy instance, KV transfer runs at **~3 GB/s instead of
+~10 GB/s**, because:
+
+1. **The RDMA write itself slows ~2× under compute load** — GPU-direct RDMA
+   (`batch_transfer_sync_write`) contends with the running attention/MLP
+   kernels for **HBM and PCIe bandwidth**. (idle 7.6 GB/s → busy 4.0 GB/s)
+2. **The connector's Python control plane gets GIL-starved** — mooncake's
+   ZMQ handshake + transfer orchestration run on asyncio threads inside the
+   engine process; when the engine's main thread is doing a long forward
+   pass (e.g. a 100k-token prefill), those threads stall for *seconds*.
+
+Both are **inherent to upstream vLLM 0.18.1 + mooncake** (reproduced on a
+clean fresh venv; the transfer path is byte-identical to upstream — our
+patches did not cause this), and both get **worse**, not better, with
+layerwise transfer. So the bandwidth gap is not a layerwise problem; it is a
+*transfer-on-a-busy-GPU* problem.
+
+---
+
+## 1. Evidence chain
+
+Three independent measurements, all on dash0 (8×H100, Qwen3-Coder-30B-A3B,
+TP=1), Mooncake `kv_both`.
+
+### 1a. Instrumented v3 trace replay — where does migration time go?
+
+Run `outputs/b3_v3_fullbreak_20260528_0338/`. Instruments:
+`instrument_dst_migration.py` (dst scheduler lifecycle) +
+`instrument_mooncake.py` (connector internals: `send_blocks` RDMA,
+`receive_kv` window, `ready_wait`).
+
+25 migrations fired over the trace. Dst-side migration overhead
+(`T_kv_pull` = scheduler marks `WAITING_FOR_REMOTE_KVS` → `finished_recving`):
+
+| component | share | what it is |
+|---|---:|---|
+| RDMA-actual (`batch_transfer_sync_write`) | **55%** (55.2 s) | the real RDMA write |
+| dst control-plane gap | **45%** (45.4 s) | scheduler↔receiver_loop dispatch + completion propagation |
+| `ready_wait` (src KV not committed) | 0% | 25/25 already committed — **ruled out** |
+
+- Pure RDMA aggregate rate: **2.03 GB/s** (vs MB2 idle 9.7 GB/s).
+- RDMA rate **collapses with transfer size**: <3 GiB → 4–9.5 GB/s,
+  >5 GiB → 0.9–2.6 GB/s.
+- The control-plane gap is **bimodal**: median 0.04 s, but a handful of
+  requests stall ~10 s. Those are small-KV transfers (0.18 s of actual RDMA)
+  whose `T_kv_pull` is 8–11 s — i.e. the dst's `receiver_loop` thread was
+  GIL-starved for ~10 s while the engine did a big forward pass.
+
+> Earlier (pre-instrumentation) we wrongly attributed ~90% of migration
+> overhead to "dst scheduler queueing" by estimating transfer at clean wire
+> speed. With real instrumentation, dst *scheduler admission* is ~0
+> (`T_admission_post_kv` = 0.003 s); the time is the transfer phase (RDMA +
+> connector control plane), both degraded by instance busy-ness.
+
+### 1b. MB6 controlled microbench — does busy-ness cause it?
+
+`microbench/fresh_setup/mb6_transfer_under_load.py` + `run_mb6.sh`: 2
+instances, transfer a fixed-size KV (prefill on A → migrate to B) while
+holding *N* background decode streams on both. Sweep N.
+
+Effective transfer bandwidth (65k-token KV ≈ 6 GiB), main venv:
+
+| background load | 65k transfer | eff bandwidth |
+|---|---:|---:|
+| **0 (idle)** | 747 ms | **8.76 GB/s** |
+| 8 (4/instance) | 2423 ms | 4.53 GB/s |
+| **24 (12/instance)** | 2015 ms | **3.33 GB/s** |
+
+Monotonic degradation with load. **The busy level (3.3 GB/s) matches the
+v3 trace's 3.3 GB/s median exactly** — because agentic instances run
+~10+ concurrent requests, i.e. the bg=24 regime.
+
+Decomposing the 65k transfer into RDMA-actual vs control-plane:
+
+| bg | RDMA rate | control-plane share |
+|---|---:|---:|
+| 0 (idle) | 7.56 GB/s | 13% |
+| 8 | 4.07 GB/s | 11% |
+| 24 (busy) | 3.97 GB/s | 15% |
+
+In the clean microbench the **RDMA write itself is the dominant degrading
+term** (7.6 → 4.0 GB/s). The ~10 s control-plane stalls seen in the trace
+(1a) don't reproduce here because steady decode forward passes are short;
+they require the long (100k-token) prefills that the real trace has.
+
+### 1c. Fresh-venv comparison — is it our patch?
+
+Same MB6 sweep on `agentic-kv-fresh/.venv` (clean upstream-style 0.18.1):
+
+| bg | 65k eff (fresh) | 65k eff (main/patched) |
+|---|---:|---:|
+| 0 | 8.73 GB/s | 8.76 GB/s |
+| 8 | 4.52 GB/s | 4.53 GB/s |
+| 24 | 3.27 GB/s | 3.33 GB/s |
+
+**Identical within noise.** Plus a static check: the v3 transfer path
+(`send_kv_to_decode`, `_send_blocks`/`batch_transfer_sync_write`,
+`_build_transfer_params`) is **byte-identical** to pristine upstream 0.18.1
+(commit `445e491`); `receive_kv_from_single_worker` differs only by a 4-line
+error branch. Our mooncake commits (`a7df84b` direct-read,
+`ea51497` partial-prefill, `e3a1d70` read→push) only touch a *separate*
+`direct_read` path that v3 does **not** use (v3 requests carry no
+`direct_read` flag → normal push path).
+
+→ **The slowdown is upstream/hardware-inherent, not introduced by us.**
+
+---
+
+## 2. Root cause
+
+Migration in agentic serving transfers KV **between instances that are
+concurrently busy with compute** — by construction, since v3 migrates *away
+from* a busy host. On a busy instance:
+
+- **HBM/PCIe contention (the dominant, irreducible part).** Mooncake's
+  transfer is GPU-direct RDMA: the NIC DMAs KV straight out of / into GPU
+  HBM. While the GPU runs attention+MLP kernels, those kernels saturate HBM
+  bandwidth, so the NIC's RDMA gets a smaller slice. Effective transfer
+  bandwidth roughly halves (7.6 → 4.0 GB/s at our load), and degrades
+  further for large multi-segment transfers.
+- **Control-plane GIL starvation (secondary, bursty).** The connector runs
+  its ZMQ handshake + `send_kv_to_decode`/`receive_kv` orchestration on
+  asyncio threads (`sender_loop`/`receiver_loop`) *inside the engine
+  process*. A long forward pass (100k-token prefill) holds the GIL for
+  seconds, stalling those threads → multi-second dispatch gaps even when the
+  actual transfer is 0.2 s.
+
+MB2 measured 9.7 GB/s precisely because both endpoints were **idle**. The
+real-workload gap is the difference between "idle benchmark" and "transfer
+while the GPU is doing the day job."
+
+---
+
+## 3. Implication: layerwise is the wrong lever; migration's tax is largely irreducible
+
+| lever | effect on the gap |
+|---|---|
+| **Model-level layerwise transfer** (push each layer's KV during prefill) | **Worse.** Prefill is the most HBM-intensive phase, so per-layer transfers contend *harder* for HBM (Cause 1); and they multiply the control-plane round-trips (Cause 2). |
+| **Control-plane fix** (move mooncake orchestration off the GIL-contended threads / separate process) | Addresses only the bursty ~10 s stalls (~15% in the clean case, up to ~45% of the trace tail). Does **not** touch the HBM-contention half. |
+| **Reduce bytes** (cache-aware target so less KV moves) | Helps linearly; v3 Mechanism B already does some. Orthogonal. |
+| **Migrate to/from idle instances** | Would restore ~10 GB/s — but defeats the purpose (we migrate *because* the host is busy). |
+
+The dominant cost (RDMA contending with compute for HBM on busy instances)
+is a **hardware reality**, not a software bug we can patch away, and not
+something layerwise improves. This reinforces
+[UNIFIED_ABLATION.md](UNIFIED_ABLATION.md): the unified no-migration path
+(A+B'+RaceFix) remains the right default; migration's transfer tax is
+structural on a loaded agentic cluster.
+
+---
+
+## 4. Repro / artifacts
+
+- Instrumented v3 breakdown: `outputs/b3_v3_fullbreak_20260528_0338/unified_v3/`
+  (`transfer_decomp.txt`, `dst_migration_breakdown.{csv,png}`,
+  `transfer_rootcause.png`)
+- MB6 main: `outputs/mb6_agentic-kv_20260528_0552/mb6_result.json`
+- MB6 fresh: `outputs/mb6_fresh_20260528_0559/mb6_result.json`
+- Instruments: `microbench/fresh_setup/instrument_dst_migration.py`,
+  `microbench/fresh_setup/instrument_mooncake.py`
+- Microbench: `microbench/fresh_setup/mb6_transfer_under_load.py` +
+  `run_mb6.sh` (`VENV=… bash run_mb6.sh`)
+- Analyzers: `analyze_dst_migration.py`, `analyze_transfer_decomp.py`
+
+All instruments apply/revert cleanly via `--apply`/`--revert`; both venvs
+were restored after the runs.

microbench/connector_tax/cache_sweep/UNIFIED_ABLATION.md

@@ -0,0 +1,431 @@
+# Unified routing ablation: A (tighter affinity) + B (decode-aware LMetric)
+
+Goal: judge whether `unified` (cache-aware hybrid affinity + LMetric fallback)
+has enough headroom to surpass v3 migration-based routing on agentic
+workloads, without invoking PD-sep migration.
+
+## Workload / baseline
+
+- Trace: `w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions)
+- Hardware: 8 × H100 (dash0), Qwen3-Coder-30B-A3B, TP=1, max_model_len=200000
+- Trace replay through `cache_aware_proxy.py` with policy `unified`
+- `b3_replay_20260527_0114/unified/` reference
+
+| Metric (ms) | baseline (`overload_factor=2.0`) |
+|---|---:|
+| TTFT p50 | 520 |
+| TTFT p90 | **8781** |
+| TTFT p99 | 47647 |
+| TPOT p90 | 17.8 |
+| E2E p90 | 19989 |
+| E2E p99 | 85841 |
+
+Reference points we're trying to beat / match:
+- v3 fixed rotation (cache-blind picker): TTFT p90 = 10828
+- v3 + Mechanism B (cache-rich picker): TTFT p90 = 9711
+- All v3 variants are +10–23% worse than `unified` baseline.
+
+## Tail-source diagnostic on baseline
+
+Decision split, baseline unified:
+
+| Decision | n | TTFT mean | TTFT p90 | TTFT p99 |
+|---|---:|---:|---:|---:|
+| affinity     | 852 | 3183 | 7011 | 47432 |
+| lmetric_fallback | 362 | 4285 | 12083 | 46036 |
+
+Long-tail (>20s, n=65):
+- 40 / 65 came from `affinity` decisions
+- 25 / 65 came from `lmetric_fallback`
+
+For the 40 slow `affinity` reqs:
+- only 12 / 40 were actually overloaded at decision time (`aff_num_req > avg_num_req`)
+- overload ratio at decision: mean=0.93, p50=0.87
+- **most slow affinity reqs looked fine when the picker stuck — load piled
+  on after dispatch**.
+
+This is a snapshot-based-routing limitation. Tightening
+`overload_factor` only helps the genuine cases above the new threshold —
+expected to be a 5-10% improvement at best.
+
+---
+
+## Direction A — tighten affinity overflow
+
+**Hypothesis.** `overload_factor=2.0` lets the picker stick to affinity
+even when `affinity.num_req` is up to 2× the cluster average. Reducing to
+1.3 forces earlier overflow to LMetric fallback, escaping busy affinity
+hosts before the tail blows up.
+
+**Change.** Single CLI flag: `--overload-factor 1.3`. No code change.
+
+**Run.** `unified_of13_20260527_1532/unified/`.
+
+### A vs baseline
+
+| Metric (ms) | baseline (of=2.0) | A (of=1.3) | Δ |
+|---|---:|---:|---:|
+| TTFT p50 | 520 | 495 | −5% |
+| TTFT p90 | 8781 | 8730 | ≈0 |
+| TTFT p99 | 47647 | 43059 | −10% |
+| TPOT p50 | 7.9 | 8.0 | ≈0 |
+| TPOT p90 | 17.8 | **15.5** | **−13%** |
+| E2E p50 | 1761 | 1824 | +4% |
+| E2E p90 | 19989 | 18407 | −8% |
+| E2E p99 | 85841 | **71396** | **−17%** |
+
+TTFT p90 is essentially unchanged but the **deeper tail (p99) and
+TPOT both improved meaningfully**. Net: A alone gives roughly −10% to
+−17% on the long tail without hurting medians.
+
+### Decision split, A vs baseline
+
+| Decision | baseline n / p90 | A n / p90 | Δ p90 |
+|---|---|---|---|
+| affinity | 852 / 7011 | 817 / **5817** | **−17%** ✅ |
+| lmetric_fallback | 362 / 12083 | 397 / **15360** | **+27%** ⚠️ |
+
+The picker now sticks to affinity 35 fewer times. The remaining affinity
+decisions are higher-quality (no longer "barely-fitting" cases), so their
+p90 drops 17%.
+
+But the 35 extra reqs that got pushed into fallback **got slower**:
+fallback p90 went from 12083 → 15360. The LMetric scorer is selecting a
+worse instance for them.
+
+### Per-worker TTFT under A (of=1.3)
+
+```
+port 8000: n=  94  mean=4424  p90=12290    port 8004: n=192  mean=2597  p90=6968
+port 8001: n= 135  mean=2779  p90= 5553    port 8005: n=202  mean=3102  p90=6113
+port 8002: n=  88  mean=5827  p90=15804    port 8006: n=136  mean=4006  p90=10899
+port 8003: n= 217  mean=2674  p90= 4598    port 8007: n=150  mean=3648  p90= 7025
+```
+
+Compared to baseline (88..217 reqs/port), A redistributes more evenly
+(88..217 still but distribution is fatter in the middle). port 8002
+remains slow (p90 15.8s) — its cache pool seems to keep getting cold
+work routed there by LMetric.
+
+### Why A alone isn't enough
+
+LMetric scorer (`unified_hybrid` fallback path):
+
+```python
+score = (pending_prefill_tokens + new_uncached_tokens) * num_requests
+```
+
+This **ignores `ongoing_decode_tokens`** entirely. An instance with no
+pending prefill but 200k tokens currently in decode looks "ideal"
+(score=0×num_req=0) — yet a new request landing there waits behind
+slow decode iters caused by the large batch KV reads.
+
+A pushes more requests into fallback, but fallback can't tell which
+instance is actually free. → Direction B is mandatory companion.
+
+---
+
+## Direction B — decode-aware LMetric
+
+**Hypothesis.** Adding a decode-load penalty to the LMetric score lets
+fallback distinguish "no prefill queued but heavy decode running" from
+"truly idle". Should restore fallback p90 ≤ 12s baseline level.
+
+**Change.**
+```python
+score = (pending_prefill + new + lmetric_decode_weight * ongoing_decode_tokens) * num_requests
+```
+- `lmetric_decode_weight=0.0` ⇒ original LMetric (control)
+- `lmetric_decode_weight=0.01` ⇒ first experiment (rationale: 1 decode token
+  in batch costs ~0.01 prefill-token-equivalent in scheduler iter time
+  on H100 + Qwen3-30B-A3B)
+
+CLI: `--lmetric-decode-weight 0.01`. Setting in code:
+`cache_aware_proxy.py:Settings.lmetric_decode_weight`.
+
+**Run.** `unified_of13_lmw001_20260527_1628/unified/`.
+
+### A+B vs baseline / A
+
+| Metric (ms) | baseline | A (of=1.3) | A+B (of=1.3, lmw=0.01) | Δ vs baseline |
+|---|---:|---:|---:|---:|
+| TTFT p50 | 520 | 495 | 514 | −1% |
+| **TTFT p90** | 8781 | 8730 | **8421** | **−4%** ✅ |
+| TTFT p99 | 47647 | 43059 | 44800 | −6% |
+| TPOT p50 | 7.9 | 8.0 | 7.9 | ≈0 |
+| TPOT p90 | 17.8 | 15.5 | 15.7 | −12% |
+| E2E p50 | 1761 | 1824 | 1870 | +6% |
+| E2E p90 | 19989 | 18407 | **21064** | **+5%** ⚠️ |
+| E2E p99 | 85841 | 71396 | **64344** | **−25%** ✅ |
+
+Long-tail counts:
+
+```
+thresh       baseline      A      A+B   v3 MechB
+>  5000ms        170      173      170        177
+> 10000ms        105      109      109        119
+> 20000ms         65       64       59         78
+> 30000ms         41       40       37         50
+> 50000ms          8        5        6         14
+```
+
+A+B is best on every long-tail-count threshold ≤30s, marginal worse at 50s.
+
+### Decision split (A+B vs A)
+
+| Decision | A (of=1.3) | A+B | Note |
+|---|---|---|---|
+| affinity p90 | 5817 | 5836 | ≈ same |
+| fallback p90 | **15360** | **13501** | B recovered some of A's fallback regression |
+
+B partially fixed fallback's selection (−12% on fallback p90 vs A alone),
+but still worse than baseline (12083).
+
+### Per-worker TTFT (A+B)
+
+```
+port 8000: n=134  mean=3495  p90=10967      port 8004: n=136  mean=3102  p90= 7906
+port 8001: n=143  mean=2981  p90=10189      port 8005: n=179  mean=1624  p90= 2735
+port 8002: n=221  mean=2355  p90= 3502      port 8006: n=137  mean=5356  p90= 9628
+port 8003: n=146  mean=3932  p90=10729      port 8007: n=118  mean=5210  p90=26798  ← new hotspot
+```
+
+A+B trades the baseline's 8002 hotspot (p90=35s) for a new 8007 hotspot
+(p90=26.8s). Lower amplitude but hotspot survives.
+
+### Why 8007 became a hotspot under A+B — **found a bug in B**
+
+8007 in A+B: 118 reqs, **53% affinity / 47% fallback** (vs other ports
+60–77% affinity), **cache_hit_mean=50.5% (lowest)**.
+
+Top-10 slowest at 8007: all are big-prompt (100k+ tokens) fallback decisions
+with `cached_tokens=0` (cold prefill). LMetric is pushing many cold-prefill
+fallbacks to 8007.
+
+Looking at the B formula:
+
+```python
+decode_pen = lmetric_decode_weight * ongoing_decode_tokens
+score = (pending_prefill + new + decode_pen) * num_requests   # ← BUG
+```
+
+When `num_requests = 0`, the entire score (including decode penalty) zeros
+out. So an idle-but-decoding host (num_req=0 because its last prefill
+finished but decode is still running) looks like score=0, beating every
+busy host.
+
+**Fix (B'):** multiply by `max(num_requests, 1)`:
+
+```python
+score = (pending_prefill + new + decode_pen) * max(num_requests, 1)
+```
+
+Now idle hosts with high decode load get score = decode_pen × 1 = real
+nonzero penalty, beating zero-load hosts only when decode is small.
+
+### A+B' — re-run with the fix
+
+**Run.** `unified_of13_lmw001_v2_20260527_1724/unified/`.
+
+| Metric (ms) | baseline | A+B (BUG) | A+B' (fix) | Δ vs baseline |
+|---|---:|---:|---:|---:|
+| TTFT p50 | 520 | 514 | **485** | −7% |
+| **TTFT p90** | 8781 | 8421 | **8287** | **−5.6%** ✅ |
+| TTFT p99 | 47647 | 44800 | **41876** | **−12%** ✅ |
+| TPOT p90 | 17.8 | 15.7 | 17.5 | −2% |
+| E2E p90 | 19989 | 21064 | 20625 | +3% |
+| E2E p99 | 85841 | 64344 | 77827 | −9% |
+
+A+B' **best of all variants on TTFT p90 (8287) and TTFT p99 (41876)**.
+Long-tail counts (>30s, >50s) also best across variants.
+
+vs v3 reference points:
+| | TTFT p90 | TPOT p90 | E2E p99 |
+|---|---:|---:|---:|
+| **A+B'** | **8287** | 17.5 | 77827 |
+| v3 fixed (cache-blind) | 10828 | 21.0 | 47610 |
+| v3 + Mech B | 9711 | 18.3 | 84492 |
+
+A+B' **beats v3 Mech B by 15% TTFT p90** with no migration overhead.
+
+### Per-worker (A+B' fixed)
+
+```
+8000: n=158  p90= 5688      8004: n=189  p90= 4249
+8001: n=159  p90= 7323      8005: n=116  p90=14598
+8002: n=114  p90= 8726      8006: n=180  p90= 6198
+8003: n=173  p90= 6715      8007: n=125  p90=22242   ← still hot
+```
+
+A+B' redistributed load more evenly (114..189) but **8007 still has p90=22s**.
+
+### 8007 deep-dive in A+B'
+
+```
+8007: n=125, affinity=69 (55%), fallback=56 (45%), cache_hit_mean=lowest
+```
+
+Top-15 slow at 8007:
+- 7 of them are session **1313181** turns 9–14 (130k+ tokens each, agentic
+  long context, ~50% cache hit)
+- Several others are cold-start turn-1 of large-prompt sessions
+- First two slow reqs arrived **0.7 s apart** — strong hint of concurrent
+  picker race
+
+### Iteration 3: race-condition fix
+
+**Diagnosis.** In `_handle_combined`:
+
+```python
+chosen, best_idx, decision = pick_instance_unified_hybrid(...)  # sync
+# ... sync breakdown updates ...
+return await _handle_local_request(...)   # ← await yields here
+                                          #   THEN reservation happens
+```
+
+`return await async_func(...)` evaluates the async call (creates coroutine)
+and yields to the event loop **before** the coroutine body executes. The
+reservation (`chosen.pending_prefill_tokens += new`, etc.) lives at the top
+of `_handle_local_request`, so between the picker and the reservation there
+is a **window where another coroutine can run and re-pick the same instance**.
+
+When two big-prompt reqs arrive within milliseconds, both run pick →
+both pick the "free" 8007 → both yield → both reserve. Result: 8007 gets
+back-to-back 130k-token cold prefills, each waiting for the other.
+
+**Fix.** Move the reservation **before** the await, inside `_handle_combined`:
+
+```python
+# Race fix: reserve atomically with pick, before any await.
+chosen.ongoing_tokens += input_length
+chosen.pending_prefill_tokens += estimated_new
+chosen.num_requests += 1
+return await _handle_local_request(..., _pre_reserved=True)
+```
+
+`_handle_local_request` skips its own reservation when `_pre_reserved=True`.
+PD-sep paths are unaffected (they have their own reservation).
+
+**Run.** Pending — `unified_of13_lmw001_racefix_*`. Hypothesis: 8007 p90
+drops to within ±3s of cluster median, since concurrent picks for the
+same "free" instance no longer happen.
+
+---
+
+## A+B'+RaceFix — results
+
+**Run.** `unified_of13_lmw001_racefix_20260527_1821/unified/`.
+
+| Metric (ms) | baseline | A+B' | A+B'+RF | Δ vs baseline |
+|---|---:|---:|---:|---:|
+| TTFT p50 | 520 | 485 | **478** | −8% |
+| **TTFT p90** | 8781 | 8287 | **7770** | **−11.5%** ✅ |
+| TTFT p99 | 47647 | 41876 | **42447** | −11% |
+| TPOT p90 | 17.8 | 17.5 | 18.0 | +1% |
+| E2E p90 | 19989 | 20625 | **18418** | −8% |
+| E2E p99 | 85841 | 77827 | **71227** | −17% |
+
+vs v3 reference:
+- **A+B'+RF TTFT p90 = 7770ms, vs v3 Mech B 9711ms → −20%** ✅
+
+Long-tail counts (best across all variants):
+```
+> 5s:   170 → 158         > 30s:   41 → 33
+>10s:   105 → 103         > 50s:    8 →  4
+>20s:    65 →  57         >100s:    0 →  0
+```
+
+### Decision split — race fix mainly helped affinity
+
+| Decision | baseline | A+B'+RF |
+|---|---:|---:|
+| affinity p90 | 7011 | **5042** ✅ (−28%) |
+| fallback p90 | 12083 | 13944 (+15%) |
+
+The race-condition was hurting affinity decisions the most. When two
+concurrent reqs both stuck to a "free-looking" affinity instance, they
+piled up and inflated affinity's tail. Fix removed this collision.
+
+### Per-worker
+
+```
+8000: n=86   p90=11541      8004: n=150  p90=11906
+8001: n=186  p90= 8307      8005: n=109  p90= 4798
+8002: n=105  p90=14540      8006: n=183  p90= 6258
+8003: n=264  p90= 3079      8007: n=131  p90=21850   ← still hot
+8000 spread now 86..264 — race fix did disperse routing
+```
+
+### 8007 still hot — but it's **workload-inherent, not a routing bug**
+
+Top sessions on 8007:
+```
+session 1279412: n=22  mean= 2208  max=18985  decisions: 91% affinity
+session 1313181: n=17  mean=17399  max=49089  decisions: 65% affinity
+session 1262354: n=15  mean=  622  max= 2325  decisions: 87% affinity
+session 1342921: n= 7  mean=17817  max=55589  decisions: 86% affinity
+session 1260327: n= 8  mean= 1636  max= 5382  decisions: 75% affinity
+session 1268831: n= 5  mean= 1443  max= 2673  decisions: 80% affinity
+```
+
+Sessions 1313181 and 1342921 are **long agentic contexts**: 100k–130k tokens
+per turn with ~50% cache hit (i.e. 50k new tokens prefill per turn). Even
+on a perfectly load-balanced instance, each turn is 7–15s of pure compute.
+
+Forcing these sessions to spread across instances would mean **cold prefill
+every turn (0% cache hit)** → each turn becomes 20–30s instead of 7–15s.
+Spreading is **net-negative**.
+
+→ The 8007 p90=22s is the floor imposed by these sessions' structure,
+not by routing policy. Unified is at its ceiling for this workload.
+
+---
+
+## Final ranking and take-aways
+
+| Policy | TTFT p90 (ms) | Δ vs baseline | Notes |
+|---|---:|---:|---|
+| baseline unified (of=2.0) | 8781 | — | reference |
+| A (of=1.3) | 8730 | ≈0 | affinity p90 -17%, fallback p90 +27% |
+| A+B (of=1.3, lmw=0.01, BUG) | 8421 | −4% | 8007 hotspot from `*num_req` zeroing bug |
+| A+B' (formula fix) | 8287 | −5.6% | Bug fixed, still 8007 mild hotspot |
+| **A+B'+RaceFix** | **7770** | **−11.5%** ✅ | **Best unified variant** |
+| v3 fixed | 10828 | +23% | PD-sep migration, cache-blind picker |
+| v3 + Mech B | 9711 | +11% | PD-sep + cache-rich target picker |
+
+### Conclusions
+
+1. **Unified path beats v3 PD-sep on this workload by 20%+ TTFT p90.**
+   PD-sep migration's fixed cost (src prefill + dst first-token waiting on
+   loaded scheduler) outweighs any decode-time savings for short-output
+   agentic turns.
+
+2. **Three orthogonal fixes compound for a 11.5% TTFT p90 win:**
+   - A (`overload_factor=1.3`): tighter affinity overflow → −0.6% but
+     much cleaner affinity decisions (p90 -17%)
+   - B' (`lmetric_decode_weight=0.01` with `max(num_req,1)`): decode-aware
+     fallback → −3.5%
+   - RaceFix (atomic reserve before await): kills concurrent-pick
+     collisions → −5.6%
+
+3. **Race condition was the biggest single hidden bug.** `return await
+   async_func(...)` yields to the event loop **before** the body of
+   `async_func` runs, so reservations done in the body don't take effect
+   in time to deter concurrent picks. This affects ANY async dispatch
+   with separate pick/reserve steps — worth checking other routing
+   policies.
+
+4. **8007 p90=22s is workload-inherent.** Sessions with 100k+ token turns
+   at 50% cache hit cannot finish faster than 7–15s per turn regardless
+   of routing. Forcing spread would hurt rather than help.
+
+5. **Migration (v3) is not necessary** when unified routing is tuned
+   well. Save the PD-sep mechanism for cases where it can be proven
+   net-positive (e.g. very-long-output sessions on extremely overloaded
+   prefill hosts) and use unified A+B'+RaceFix as the default.
+
+---
+
+## Direction A+B — run pending
+
+(Will be filled when `unified_of13_lmw001_*/unified/` finishes.)

microbench/connector_tax/cache_sweep/analyze_b3_replay.py

@@ -0,0 +1,169 @@
+#!/usr/bin/env python3
+"""B3 5-policy re-test analyser.
+
+Compute TTFT/TPOT/E2E mean/p50/p90/p99 for each policy from
+metrics.jsonl, compare against the historical b3_policy_comparison.json
+that drives fig_b3_latency_bars.png, and emit a side-by-side table
+plus a new figure with the same layout as the original.
+
+Usage:
+  python analyze_b3_replay.py --root <outroot> [--old-data <path>] [--figure <path>]
+"""
+
+import argparse
+import json
+import statistics
+from pathlib import Path
+
+
+POLICIES = ["lmetric", "load_only", "sticky", "unified", "unified_v2"]
+
+
+def pct(xs, p):
+    if not xs:
+        return None
+    xs = sorted(xs)
+    k = max(0, min(len(xs) - 1, int(p / 100.0 * (len(xs) - 1))))
+    return xs[k]
+
+
+def summarise(path):
+    rows = [json.loads(l) for l in open(path) if l.strip()]
+    ok = [r for r in rows if not r.get("error")]
+    ttft = [r["ttft_s"] * 1000 for r in ok if r.get("ttft_s") is not None]
+    tpot = [r["tpot_s"] * 1000 for r in ok if r.get("tpot_s")]
+    e2e = [r["latency_s"] * 1000 for r in ok if r.get("latency_s") is not None]
+    return {
+        "n_total": len(rows),
+        "n_ok": len(ok),
+        "ttft_mean_ms": statistics.mean(ttft) if ttft else None,
+        "ttft_p50_ms": pct(ttft, 50),
+        "ttft_p90_ms": pct(ttft, 90),
+        "ttft_p99_ms": pct(ttft, 99),
+        "tpot_mean_ms": statistics.mean(tpot) if tpot else None,
+        "tpot_p50_ms": pct(tpot, 50),
+        "tpot_p90_ms": pct(tpot, 90),
+        "tpot_p99_ms": pct(tpot, 99),
+        "e2e_mean_ms": statistics.mean(e2e) if e2e else None,
+        "e2e_p50_ms": pct(e2e, 50),
+        "e2e_p90_ms": pct(e2e, 90),
+        "e2e_p99_ms": pct(e2e, 99),
+    }
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--root", type=Path, required=True)
+    ap.add_argument("--old-data", type=Path,
+                    default=Path("analysis/characterization/window_1_results/b3_policy_comparison.json"))
+    ap.add_argument("--figure", type=Path, default=None)
+    args = ap.parse_args()
+
+    new = {}
+    for p in POLICIES:
+        path = args.root / p / "metrics.jsonl"
+        if not path.exists():
+            print(f"MISSING: {path}")
+            continue
+        new[p] = summarise(path)
+
+    old = {}
+    if args.old_data.exists():
+        d = json.load(open(args.old_data))
+        for r in d.get("rows", []):
+            old[r["policy"]] = {
+                "ttft_p50_ms": r["ttft_p50_s"] * 1000,
+                "ttft_p90_ms": r["ttft_p90_s"] * 1000,
+                "ttft_p99_ms": r["ttft_p99_s"] * 1000,
+                "tpot_p90_ms": r["tpot_p90_s"] * 1000,
+                "e2e_p90_ms":  r.get("e2e_p90_s", 0) * 1000,
+            }
+
+    def fmt(v): return f"{v:.0f}" if v is not None else "-"
+    def pctd(a, b):
+        if a is None or b is None or a == 0: return "-"
+        return f"{(b/a-1)*100:+.1f}%"
+
+    # Headline table
+    print(f"\n# NEW: today's re-test")
+    print(f"{'policy':<14}{'n_ok':>6}{'TTFTp50':>10}{'TTFTp90':>10}{'TTFTp99':>10}{'TPOTp90':>10}{'E2Ep90':>10}")
+    print("-" * 70)
+    for p in POLICIES:
+        if p not in new: continue
+        r = new[p]
+        print(f"{p:<14}{r['n_ok']:>6}{fmt(r['ttft_p50_ms']):>9}ms{fmt(r['ttft_p90_ms']):>9}ms{fmt(r['ttft_p99_ms']):>9}ms{fmt(r['tpot_p90_ms']):>9}ms{fmt(r['e2e_p90_ms']):>9}ms")
+
+    print(f"\n# OLD: window_1_results/b3_policy_comparison.json")
+    print(f"{'policy':<14}{'TTFTp50':>10}{'TTFTp90':>10}{'TTFTp99':>10}{'TPOTp90':>10}{'E2Ep90':>10}")
+    print("-" * 60)
+    for p in POLICIES:
+        if p not in old: continue
+        r = old[p]
+        print(f"{p:<14}{fmt(r['ttft_p50_ms']):>9}ms{fmt(r['ttft_p90_ms']):>9}ms{fmt(r['ttft_p99_ms']):>9}ms{fmt(r['tpot_p90_ms']):>9}ms{fmt(r['e2e_p90_ms']):>9}ms")
+
+    print(f"\n# DRIFT: today vs old (same policy)")
+    print(f"{'policy':<14}{'ΔTTFTp50':>10}{'ΔTTFTp90':>10}{'ΔTTFTp99':>10}{'ΔTPOTp90':>10}{'ΔE2Ep90':>10}")
+    print("-" * 60)
+    for p in POLICIES:
+        if p not in new or p not in old: continue
+        n, o = new[p], old[p]
+        print(f"{p:<14}{pctd(o['ttft_p50_ms'], n['ttft_p50_ms']):>10}"
+              f"{pctd(o['ttft_p90_ms'], n['ttft_p90_ms']):>10}"
+              f"{pctd(o['ttft_p99_ms'], n['ttft_p99_ms']):>10}"
+              f"{pctd(o['tpot_p90_ms'], n['tpot_p90_ms']):>10}"
+              f"{pctd(o['e2e_p90_ms'], n['e2e_p90_ms']):>10}")
+
+    # Relative ordering check
+    def ranks(values_dict, key):
+        items = [(p, r[key]) for p, r in values_dict.items() if r.get(key)]
+        items.sort(key=lambda x: x[1])
+        return [p for p, _ in items]
+
+    print(f"\n# TTFT p90 ranking (best → worst)")
+    for label, src in [("OLD", old), ("NEW", new)]:
+        if src:
+            order = ranks(src, "ttft_p90_ms")
+            print(f"  {label}: {' < '.join(order)}")
+
+    out = {"new": new, "old": old}
+    out_path = args.root / "b3_replay_summary.json"
+    out_path.write_text(json.dumps(out, indent=2))
+    print(f"\nWrote {out_path}")
+
+    # Bar plot (matplotlib)
+    if not args.figure:
+        args.figure = args.root / "fig_b3_latency_bars_new.png"
+    try:
+        import matplotlib
+        matplotlib.use("Agg")
+        import matplotlib.pyplot as plt
+
+        pols = [p for p in POLICIES if p in new]
+        metrics = [("TTFT p90 (s)", "ttft_p90_ms", 1000),
+                    ("TPOT p90 (ms)", "tpot_p90_ms", 1),
+                    ("E2E p90 (s)", "e2e_p90_ms", 1000)]
+        colors = {"lmetric": "tab:blue", "load_only": "tab:orange",
+                  "sticky": "tab:green", "unified": "tab:red",
+                  "unified_v2": "tab:purple"}
+        fig, axes = plt.subplots(1, 3, figsize=(14, 4.5))
+        for ax, (label, key, div) in zip(axes, metrics):
+            vals = [new[p][key] / div for p in pols]
+            bars = ax.bar(pols, vals,
+                          color=[colors.get(p, "gray") for p in pols],
+                          edgecolor="black", linewidth=0.5)
+            ax.set_title(label)
+            ax.tick_params(axis="x", rotation=20)
+            for b, v in zip(bars, vals):
+                ax.text(b.get_x() + b.get_width() / 2, v, f"{v:.1f}",
+                        ha="center", va="bottom", fontsize=9)
+            ax.grid(alpha=0.3, axis="y")
+        fig.suptitle(f"B3 5-policy re-test ({args.root.name})")
+        fig.tight_layout()
+        fig.savefig(args.figure, dpi=120)
+        print(f"Wrote {args.figure}")
+    except Exception as e:
+        print(f"(figure skipped: {e})")
+
+
+if __name__ == "__main__":
+    main()

microbench/connector_tax/cache_sweep/analyze_dst_migration.py

@@ -0,0 +1,333 @@
+#!/usr/bin/env python3
+"""Analyze dst-side migration breakdown for unified_v3 runs.
+
+Joins the proxy `breakdown.json` (per-request route + phase timestamps)
+with the dst engine per-PID logs written by
+`instrument_dst_migration.py` (`dm_mig_pid<pid>.jsonl`), to attribute
+each migration's dst-side wall-clock into:
+
+  T_relay              proxy decode-sent → dst arrival
+  T_admission_pre_kv   dst arrival → status=WAITING_FOR_REMOTE_KVS
+                       (waiting in dst's scheduler queue before KV pull
+                        is even initiated)
+  T_kv_pull            WAITING_FOR_REMOTE_KVS → finished_recving
+                       (the actual RDMA transfer + connector ack)
+  T_admission_post_kv  finished_recving → first time in self.running
+                       (KV ready, waiting for batch slot)
+  T_first_iter         first scheduled → first generated token
+                       (one decode-iter compute + sampler latency)
+
+Layerwise transfer can at best eliminate T_kv_pull. Everything else is
+queueing or compute that layerwise does not touch.
+
+Usage:
+    python analyze_dst_migration.py \
+        --proxy-breakdown <RUNDIR>/breakdown.json \
+        --dst-log-dir     <DST_LOG_DIR>
+        [--output         <RUNDIR>/dst_migration_breakdown.csv]
+        [--plot           <RUNDIR>/dst_migration_breakdown.png]
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import re
+import statistics
+import sys
+from pathlib import Path
+
+
+def _core_req_id(rid: str) -> str:
+    """Normalize a vLLM engine req_id back to the proxy's request_id.
+
+    vLLM wraps the proxy id `S:T:U:N` as `cmpl-S:T:U:N-<dp_rank>-<hex>`.
+    Strip the `cmpl-` prefix and the trailing `-<digits>-<hex>` suffix so
+    it joins against the proxy `breakdown.json` request_id.
+    """
+    if not rid:
+        return rid
+    s = rid
+    if s.startswith("cmpl-"):
+        s = s[len("cmpl-"):]
+    m = re.match(r"^(.*)-\d+-[0-9a-fA-F]+$", s)
+    if m:
+        s = m.group(1)
+    return s
+
+
+def _pct(vals: list[float], q: float) -> float:
+    if not vals:
+        return float("nan")
+    vs = sorted(vals)
+    i = max(0, min(len(vs) - 1, int(math.ceil(q * len(vs))) - 1))
+    return vs[i]
+
+
+def _summary(name: str, vals: list[float]) -> dict:
+    if not vals:
+        return {"name": name, "n": 0}
+    return {
+        "name": name,
+        "n": len(vals),
+        "mean_s": statistics.mean(vals),
+        "p50_s": _pct(vals, 0.5),
+        "p90_s": _pct(vals, 0.9),
+        "p99_s": _pct(vals, 0.99),
+        "max_s": max(vals),
+        "sum_s": sum(vals),
+    }
+
+
+def load_dst_log(dst_log_dir: Path) -> dict[str, dict]:
+    by_req: dict[str, dict] = {}
+    found_files = sorted(dst_log_dir.glob("dm_mig_pid*.jsonl"))
+    print(f"[analyze] dst log files: {len(found_files)} under {dst_log_dir}")
+    for f in found_files:
+        with f.open() as fh:
+            for line in fh:
+                try:
+                    rec = json.loads(line)
+                except Exception:
+                    continue
+                rid = rec.get("req_id")
+                if not rid:
+                    continue
+                key = _core_req_id(rid)
+                rec["_raw_req_id"] = rid
+                # If a req shows up twice (shouldn't, but be safe), prefer the
+                # one with t_first_token_unix populated.
+                prev = by_req.get(key)
+                if prev is None or (
+                    rec.get("t_first_token_unix") and
+                    not prev.get("t_first_token_unix")
+                ):
+                    by_req[key] = rec
+    print(f"[analyze] unique dst records: {len(by_req)}")
+    return by_req
+
+
+def load_proxy_breakdown(path: Path) -> list[dict]:
+    with path.open() as fh:
+        data = json.load(fh)
+    assert isinstance(data, list), f"unexpected breakdown.json shape: {type(data)}"
+    return data
+
+
+def decompose(proxy_recs: list[dict], dst_by_req: dict[str, dict]) -> list[dict]:
+    """Build per-migration breakdown rows by joining proxy + dst by req_id."""
+    rows: list[dict] = []
+    migrations = [x for x in proxy_recs if x.get("route_class") == "PD_SEP_V2"]
+    print(f"[analyze] proxy migrations: {len(migrations)} "
+          f"(of {len(proxy_recs)} total requests)")
+
+    miss_in_dst = 0
+    missing_phases = 0
+    for p in migrations:
+        rid = p.get("request_id")
+        dst = dst_by_req.get(rid)
+        if dst is None:
+            miss_in_dst += 1
+            continue
+        if dst.get("t_first_token_unix") is None:
+            missing_phases += 1
+            # still include the row but mark phases as NaN downstream
+        t_decode_sent = p.get("t_decode_sent_unix")
+        t_first_tok = p.get("t_first_token_unix")
+        t_arrival = dst.get("t_arrival_unix")
+        t_wait_kvs = dst.get("t_wait_for_kvs_unix")
+        t_kv_done = dst.get("t_kv_recv_done_unix")
+        t_first_sched = dst.get("t_first_scheduled_unix")
+        t_first_tok_dst = dst.get("t_first_token_unix")
+
+        def _diff(a, b):
+            if a is None or b is None:
+                return None
+            return float(a) - float(b)
+
+        rows.append({
+            "request_id": rid,
+            "session_id": p.get("session_id"),
+            "input_length": p.get("input_length"),
+            "v3_new_local": p.get("v3_new_local"),
+            "v3_target_idx": p.get("v3_target_idx") or p.get("v3_decode_target_idx"),
+            "arrival_n_running": (dst.get("arrival_state") or {}).get("n_running"),
+            "arrival_n_waiting": (dst.get("arrival_state") or {}).get("n_waiting"),
+            "arrival_pending_prefill_tok": (dst.get("arrival_state") or {}).get("pending_prefill_tok"),
+            "arrival_n_waiting_for_kvs": (dst.get("arrival_state") or {}).get("n_waiting_for_kvs"),
+            # Phase durations (seconds)
+            "T_proxy_total_dst_first_token_s": _diff(t_first_tok, t_decode_sent),
+            "T_relay_s": _diff(t_arrival, t_decode_sent),
+            "T_admission_pre_kv_s": _diff(t_wait_kvs, t_arrival),
+            "T_kv_pull_s": _diff(t_kv_done, t_wait_kvs),
+            "T_admission_post_kv_s": _diff(t_first_sched, t_kv_done),
+            "T_first_iter_s": _diff(t_first_tok_dst, t_first_sched),
+            # Raw timestamps for debugging
+            "t_decode_sent_unix": t_decode_sent,
+            "t_dst_arrival_unix": t_arrival,
+            "t_dst_wait_for_kvs_unix": t_wait_kvs,
+            "t_dst_kv_recv_done_unix": t_kv_done,
+            "t_dst_first_scheduled_unix": t_first_sched,
+            "t_dst_first_token_unix": t_first_tok_dst,
+            "t_proxy_first_token_unix": t_first_tok,
+        })
+
+    print(f"[analyze] missing in dst log: {miss_in_dst}")
+    print(f"[analyze] dst record incomplete (no t_first_token): {missing_phases}")
+    return rows
+
+
+def emit_summary(rows: list[dict]) -> None:
+    if not rows:
+        print("[analyze] no rows — nothing to summarize.")
+        return
+
+    phase_keys = [
+        "T_proxy_total_dst_first_token_s",
+        "T_relay_s",
+        "T_admission_pre_kv_s",
+        "T_kv_pull_s",
+        "T_admission_post_kv_s",
+        "T_first_iter_s",
+    ]
+
+    print()
+    print("=" * 88)
+    print(f"Migration dst-side phase breakdown  (n_migrations={len(rows)})")
+    print("=" * 88)
+    print(f"{'phase':<36} {'n':>4} {'mean(s)':>9} {'p50':>8} {'p90':>8} "
+          f"{'p99':>8} {'max':>8} {'sum(s)':>9}")
+    print("-" * 88)
+    for k in phase_keys:
+        vals = [r[k] for r in rows if r.get(k) is not None]
+        if not vals:
+            print(f"{k:<36} {'n/a':>4}")
+            continue
+        s = _summary(k, vals)
+        print(f"{k:<36} {s['n']:>4} {s['mean_s']:>9.3f} {s['p50_s']:>8.3f} "
+              f"{s['p90_s']:>8.3f} {s['p99_s']:>8.3f} {s['max_s']:>8.3f} "
+              f"{s['sum_s']:>9.2f}")
+
+    print()
+    print("Aggregate attribution (sum across all migrations):")
+    sums = {}
+    for k in ("T_relay_s", "T_admission_pre_kv_s", "T_kv_pull_s",
+              "T_admission_post_kv_s", "T_first_iter_s"):
+        sums[k] = sum(r[k] for r in rows if r.get(k) is not None)
+    total = sum(sums.values())
+    total_proxy = sum(r["T_proxy_total_dst_first_token_s"] for r in rows
+                      if r.get("T_proxy_total_dst_first_token_s") is not None)
+    print(f"  decomposed sum   : {total:>8.2f} s")
+    print(f"  proxy total sum  : {total_proxy:>8.2f} s   "
+          f"(should be ~equal; gap = uninstrumented)")
+    if total > 0:
+        for k, v in sums.items():
+            print(f"    {k:<28} {v:>8.2f} s  ({v/total*100:5.1f} %)")
+
+    # Headline: "How much could layerwise save?"
+    layerwise_addressable = sums.get("T_kv_pull_s", 0.0)
+    queue_residual = sum(v for k, v in sums.items() if k != "T_kv_pull_s")
+    print()
+    print("Layerwise-addressable vs queue-residual:")
+    print(f"  T_kv_pull_s (addressable by layerwise)   : {layerwise_addressable:>8.2f} s "
+          f"({layerwise_addressable / total * 100 if total else 0:5.1f} %)")
+    print(f"  everything else (queue/admission/iter)   : {queue_residual:>8.2f} s "
+          f"({queue_residual / total * 100 if total else 0:5.1f} %)")
+
+
+def write_csv(rows: list[dict], path: Path) -> None:
+    import csv
+    if not rows:
+        path.write_text("")
+        return
+    fields = list(rows[0].keys())
+    with path.open("w", newline="") as fh:
+        w = csv.DictWriter(fh, fieldnames=fields)
+        w.writeheader()
+        w.writerows(rows)
+    print(f"[analyze] wrote CSV: {path}  (n={len(rows)})")
+
+
+def maybe_plot(rows: list[dict], out_path: Path) -> None:
+    try:
+        import matplotlib
+        matplotlib.use("Agg")
+        import matplotlib.pyplot as plt
+    except Exception as e:
+        print(f"[analyze] matplotlib unavailable ({e}); skipping plot.")
+        return
+
+    if not rows:
+        return
+
+    rows_sorted = sorted(
+        rows,
+        key=lambda r: r.get("T_proxy_total_dst_first_token_s") or 0.0,
+    )
+    n = len(rows_sorted)
+    idx = list(range(n))
+
+    def col(k):
+        return [(r.get(k) or 0.0) for r in rows_sorted]
+
+    relay = col("T_relay_s")
+    pre = col("T_admission_pre_kv_s")
+    pull = col("T_kv_pull_s")
+    post = col("T_admission_post_kv_s")
+    first_iter = col("T_first_iter_s")
+
+    fig, ax = plt.subplots(figsize=(11, 5))
+    bot = [0.0] * n
+    for vals, label, color in [
+        (relay, "HTTP relay", "#cccccc"),
+        (pre, "admission pre-KV", "#f4a261"),
+        (pull, "KV pull (layerwise-addressable)", "#e76f51"),
+        (post, "admission post-KV", "#2a9d8f"),
+        (first_iter, "first decode iter", "#264653"),
+    ]:
+        ax.bar(idx, vals, bottom=bot, color=color, label=label, width=0.85)
+        bot = [b + v for b, v in zip(bot, vals)]
+
+    ax.set_xticks(idx)
+    ax.set_xticklabels([str(i + 1) for i in idx], rotation=0, fontsize=8)
+    ax.set_xlabel("Migrated request (sorted by total dst wait, ascending)")
+    ax.set_ylabel("Time (s)")
+    ax.set_title("Per-migration dst-side phase breakdown (v3 unified_v3 run)")
+    ax.legend(loc="upper left", fontsize=9)
+    ax.grid(axis="y", linestyle=":", alpha=0.5)
+    fig.tight_layout()
+    fig.savefig(out_path, dpi=120)
+    plt.close(fig)
+    print(f"[analyze] wrote plot: {out_path}")
+
+
+def main() -> None:
+    p = argparse.ArgumentParser()
+    p.add_argument("--proxy-breakdown", type=Path, required=True)
+    p.add_argument("--dst-log-dir", type=Path, required=True)
+    p.add_argument("--output", type=Path, default=None,
+                   help="CSV path (default: <run>/dst_migration_breakdown.csv)")
+    p.add_argument("--plot", type=Path, default=None,
+                   help="PNG path (default: <run>/dst_migration_breakdown.png)")
+    args = p.parse_args()
+
+    if not args.proxy_breakdown.is_file():
+        sys.exit(f"missing proxy breakdown: {args.proxy_breakdown}")
+    if not args.dst_log_dir.is_dir():
+        sys.exit(f"missing dst log dir: {args.dst_log_dir}")
+
+    run_dir = args.proxy_breakdown.parent
+    out_csv = args.output or (run_dir / "dst_migration_breakdown.csv")
+    out_png = args.plot or (run_dir / "dst_migration_breakdown.png")
+
+    proxy_recs = load_proxy_breakdown(args.proxy_breakdown)
+    dst_by_req = load_dst_log(args.dst_log_dir)
+    rows = decompose(proxy_recs, dst_by_req)
+    emit_summary(rows)
+    write_csv(rows, out_csv)
+    maybe_plot(rows, out_png)
+
+
+if __name__ == "__main__":
+    main()

microbench/connector_tax/cache_sweep/analyze_migration_log.py

@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+"""Per-migration log + per-instance summary for a v3 trace replay.
+
+Reads <run_dir>/breakdown.json and <run_dir>/metrics.jsonl and emits:
+  1. A row per migration showing src→dst, per-side state snapshots, and
+     the resulting TTFT.
+  2. Histograms: migrations received per inst, sent per inst, all
+     (src→dst) pairs.
+  3. Post-rotation tail: how many turns of migrated sessions ended up on
+     each inst (downstream impact of rotation).
+  4. Anti-hotspot signal: recent_mig_received_in_window at decision time.
+
+Run any v3 replay through this to spot pathological clustering of
+migrations on the same dst within a short window.
+
+Usage:
+    python analyze_migration_log.py <RUN_DIR>
+where <RUN_DIR> contains breakdown.json + metrics.jsonl (i.e. the proxy's
+per-policy output folder, e.g. .../b3_v3_20260527_1344/unified_v3).
+"""
+import json
+import sys
+from collections import Counter, defaultdict
+from pathlib import Path
+
+
+def main(run_dir: Path) -> None:
+    bd = json.load(open(run_dir / "breakdown.json"))
+    m = {json.loads(l)["request_id"]: json.loads(l)
+         for l in open(run_dir / "metrics.jsonl")}
+
+    mig = [e for e in bd if e.get("v3_migrate")]
+    mig.sort(key=lambda x: x.get("t_decision_unix", 0))
+
+    print(f"=== {len(mig)} migrations in {run_dir.name} ===\n")
+    cols = (
+        "#", "t_rel", "session", "turn",
+        "src", "dst", "src_nreq", "src_dec_tok",
+        "dst_nreq", "dst_cache", "dst_recent_recv",
+        "inlen", "self_ttft_ms",
+    )
+    print("  " + " ".join(f"{c:>13}" for c in cols))
+    print("-" * (15 * len(cols)))
+
+    t0 = mig[0]["t_decision_unix"] if mig else 0
+    for i, e in enumerate(mig):
+        rid = e["request_id"]
+        src_idx = e.get("v3_src_idx", e["chosen_idx"])
+        dst_idx = e.get("v3_target_idx", -1)
+        src_state = e.get("v3_src_state") or {}
+        dst_state = e.get("v3_target_state") or {}
+        cands = {c["idx"]: c for c in e.get("candidate_scores", [])}
+        # Fall back to candidate_scores if dedicated v3_*_state fields aren't present.
+        src_nreq = src_state.get("num_requests", cands.get(src_idx, {}).get("num_requests", "-"))
+        src_dec_tok = src_state.get("ongoing_decode_tokens",
+                                    cands.get(src_idx, {}).get("ongoing_decode_tokens", "-"))
+        dst_nreq = dst_state.get("num_requests", cands.get(dst_idx, {}).get("num_requests", "-"))
+        dst_cache = e.get("v3_target_cache_hit", dst_state.get("cache_hit_estimate", 0))
+        dst_recent = e.get("v3_target_recent_received",
+                            dst_state.get("recent_mig_received_in_window", "-"))
+        inlen = e.get("input_length") or m.get(rid, {}).get("input_length", 0)
+        ttft = m.get(rid, {}).get("ttft_s") or 0
+        t_rel = e["t_decision_unix"] - t0
+        turn = m.get(rid, {}).get("turn_id", "?")
+        print(
+            f"  {i+1:>13} {t_rel:>13.1f} {e['session_id']:>13} {turn:>13} "
+            f"{src_idx:>13} {dst_idx:>13} {src_nreq:>13} {src_dec_tok:>13} "
+            f"{dst_nreq:>13} {dst_cache:>13} {dst_recent:>13} "
+            f"{inlen:>13} {ttft*1000:>13.0f}"
+        )
+
+    # Aggregate counts
+    print("\n=== Migrations TO each instance ===")
+    to_count = Counter(e.get("v3_target_idx", -1) for e in mig)
+    for idx in range(8):
+        print(f"  inst_{idx}: {to_count.get(idx, 0)} migrations received")
+
+    print("\n=== Migrations FROM each instance ===")
+    from_count = Counter(e.get("v3_src_idx", e["chosen_idx"]) for e in mig)
+    for idx in range(8):
+        print(f"  inst_{idx}: {from_count.get(idx, 0)} migrations sent")
+
+    print("\n=== Migration pairs (src→dst, count) ===")
+    pair_count = Counter(
+        (e.get("v3_src_idx", e["chosen_idx"]), e.get("v3_target_idx", -1))
+        for e in mig
+    )
+    for (s, d), n in sorted(pair_count.items(), key=lambda x: -x[1]):
+        print(f"  {s} → {d}: {n}")
+
+    print("\n=== Sessions migrating multiple times ===")
+    sess_mig = defaultdict(list)
+    for e in mig:
+        sess_mig[e["session_id"]].append(
+            (e.get("t_decision_unix", 0),
+             e.get("v3_src_idx", e["chosen_idx"]),
+             e.get("v3_target_idx", -1))
+        )
+    multi = {s: ev for s, ev in sess_mig.items() if len(ev) > 1}
+    if not multi:
+        print("  (none)")
+    for sess, events in sorted(multi.items()):
+        chain = " → ".join(f"{s}->{d}" for _, s, d in sorted(events))
+        print(f"  session {sess}: {chain}")
+
+    # Recent-received hotspot signal — non-zero values mean the picker
+    # accepted a target that recently got another migration.
+    print("\n=== Anti-hotspot signal: dst.recent_mig_received_in_window ===")
+    rec = [e.get("v3_target_recent_received", 0) for e in mig]
+    if rec:
+        nonzero = [v for v in rec if v]
+        print(f"  total migrations: {len(rec)}, "
+              f"with recent_received > 0: {len(nonzero)}, "
+              f"max recent_received: {max(rec)}")
+
+    # Post-rotation tail: turns of migrated sessions after their LAST mig
+    print("\n=== Post-rotation tail per inst (turns of migrated sessions after last mig) ===")
+    tail = Counter()
+    for sess, events in sess_mig.items():
+        final_dst = sorted(events)[-1][2]
+        last_t = max(t for t, _, _ in events)
+        sess_turns = [mm for rid, mm in m.items() if mm["session_id"] == sess]
+        tail[final_dst] += sum(1 for mm in sess_turns
+                               if mm.get("t_dispatch_unix", 0) > last_t)
+    for idx in range(8):
+        print(f"  inst_{idx}: {tail.get(idx, 0)} tail turns")
+
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        print("usage: analyze_migration_log.py <run_dir>", file=sys.stderr)
+        sys.exit(1)
+    main(Path(sys.argv[1]))

microbench/connector_tax/cache_sweep/analyze_transfer_decomp.py

@@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+"""Decompose migration KV-transfer time into RDMA-actual vs control-plane.
+
+Joins three logs from an instrumented unified_v3 run:
+
+  proxy breakdown.json        — per-request route + phase timestamps
+  dst_mig_log/dm_mig_pid*.jsonl  — dst lifecycle (instrument_dst_migration.py)
+                                   gives T_kv_pull = wait_for_kvs -> recv_done
+  mooncake xfer/mb2_transfer_pid*.jsonl — connector internals
+                                   (instrument_mooncake.py):
+      send_blocks            : pure RDMA (total_bytes, duration_s)  [producer]
+      receive_kv_enter/finish: consumer-observed transfer window    [consumer]
+      ready_wait             : producer wait for src KV commit       [producer]
+      send_kv_to_decode_enter: producer received the pull request    [producer]
+
+Decisive question: of the 87% dst-side overhead that is T_kv_pull, how
+much is the actual RDMA write (`send_blocks`) vs control-plane
+(handshake / ready-wait / GIL starvation on the busy src)?
+
+  - send_blocks bandwidth ~ wire (10 GB/s) AND << T_kv_pull
+        => loss is control-plane; layerwise (which only moves WHEN the
+           RDMA fires) will NOT fix it.
+  - send_blocks bandwidth << wire
+        => the RDMA write itself is slow (NIC / src-side servicing);
+           characterize with a load microbench next.
+
+Usage:
+  python analyze_transfer_decomp.py \
+      --proxy-breakdown <RUN>/unified_v3/breakdown.json \
+      --dst-log-dir     <RUN>/dst_mig_log \
+      --xfer-log-dir    <RUN>/xfer_log
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import re
+import statistics
+import sys
+from pathlib import Path
+
+
+def _core_req_id(rid: str) -> str:
+    if not rid:
+        return rid
+    s = rid
+    if s.startswith("cmpl-"):
+        s = s[len("cmpl-"):]
+    m = re.match(r"^(.*)-\d+-[0-9a-fA-F]+$", s)
+    if m:
+        s = m.group(1)
+    return s
+
+
+def _pct(vals, q):
+    if not vals:
+        return float("nan")
+    vs = sorted(vals)
+    i = max(0, min(len(vs) - 1, int(math.ceil(q * len(vs))) - 1))
+    return vs[i]
+
+
+def _stat_line(name, vals, unit="s"):
+    if not vals:
+        print(f"{name:<34} n=0")
+        return
+    print(f"{name:<34} n={len(vals):>3}  mean={statistics.mean(vals):>8.3f}  "
+          f"p50={_pct(vals,0.5):>8.3f}  p90={_pct(vals,0.9):>8.3f}  "
+          f"max={max(vals):>8.3f}  sum={sum(vals):>8.2f} {unit}")
+
+
+def load_events(xfer_dir: Path):
+    files = sorted(xfer_dir.glob("mb2_transfer_pid*.jsonl"))
+    print(f"[xfer] log files: {len(files)} under {xfer_dir}")
+    send_blocks, recv_enter, recv_finish, ready_wait, send_enter = [], [], [], [], []
+    for f in files:
+        pid = f.stem.replace("mb2_transfer_pid", "")
+        with f.open() as fh:
+            for line in fh:
+                try:
+                    e = json.loads(line)
+                except Exception:
+                    continue
+                e["_pid"] = pid
+                ev = e.get("event")
+                if ev == "send_blocks":
+                    send_blocks.append(e)
+                elif ev == "receive_kv_enter":
+                    recv_enter.append(e)
+                elif ev == "receive_kv_finish":
+                    recv_finish.append(e)
+                elif ev == "ready_wait":
+                    ready_wait.append(e)
+                elif ev == "send_kv_to_decode_enter":
+                    send_enter.append(e)
+    print(f"[xfer] events: send_blocks={len(send_blocks)} "
+          f"recv_enter={len(recv_enter)} recv_finish={len(recv_finish)} "
+          f"ready_wait={len(ready_wait)} send_enter={len(send_enter)}")
+    return send_blocks, recv_enter, recv_finish, ready_wait, send_enter
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--proxy-breakdown", type=Path, required=True)
+    p.add_argument("--dst-log-dir", type=Path, required=True)
+    p.add_argument("--xfer-log-dir", type=Path, required=True)
+    args = p.parse_args()
+
+    for pth in (args.proxy_breakdown, args.dst_log_dir, args.xfer_log_dir):
+        if not pth.exists():
+            sys.exit(f"missing: {pth}")
+
+    proxy = json.load(args.proxy_breakdown.open())
+    migrations = [x for x in proxy if x.get("route_class") == "PD_SEP_V2"]
+    mig_ids = {x.get("request_id") for x in migrations}
+    print(f"[proxy] migrations: {len(migrations)} / {len(proxy)} total")
+
+    # dst lifecycle: T_kv_pull per migration (core req id)
+    dst_pull = {}
+    for f in sorted(args.dst_log_dir.glob("dm_mig_pid*.jsonl")):
+        for line in f.open():
+            try:
+                r = json.loads(line)
+            except Exception:
+                continue
+            tw = r.get("t_wait_for_kvs_unix")
+            td = r.get("t_kv_recv_done_unix")
+            if tw and td:
+                dst_pull[_core_req_id(r.get("req_id"))] = td - tw
+
+    sb, re_enter, re_finish, rw, se = load_events(args.xfer_log_dir)
+
+    # ---- 1. Pure RDMA bandwidth from send_blocks (the decisive number) ----
+    print("\n" + "=" * 90)
+    print("1. PURE RDMA WRITE rate (`send_blocks` = batch_transfer_sync_write)")
+    print("=" * 90)
+    bws, durs, bytes_l = [], [], []
+    for e in sb:
+        b = e.get("total_bytes", 0)
+        d = e.get("duration_s", 0)
+        if d and d > 0 and b > 0:
+            bws.append(b / 1e9 / d)
+            durs.append(d)
+            bytes_l.append(b)
+    if bws:
+        tot_b = sum(bytes_l)
+        tot_d = sum(durs)
+        print(f"  send_blocks calls: {len(bws)}")
+        print(f"  total bytes moved : {tot_b/2**30:.2f} GiB")
+        print(f"  total RDMA time   : {tot_d:.2f} s")
+        print(f"  AGGREGATE rate    : {tot_b/1e9/tot_d:.2f} GB/s  "
+              f"(MB2 idle-src steady-state = ~9.7-10 GB/s)")
+        _stat_line("  per-call rate (GB/s)", bws, unit="GB/s")
+        _stat_line("  per-call duration", durs)
+        # bandwidth vs size — small ops are latency-bound
+        print("\n  rate vs transfer size:")
+        pairs = sorted(zip(bytes_l, bws))
+        for b, w in pairs:
+            bar = "#" * int(min(40, w * 4))
+            print(f"    {b/2**20:>8.1f} MiB  {w:>6.2f} GB/s  {bar}")
+    else:
+        print("  no send_blocks events with positive duration")
+
+    # ---- 2. Producer ready-wait (src KV commit) ----
+    print("\n" + "=" * 90)
+    print("2. PRODUCER ready-wait (src KV not yet committed when pull arrived)")
+    print("=" * 90)
+    rw_vals = [e.get("ready_wait_s", 0) for e in rw if e.get("ready_wait_s") is not None]
+    already = sum(1 for e in rw if e.get("ready_already_set"))
+    _stat_line("  ready_wait", rw_vals)
+    print(f"  ready_already_set at entry: {already}/{len(rw)} "
+          f"(if most are True, src commit is not the bottleneck)")
+
+    # ---- 3. Consumer-observed receive_kv window ----
+    print("\n" + "=" * 90)
+    print("3. CONSUMER receive_kv window (enter->FINISH, ~most of T_kv_pull)")
+    print("=" * 90)
+    rf_vals = [e.get("duration_s", 0) for e in re_finish if e.get("duration_s")]
+    _stat_line("  receive_kv duration", rf_vals)
+
+    # ---- 4. Per-migration join: T_kv_pull vs receive_kv vs ready_wait ----
+    print("\n" + "=" * 90)
+    print("4. PER-MIGRATION join (T_kv_pull from dst vs connector internals)")
+    print("=" * 90)
+    # index connector events by core req id
+    rf_by_req = {}
+    for e in re_finish:
+        for rid in e.get("req_ids", []):
+            rf_by_req[_core_req_id(rid)] = e.get("duration_s")
+    rw_by_req = {}
+    for e in rw:
+        rw_by_req[_core_req_id(e.get("d_req_id", ""))] = e.get("ready_wait_s")
+
+    joined = 0
+    sum_pull = sum_recv = sum_rw = 0.0
+    rows = []
+    for m in migrations:
+        core = m.get("request_id")
+        pull = dst_pull.get(core)
+        recv = rf_by_req.get(core)
+        rwv = rw_by_req.get(core)
+        if pull is None and recv is None:
+            continue
+        joined += 1
+        if pull: sum_pull += pull
+        if recv: sum_recv += recv
+        if rwv: sum_rw += rwv
+        rows.append((core, m.get("input_length"), m.get("v3_target_cache_hit"),
+                     pull, recv, rwv))
+    print(f"  joined migrations: {joined}")
+    print(f"  Σ T_kv_pull (dst)        = {sum_pull:8.2f} s")
+    print(f"  Σ receive_kv (consumer)  = {sum_recv:8.2f} s")
+    print(f"  Σ ready_wait (producer)  = {sum_rw:8.2f} s")
+    # The RDMA share: best-effort total send_blocks time
+    sum_rdma = sum(durs) if durs else 0.0
+    print(f"  Σ send_blocks RDMA       = {sum_rdma:8.2f} s   (all transfers, "
+          f"not just migrations)")
+    if sum_pull > 0:
+        print(f"\n  RDMA-actual / T_kv_pull   ≈ {sum_rdma/sum_pull*100:5.1f} %")
+        print(f"  ready-wait / T_kv_pull    ≈ {sum_rw/sum_pull*100:5.1f} %")
+        resid = sum_pull - sum_rdma - sum_rw
+        print(f"  control-plane residual    ≈ {resid/sum_pull*100:5.1f} %  "
+              f"(handshake / ZMQ / GIL starvation)")
+
+    print("\n  per-migration detail:")
+    print(f"  {'req_id':<22} {'in_len':>7} {'dst_hit':>8} {'kv_pull':>8} "
+          f"{'recv_kv':>8} {'rdy_wait':>8}")
+    for core, il, hit, pull, recv, rwv in sorted(
+            rows, key=lambda r: -(r[3] or 0)):
+        def s(v): return f"{v:.2f}" if v is not None else "  --"
+        print(f"  {core:<22} {str(il):>7} {str(hit):>8} {s(pull):>8} "
+              f"{s(recv):>8} {s(rwv):>8}")
+
+
+if __name__ == "__main__":
+    main()

microbench/connector_tax/cache_sweep/run_5policy_600s.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# 5-policy comparison on the first-600s trace, fresh vLLM (cold APC) each arm.
+#   leastwork    LPWL (parameter-free)
+#   unified_ab   unified hybrid, A+B' tuned (of=1.3, lmw=0.01)
+#   unified_def  unified hybrid, defaults  (of=2.0, lmw=0.0)
+#   lmetric      P_tokens x BS, no affinity
+#   sticky       hard session affinity
+set -uo pipefail
+export ROOT=/home/admin/cpfs/wjh/agentic-kv
+export MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct
+TRACE="${TRACE:-$ROOT/traces/w600_r0.0015_st30_first600s.jsonl}"
+DATE="$(date +%Y%m%d_%H%M)"
+OUT="${OUTROOT:-$ROOT/outputs/policy5_600s_$DATE}"
+mkdir -p "$OUT"
+echo "OUTROOT=$OUT" | tee "$OUT/RUNINFO.txt"
+echo "TRACE=$TRACE  ($(wc -l < "$TRACE") reqs)" | tee -a "$OUT/RUNINFO.txt"
+date | tee -a "$OUT/RUNINFO.txt"
+
+run_arm() {  # name policy extra_args
+    local name="$1" policy="$2" extra="$3"
+    echo "===== ARM: $name (policy=$policy args='$extra') =====" | tee -a "$OUT/RUNINFO.txt"
+    local t0=$(date +%s)
+    EXTRA_PROXY_ARGS="$extra" bash "$ROOT/scripts/b3_isolated_policy.sh" \
+        "$policy" "$TRACE" "$OUT/$name" > "$OUT/$name.log" 2>&1
+    echo "$name rc=$? rows=$(wc -l < "$OUT/$name/metrics.jsonl" 2>/dev/null) dur=$(( $(date +%s) - t0 ))s" | tee -a "$OUT/RUNINFO.txt"
+}
+
+# Headline contrasts first (so a late failure still leaves the key arms).
+run_arm leastwork   leastwork ""
+run_arm unified_ab  unified   "--overload-factor 1.3 --lmetric-decode-weight 0.01"
+run_arm unified_def unified   "--overload-factor 2.0 --lmetric-decode-weight 0.0"
+run_arm lmetric     lmetric   ""
+run_arm sticky      sticky    ""
+
+echo "===== ALL DONE: $OUT =====" | tee -a "$OUT/RUNINFO.txt"
+date | tee -a "$OUT/RUNINFO.txt"

microbench/connector_tax/cache_sweep/run_5policy_both_modes.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+# 5-policy comparison in BOTH dispatch modes on the SAME ttp-annotated trace,
+# so the only variable is dispatch-mode (tracets vs thinktime). Fresh vLLM
+# (cold APC) per arm via run_5policy_600s.sh -> b3_isolated_policy.sh.
+set -uo pipefail
+ROOT="${ROOT:-/home/admin/cpfs/wjh/agentic-kv}"
+TRACE_FILE="${TRACE_FILE:-$ROOT/traces/w600_r0.0015_st30_first600s_ttp.jsonl}"
+RUN5="$ROOT/microbench/connector_tax/cache_sweep/run_5policy_600s.sh"
+DATE="$(date +%Y%m%d_%H%M)"
+
+echo "=== 5policy x {tracets,thinktime} | trace=$(basename "$TRACE_FILE") | $DATE ==="
+for MODE in tracets thinktime; do
+    OUT="$ROOT/outputs/policy5_600s_${MODE}_${DATE}"
+    echo "############ MODE=$MODE  OUT=$OUT  $(date) ############"
+    TRACE="$TRACE_FILE" REPLAY_DISPATCH_MODE="$MODE" OUTROOT="$OUT" \
+        bash "$RUN5"
+    echo "dispatch_mode=$MODE" >> "$OUT/RUNINFO.txt"
+    echo "trace=$TRACE_FILE" >> "$OUT/RUNINFO.txt"
+done
+echo "=== ALL DONE (both modes) $(date) ==="

microbench/connector_tax/cache_sweep/run_b3_replay.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# B3 routing-policy reproducibility re-test.
+#
+# Re-runs the 5 routing policies from fig_b3_latency_bars.png on the same
+# trace, in a single same-day session, to check whether the ordering
+# (unified < load_only < sticky etc.) still holds today.
+#
+# Policies (in run order):
+#   lmetric        plain — cache-aware P_tokens × BS
+#   load_only      plain — pure min-num_requests
+#   sticky         plain — hard session affinity
+#   unified        plain — hybrid affinity + LMetric fallback
+#   unified_v2     Mooncake kv_both + selective PD-sep (with DR-fix applied)
+#
+# unified_v2 is run with VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC=1 so we
+# get the "best Mooncake state" we have today (DR-fix on top of the
+# already-fixed mainline after e3a1d70 etc.). The other 4 policies don't
+# load any connector so the patch is irrelevant.
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+TRACE="${TRACE:-$PROJ_DIR/traces/w600_r0.0015_st30.jsonl}"
+DATE="$(date +%Y%m%d_%H%M)"
+OUTROOT="${OUTROOT:-$PROJ_DIR/outputs/b3_replay_${DATE}}"
+PYTHON="$PROJ_DIR/.venv/bin/python"
+DR_FIX_SCRIPT="$PROJ_DIR/microbench/connector_tax/cache_sweep/apply_direct_read_fix.py"
+VLLM_ROOT="${VLLM_ROOT:-$PROJ_DIR/.venv/lib/python3.12/site-packages/vllm}"
+
+mkdir -p "$OUTROOT"
+echo "=== B3 5-policy re-test ==="
+echo "Trace : $TRACE"
+echo "Out   : $OUTROOT"
+echo "Order : lmetric → load_only → sticky → unified → unified_v2 (DR-fix on)"
+echo ""
+
+cleanup_all() {
+    pkill -9 -f cache_aware_proxy 2>/dev/null || true
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 5
+    "$PYTHON" "$DR_FIX_SCRIPT" --revert --vllm-root "$VLLM_ROOT" 2>/dev/null || true
+}
+trap cleanup_all EXIT
+cleanup_all
+
+# Apply DR-fix once — it's env-gated so only unified_v2 (with env=1) sees it
+echo "[stage 0] applying CT_DR_FIX (env-gated, only activates when VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC=1)"
+"$PYTHON" "$DR_FIX_SCRIPT" --apply --vllm-root "$VLLM_ROOT"
+
+run_policy() {
+    local policy="$1"
+    local skip_dr="$2"
+    local rundir="$OUTROOT/$policy"
+    mkdir -p "$rundir"
+
+    echo ""
+    echo "====== $policy ; DR_SYNC_DISABLED=$skip_dr ======"
+
+    if [ "$skip_dr" = "1" ]; then
+        export VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC=1
+    else
+        unset VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC
+    fi
+
+    bash "$PROJ_DIR/scripts/b3_isolated_policy.sh" "$policy" "$TRACE" "$rundir" \
+        2>&1 | tee "$rundir/orchestrator.log" | tail -30
+    rc="${PIPESTATUS[0]}"
+    if [ "$rc" != "0" ]; then
+        echo "[FAIL] policy $policy rc=$rc"
+    fi
+    # Belt-and-braces cleanup between policies
+    pkill -9 -f cache_aware_proxy 2>/dev/null || true
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 10
+    return 0
+}
+
+run_policy "lmetric"    "0"
+run_policy "load_only"  "0"
+run_policy "sticky"     "0"
+run_policy "unified"    "0"
+run_policy "unified_v2" "1"  # uses Mooncake kv_both; activate DR-fix
+
+echo ""
+echo "[stage Z] reverting CT_DR_FIX"
+"$PYTHON" "$DR_FIX_SCRIPT" --revert --vllm-root "$VLLM_ROOT"
+
+echo ""
+echo "Done. Artifacts: $OUTROOT"
+for p in lmetric load_only sticky unified unified_v2; do
+    echo "  $p: $OUTROOT/$p/metrics.jsonl"
+done

microbench/connector_tax/cache_sweep/run_smoke_nixl.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+# Smoke test for Nixl-based PD-sep migration (NVLink intra-node via UCX).
+#
+# Drops 2 vLLM kv_both NixlConnector instances on GPU 0,1 and runs
+# smoke_test_migrate_cache.py against them with the kv_transfer_params
+# format Nixl expects (only do_remote_decode on src; proxy must forward
+# kv_transfer_params from src's response to dst).
+#
+# Since smoke_test_migrate_cache.py is currently hard-coded for Mooncake
+# (transfer_id + remote_bootstrap_addr), we use a tiny Python in-line
+# variant here that does the Nixl response-forward handshake directly.
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+MODEL="${MODEL:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
+VENV="$PROJ_DIR/.venv/bin"
+LOGS_DIR="${LOGS_DIR:-$PROJ_DIR/outputs/smoke_nixl_$(date +%Y%m%d_%H%M%S)}"
+mkdir -p "$LOGS_DIR"
+
+cleanup() {
+    echo "[smoke-nixl] cleaning up vLLM..."
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 2
+}
+trap cleanup EXIT
+cleanup
+
+echo "[smoke-nixl] starting 2 vLLM kv_both NixlConnector on GPU 0,1"
+for i in 0 1; do
+    port=$((8000 + i))
+    nixl_port=$((5600 + i))
+    master=$((29500 + i))
+    PYTHONHASHSEED=42 \
+    VLLM_NIXL_SIDE_CHANNEL_PORT=$nixl_port \
+    CUDA_VISIBLE_DEVICES=$i \
+    MASTER_PORT=$master \
+    nohup "$VENV/vllm" serve "$MODEL" \
+        --host 0.0.0.0 --port "$port" \
+        --tensor-parallel-size 1 \
+        --trust-remote-code --enable-prefix-caching \
+        --dtype auto --gpu-memory-utilization 0.9 \
+        --max-model-len 200000 \
+        --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' \
+        --enable-prompt-tokens-details \
+        > "$LOGS_DIR/vllm_inst_${i}_gpu${i}.log" 2>&1 &
+    disown
+    sleep 2
+done
+
+echo "[smoke-nixl] waiting for health on 8000 and 8001 ..."
+for port in 8000 8001; do
+    tries=0
+    while ! curl -sf "http://127.0.0.1:$port/health" >/dev/null 2>&1; do
+        tries=$((tries+1))
+        if [ $tries -gt 240 ]; then
+            echo "[smoke-nixl] FATAL: $port not ready"; exit 1
+        fi
+        sleep 2
+    done
+    echo "  port=$port ready"
+done
+
+echo "[smoke-nixl] running smoke_nixl_migrate.py"
+"$VENV/python" "$PROJ_DIR/microbench/connector_tax/cache_sweep/smoke_nixl_migrate.py" \
+    --src-port 8000 --dst-port 8001 \
+    ${EXTRA_SMOKE_ARGS:-} \
+    2>&1 | tee "$LOGS_DIR/smoke_output.log"
+
+ec=${PIPESTATUS[0]}
+echo "[smoke-nixl] test exit=$ec, logs at $LOGS_DIR"
+exit $ec

microbench/connector_tax/cache_sweep/run_smoke_partial.sh

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+# Smoke test for Mechanism B (partial KV transfer):
+# Start 3 vLLM kv_both Mooncake instances on GPU 0,1,2:
+#   - inst_0 = src (port 8000, bp 8998)
+#   - inst_1 = dst_warm (port 8001, bp 8999) — will be pre-warmed
+#   - inst_2 = dst_cold (port 8002, bp 9000) — control, no cache
+#
+# Then run smoke_partial_transfer.py which migrates the same prompt
+# to both warm and cold dst, comparing transfer cost.
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+MODEL="${MODEL:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
+VENV="$PROJ_DIR/.venv/bin"
+LOGS_DIR="${LOGS_DIR:-$PROJ_DIR/outputs/smoke_partial_$(date +%Y%m%d_%H%M%S)}"
+mkdir -p "$LOGS_DIR"
+
+cleanup() {
+    echo "[smoke-partial] cleaning up vLLM..."
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 2
+}
+trap cleanup EXIT
+cleanup
+
+echo "[smoke-partial] starting 3 vLLM kv_both Mooncake on GPU 0,1,2"
+for i in 0 1 2; do
+    port=$((8000 + i))
+    bp=$((8998 + i))
+    master=$((29500 + i))
+    PYTHONHASHSEED=42 \
+    VLLM_MOONCAKE_BOOTSTRAP_PORT=$bp \
+    CUDA_VISIBLE_DEVICES=$i \
+    MASTER_PORT=$master \
+    nohup "$VENV/vllm" serve "$MODEL" \
+        --host 0.0.0.0 --port "$port" \
+        --tensor-parallel-size 1 \
+        --trust-remote-code --enable-prefix-caching \
+        --dtype auto --gpu-memory-utilization 0.9 \
+        --max-model-len 200000 \
+        --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \
+        --enable-prompt-tokens-details \
+        > "$LOGS_DIR/vllm_inst_${i}_gpu${i}.log" 2>&1 &
+    disown
+    sleep 2
+done
+
+echo "[smoke-partial] waiting for health ..."
+for port in 8000 8001 8002; do
+    tries=0
+    while ! curl -sf "http://127.0.0.1:$port/health" >/dev/null 2>&1; do
+        tries=$((tries+1))
+        if [ $tries -gt 240 ]; then echo "FATAL: $port"; exit 1; fi
+        sleep 2
+    done
+    echo "  port=$port ready"
+done
+
+echo "[smoke-partial] running smoke_partial_transfer.py"
+"$VENV/python" "$PROJ_DIR/microbench/connector_tax/cache_sweep/smoke_partial_transfer.py" \
+    ${EXTRA_SMOKE_ARGS:-} \
+    2>&1 | tee "$LOGS_DIR/smoke_output.log"
+
+ec=${PIPESTATUS[0]}
+echo "[smoke-partial] exit=$ec, logs at $LOGS_DIR"
+exit $ec

microbench/connector_tax/cache_sweep/run_smoke_sweep.sh

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+# Single vLLM warmup, multiple smoke-test iterations under varying load.
+#
+# Each iteration uses a distinct --prefix-base to avoid prefix-cache pollution
+# from prior iterations. We sweep noise levels 0, 8, 32, 64 to see at which
+# point the migration cache becomes invisible to the follow-up.
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+MODEL="${MODEL:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
+VENV="$PROJ_DIR/.venv/bin"
+LOGS_DIR="${LOGS_DIR:-$PROJ_DIR/outputs/smoke_sweep_$(date +%Y%m%d_%H%M%S)}"
+mkdir -p "$LOGS_DIR"
+
+cleanup() {
+    echo "[sweep] cleaning up vLLM..."
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 2
+}
+trap cleanup EXIT
+cleanup
+
+echo "[sweep] starting 2 vLLM kv_both on GPU 0,1"
+for i in 0 1; do
+    port=$((8000 + i))
+    bp=$((8998 + i))
+    master=$((29500 + i))
+    PYTHONHASHSEED=42 \
+    VLLM_MOONCAKE_BOOTSTRAP_PORT=$bp \
+    CUDA_VISIBLE_DEVICES=$i \
+    MASTER_PORT=$master \
+    nohup "$VENV/vllm" serve "$MODEL" \
+        --host 0.0.0.0 --port "$port" \
+        --tensor-parallel-size 1 \
+        --trust-remote-code --enable-prefix-caching \
+        --dtype auto --gpu-memory-utilization 0.9 \
+        --max-model-len 200000 \
+        --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \
+        --enable-prompt-tokens-details \
+        > "$LOGS_DIR/vllm_inst_${i}_gpu${i}.log" 2>&1 &
+    disown
+    sleep 2
+done
+
+echo "[sweep] waiting for health ..."
+for port in 8000 8001; do
+    tries=0
+    while ! curl -sf "http://127.0.0.1:$port/health" >/dev/null 2>&1; do
+        tries=$((tries+1))
+        if [ $tries -gt 180 ]; then echo "[sweep] FATAL: $port not ready"; exit 1; fi
+        sleep 2
+    done
+    echo "  port=$port ready"
+done
+
+base=100
+for noise in 0 8 32 64 128; do
+    echo ""
+    echo "============================================"
+    echo "[sweep] iteration noise=$noise prefix_base=$base"
+    echo "============================================"
+    "$VENV/python" "$PROJ_DIR/microbench/connector_tax/cache_sweep/smoke_test_migrate_cache.py" \
+        --src-port 8000 --dst-port 8001 \
+        --src-bp 8998 --dst-bp 8999 \
+        --noise-reqs "$noise" \
+        --prefix-base "$base" \
+        2>&1 | tee "$LOGS_DIR/iter_noise${noise}.log" | tail -25
+    base=$((base + 100000))
+done
+
+echo ""
+echo "[sweep] all iterations done; logs in $LOGS_DIR"

microbench/connector_tax/cache_sweep/run_smoke_test.sh

@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+# Fast iteration: start 2 vLLM kv_both, run smoke_test_migrate_cache, tear down.
+#
+# Usage: bash run_smoke_test.sh [WAIT_BETWEEN_S]
+#
+# Iteration overhead: ~3-4 min warmup + a few sec for the test. Cleanly
+# tears everything down on exit so you can re-run repeatedly.
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+MODEL="${MODEL:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
+VENV="$PROJ_DIR/.venv/bin"
+LOGS_DIR="${LOGS_DIR:-$PROJ_DIR/outputs/smoke_test_$(date +%Y%m%d_%H%M%S)}"
+mkdir -p "$LOGS_DIR"
+
+# MOONCAKE_PROTOCOL controls Mooncake's C++ TransferEngine transport.
+# Options exposed: rdma (default), tcp, nvlink_intra (NVLink intra-node).
+PROTO="${MOONCAKE_PROTOCOL:-rdma}"
+echo "[smoke] using Mooncake protocol: $PROTO"
+
+cleanup() {
+    echo "[smoke] cleaning up vLLM..."
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 2
+}
+trap cleanup EXIT
+cleanup
+
+echo "[smoke] starting 2 vLLM kv_both on GPU 0,1"
+for i in 0 1; do
+    port=$((8000 + i))
+    bp=$((8998 + i))
+    master=$((29500 + i))
+    PYTHONHASHSEED=42 \
+    VLLM_MOONCAKE_BOOTSTRAP_PORT=$bp \
+    CUDA_VISIBLE_DEVICES=$i \
+    MASTER_PORT=$master \
+    nohup "$VENV/vllm" serve "$MODEL" \
+        --host 0.0.0.0 --port "$port" \
+        --tensor-parallel-size 1 \
+        --trust-remote-code --enable-prefix-caching \
+        --dtype auto --gpu-memory-utilization 0.9 \
+        --max-model-len 200000 \
+        --kv-transfer-config "{\"kv_connector\":\"MooncakeConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"mooncake_protocol\":\"$PROTO\"}}" \
+        --enable-prompt-tokens-details \
+        > "$LOGS_DIR/vllm_inst_${i}_gpu${i}.log" 2>&1 &
+    disown
+    sleep 2
+done
+
+echo "[smoke] waiting for health on 8000 and 8001 ..."
+for port in 8000 8001; do
+    tries=0
+    while ! curl -sf "http://127.0.0.1:$port/health" >/dev/null 2>&1; do
+        tries=$((tries+1))
+        if [ $tries -gt 180 ]; then
+            echo "[smoke] FATAL: $port not ready"; exit 1
+        fi
+        sleep 2
+    done
+    echo "  port=$port ready"
+done
+
+echo "[smoke] running migration smoke test"
+"$VENV/python" "$PROJ_DIR/microbench/connector_tax/cache_sweep/smoke_test_migrate_cache.py" \
+    --src-port 8000 --dst-port 8001 \
+    --src-bp 8998 --dst-bp 8999 \
+    ${EXTRA_SMOKE_ARGS:-} \
+    2>&1 | tee "$LOGS_DIR/smoke_output.log"
+
+ec=${PIPESTATUS[0]}
+echo "[smoke] test exit=$ec, logs at $LOGS_DIR"
+exit $ec

microbench/connector_tax/cache_sweep/run_unified_ablation.sh

@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# Single-policy trace replay for unified, with tunable overload-factor.
+# Used to test direction A: does tightening affinity overflow improve unified?
+#
+# Usage:
+#   OVERLOAD_FACTOR=1.3 bash run_unified_ablation.sh
+#   OVERLOAD_FACTOR=1.0 bash run_unified_ablation.sh
+#
+# Output: $PROJ_DIR/outputs/unified_of${OF}_${DATE}/unified/
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+TRACE="${TRACE:-$PROJ_DIR/traces/w600_r0.0015_st30.jsonl}"
+OF="${OVERLOAD_FACTOR:-1.3}"
+LMW="${LMETRIC_DECODE_WEIGHT:-0.0}"
+TAG_DEFAULT="of${OF/./}"
+if [ "$(printf '%s' "$LMW" | grep -v '^0\.\?0*$' || true)" != "" ]; then
+    TAG_DEFAULT="${TAG_DEFAULT}_lmw${LMW/./}"
+fi
+TAG="${TAG:-$TAG_DEFAULT}"
+DATE="$(date +%Y%m%d_%H%M)"
+OUTROOT="${OUTROOT:-$PROJ_DIR/outputs/unified_${TAG}_${DATE}}"
+
+mkdir -p "$OUTROOT"
+echo "=== unified ablation: overload_factor=$OF ==="
+echo "Trace : $TRACE"
+echo "Out   : $OUTROOT"
+echo ""
+
+cleanup() {
+    pkill -9 -f cache_aware_proxy 2>/dev/null || true
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 3
+}
+trap cleanup EXIT
+cleanup
+
+cfg_dir="$OUTROOT/unified"
+mkdir -p "$cfg_dir"
+
+export EXTRA_PROXY_ARGS="--overload-factor $OF --lmetric-decode-weight $LMW"
+
+echo ""
+echo "====== unified ; overload_factor=$OF lmetric_decode_weight=$LMW ======"
+bash "$PROJ_DIR/scripts/b3_isolated_policy.sh" "unified" "$TRACE" "$cfg_dir" \
+    2>&1 | tee "$cfg_dir/orchestrator.log" | tail -30
+
+pkill -9 -f cache_aware_proxy 2>/dev/null || true
+pkill -9 -f "vllm serve" 2>/dev/null || true
+pkill -9 -f "EngineCore" 2>/dev/null || true
+sleep 5
+
+echo ""
+echo "Done. Artifacts: $OUTROOT/unified/metrics.jsonl"

microbench/connector_tax/cache_sweep/run_v3_dst_breakdown.sh

@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+# v3 trace replay with dst-side migration breakdown instrumentation.
+#
+# Same trace + DR_FIX as `run_v3_replay.sh`, plus:
+#   - instrument_dst_migration.py applied to vLLM scheduler
+#   - DM_LOG_DIR exported to all 8 vLLM instances so per-PID
+#     dst-migration logs land in <RUNDIR>/dst_mig_log/
+#   - analyze_dst_migration.py runs on completion to print the
+#     T_kv_pull vs queue-residual decomposition
+#
+# Usage: bash run_v3_dst_breakdown.sh
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+TRACE="${TRACE:-$PROJ_DIR/traces/w600_r0.0015_st30.jsonl}"
+DATE="$(date +%Y%m%d_%H%M)"
+OUTROOT="${OUTROOT:-$PROJ_DIR/outputs/b3_v3_dstbreak_${DATE}}"
+PYTHON="$PROJ_DIR/.venv/bin/python"
+VLLM_ROOT="${VLLM_ROOT:-$PROJ_DIR/.venv/lib/python3.12/site-packages/vllm}"
+DR_FIX_SCRIPT="$PROJ_DIR/microbench/connector_tax/cache_sweep/apply_direct_read_fix.py"
+DM_INSTRUMENT="$PROJ_DIR/microbench/fresh_setup/instrument_dst_migration.py"
+ANALYZE="$PROJ_DIR/microbench/connector_tax/cache_sweep/analyze_dst_migration.py"
+
+mkdir -p "$OUTROOT"
+DST_LOG_DIR="$OUTROOT/dst_mig_log"
+mkdir -p "$DST_LOG_DIR"
+
+echo "=== unified_v3 + dst-side migration breakdown ==="
+echo "Trace      : $TRACE"
+echo "Out        : $OUTROOT"
+echo "DST logs   : $DST_LOG_DIR"
+echo ""
+
+cleanup_all() {
+    pkill -9 -f cache_aware_proxy 2>/dev/null || true
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 5
+    "$PYTHON" "$DR_FIX_SCRIPT" --revert --vllm-root "$VLLM_ROOT" 2>/dev/null || true
+    "$PYTHON" "$DM_INSTRUMENT" --revert --venv "$PROJ_DIR/.venv" 2>/dev/null || true
+}
+trap cleanup_all EXIT
+cleanup_all
+
+echo "[stage 0a] applying CT_DR_FIX (env-gated)"
+"$PYTHON" "$DR_FIX_SCRIPT" --apply --vllm-root "$VLLM_ROOT"
+
+echo "[stage 0b] applying DST migration instrumentation"
+"$PYTHON" "$DM_INSTRUMENT" --apply --venv "$PROJ_DIR/.venv"
+"$PYTHON" "$DM_INSTRUMENT" --check --venv "$PROJ_DIR/.venv"
+
+cfg_dir="$OUTROOT/unified_v3"
+mkdir -p "$cfg_dir"
+
+# Activate DR-fix env gate (consistent with run_v3_replay.sh)
+export VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC=1
+# Export DM_LOG_DIR — every vLLM EngineCore inherits this env and writes
+# its own dm_mig_pid<pid>.jsonl into it.
+export DM_LOG_DIR="$DST_LOG_DIR"
+
+echo ""
+echo "====== unified_v3 ; DR_SYNC_DISABLED=1 ; DM_LOG_DIR=$DST_LOG_DIR ======"
+bash "$PROJ_DIR/scripts/b3_isolated_policy.sh" "unified_v3" "$TRACE" "$cfg_dir" \
+    2>&1 | tee "$cfg_dir/orchestrator.log" | tail -30
+
+pkill -9 -f cache_aware_proxy 2>/dev/null || true
+pkill -9 -f "vllm serve" 2>/dev/null || true
+pkill -9 -f "EngineCore" 2>/dev/null || true
+sleep 5
+
+echo ""
+echo "[stage Z] reverting DR_FIX + DM instrument"
+"$PYTHON" "$DR_FIX_SCRIPT" --revert --vllm-root "$VLLM_ROOT"
+"$PYTHON" "$DM_INSTRUMENT" --revert --venv "$PROJ_DIR/.venv"
+
+echo ""
+echo "[stage analyze] dst-side migration breakdown"
+"$PYTHON" "$ANALYZE" \
+    --proxy-breakdown "$cfg_dir/breakdown.json" \
+    --dst-log-dir "$DST_LOG_DIR" \
+    --output "$cfg_dir/dst_migration_breakdown.csv" \
+    --plot "$cfg_dir/dst_migration_breakdown.png" \
+    2>&1 | tee "$cfg_dir/dst_migration_breakdown.txt"
+
+echo ""
+echo "Done."
+echo "  proxy breakdown : $cfg_dir/breakdown.json"
+echo "  dst per-PID log : $DST_LOG_DIR/"
+echo "  decomposition   : $cfg_dir/dst_migration_breakdown.{csv,png,txt}"

microbench/connector_tax/cache_sweep/run_v3_full_breakdown.sh

@@ -0,0 +1,102 @@
+#!/usr/bin/env bash
+# v3 trace replay with FULL migration instrumentation:
+#   - instrument_dst_migration.py  : dst lifecycle -> T_kv_pull
+#   - instrument_mooncake.py        : connector internals (send_blocks RDMA,
+#                                      receive_kv window, ready_wait)
+# Goal: decompose the 87% T_kv_pull into RDMA-actual vs control-plane to
+# explain why effective bandwidth is far below the ~10 GB/s wire rate.
+#
+# Usage: bash run_v3_full_breakdown.sh
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+TRACE="${TRACE:-$PROJ_DIR/traces/w600_r0.0015_st30.jsonl}"
+DATE="$(date +%Y%m%d_%H%M)"
+OUTROOT="${OUTROOT:-$PROJ_DIR/outputs/b3_v3_fullbreak_${DATE}}"
+PYTHON="$PROJ_DIR/.venv/bin/python"
+VENV="$PROJ_DIR/.venv"
+VLLM_ROOT="${VLLM_ROOT:-$VENV/lib/python3.12/site-packages/vllm}"
+DR_FIX="$PROJ_DIR/microbench/connector_tax/cache_sweep/apply_direct_read_fix.py"
+DM_INSTR="$PROJ_DIR/microbench/fresh_setup/instrument_dst_migration.py"
+MC_INSTR="$PROJ_DIR/microbench/fresh_setup/instrument_mooncake.py"
+ANALYZE_DST="$PROJ_DIR/microbench/connector_tax/cache_sweep/analyze_dst_migration.py"
+ANALYZE_XFER="$PROJ_DIR/microbench/connector_tax/cache_sweep/analyze_transfer_decomp.py"
+
+mkdir -p "$OUTROOT"
+DST_LOG_DIR="$OUTROOT/dst_mig_log"
+XFER_LOG_DIR="$OUTROOT/xfer_log"
+mkdir -p "$DST_LOG_DIR" "$XFER_LOG_DIR"
+
+echo "=== unified_v3 + FULL migration breakdown ==="
+echo "Out      : $OUTROOT"
+echo "DST logs : $DST_LOG_DIR"
+echo "XFER logs: $XFER_LOG_DIR"
+echo ""
+
+cleanup_all() {
+    pkill -9 -f cache_aware_proxy 2>/dev/null || true
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 5
+    "$PYTHON" "$DR_FIX"   --revert --vllm-root "$VLLM_ROOT" 2>/dev/null || true
+    "$PYTHON" "$DM_INSTR" --revert --venv "$VENV" 2>/dev/null || true
+    "$PYTHON" "$MC_INSTR" --revert --venv "$VLLM_ROOT/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py" 2>/dev/null || true
+}
+trap cleanup_all EXIT
+cleanup_all
+
+echo "[0a] DR_FIX"
+"$PYTHON" "$DR_FIX" --apply --vllm-root "$VLLM_ROOT"
+echo "[0b] DST migration instrument"
+"$PYTHON" "$DM_INSTR" --apply --venv "$VENV"
+echo "[0c] Mooncake transfer instrument"
+"$PYTHON" "$MC_INSTR" --apply --venv "$VLLM_ROOT/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py"
+"$PYTHON" "$DM_INSTR" --check --venv "$VENV"
+"$PYTHON" "$MC_INSTR" --check --venv "$VLLM_ROOT/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py"
+
+cfg_dir="$OUTROOT/unified_v3"
+mkdir -p "$cfg_dir"
+
+export VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC=1
+export DM_LOG_DIR="$DST_LOG_DIR"
+export MB2_LOG_DIR="$XFER_LOG_DIR"
+
+echo ""
+echo "====== unified_v3 ; DM_LOG_DIR + MB2_LOG_DIR set ======"
+bash "$PROJ_DIR/scripts/b3_isolated_policy.sh" "unified_v3" "$TRACE" "$cfg_dir" \
+    2>&1 | tee "$cfg_dir/orchestrator.log" | tail -25
+
+pkill -9 -f cache_aware_proxy 2>/dev/null || true
+pkill -9 -f "vllm serve" 2>/dev/null || true
+pkill -9 -f "EngineCore" 2>/dev/null || true
+sleep 5
+
+echo ""
+echo "[Z] revert all instruments"
+"$PYTHON" "$DR_FIX"   --revert --vllm-root "$VLLM_ROOT"
+"$PYTHON" "$DM_INSTR" --revert --venv "$VENV"
+"$PYTHON" "$MC_INSTR" --revert --venv "$VLLM_ROOT/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py"
+
+echo ""
+echo "[analyze 1] dst-side T_kv_pull breakdown"
+"$PYTHON" "$ANALYZE_DST" \
+    --proxy-breakdown "$cfg_dir/breakdown.json" \
+    --dst-log-dir "$DST_LOG_DIR" \
+    --output "$cfg_dir/dst_migration_breakdown.csv" \
+    --plot "$cfg_dir/dst_migration_breakdown.png" \
+    2>&1 | tee "$cfg_dir/dst_migration_breakdown.txt" || echo "(dst analyze failed)"
+
+echo ""
+echo "[analyze 2] transfer decomposition: RDMA-actual vs control-plane"
+"$PYTHON" "$ANALYZE_XFER" \
+    --proxy-breakdown "$cfg_dir/breakdown.json" \
+    --dst-log-dir "$DST_LOG_DIR" \
+    --xfer-log-dir "$XFER_LOG_DIR" \
+    2>&1 | tee "$cfg_dir/transfer_decomp.txt" || echo "(xfer analyze failed)"
+
+echo ""
+echo "Done. Artifacts in $cfg_dir/"
+echo "  dst_migration_breakdown.{csv,png,txt}"
+echo "  transfer_decomp.txt"
+echo "  raw: $DST_LOG_DIR/  $XFER_LOG_DIR/"

microbench/connector_tax/cache_sweep/run_v3_norot_replay.sh

@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+# Trace replay for unified_v3 WITHOUT affinity rotation.
+#
+# This is the #2 follow-up to cache_miss_audit: prior run showed v3 with
+# rotation hits 9.5% cache on post-migration next turn vs 80.6% for unified.
+# Hypothesis: rotation destroys prefix cache locality. Test by keeping the
+# session affinity on prefill_host even after migration (i.e., the same
+# behavior as unified for the post-migration write), so only the *current*
+# turn's decode is migrated.
+#
+# Applies CT_DR_FIX (Mooncake DR sync disabled).
+# Output: $PROJ_DIR/outputs/b3_v3_norot_${DATE}/unified_v3/
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+TRACE="${TRACE:-$PROJ_DIR/traces/w600_r0.0015_st30.jsonl}"
+DATE="$(date +%Y%m%d_%H%M)"
+OUTROOT="${OUTROOT:-$PROJ_DIR/outputs/b3_v3_norot_${DATE}}"
+PYTHON="$PROJ_DIR/.venv/bin/python"
+DR_FIX_SCRIPT="$PROJ_DIR/microbench/connector_tax/cache_sweep/apply_direct_read_fix.py"
+VLLM_ROOT="${VLLM_ROOT:-$PROJ_DIR/.venv/lib/python3.12/site-packages/vllm}"
+
+mkdir -p "$OUTROOT"
+echo "=== unified_v3 (no rotation) trace replay ==="
+echo "Trace : $TRACE"
+echo "Out   : $OUTROOT"
+echo ""
+
+cleanup_all() {
+    pkill -9 -f cache_aware_proxy 2>/dev/null || true
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 5
+    "$PYTHON" "$DR_FIX_SCRIPT" --revert --vllm-root "$VLLM_ROOT" 2>/dev/null || true
+}
+trap cleanup_all EXIT
+cleanup_all
+
+echo "[stage 0] applying CT_DR_FIX (env-gated)"
+"$PYTHON" "$DR_FIX_SCRIPT" --apply --vllm-root "$VLLM_ROOT"
+
+cfg_dir="$OUTROOT/unified_v3"
+mkdir -p "$cfg_dir"
+
+export VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC=1
+export EXTRA_PROXY_ARGS="--v3-rotate-affinity 0"
+
+echo ""
+echo "====== unified_v3 (no rotation) ; DR_SYNC_DISABLED=1 ======"
+bash "$PROJ_DIR/scripts/b3_isolated_policy.sh" "unified_v3" "$TRACE" "$cfg_dir" \
+    2>&1 | tee "$cfg_dir/orchestrator.log" | tail -30
+
+pkill -9 -f cache_aware_proxy 2>/dev/null || true
+pkill -9 -f "vllm serve" 2>/dev/null || true
+pkill -9 -f "EngineCore" 2>/dev/null || true
+sleep 5
+
+echo ""
+echo "[stage Z] reverting CT_DR_FIX"
+"$PYTHON" "$DR_FIX_SCRIPT" --revert --vllm-root "$VLLM_ROOT"
+
+echo ""
+echo "Done. Artifacts: $OUTROOT/unified_v3/metrics.jsonl"

microbench/connector_tax/cache_sweep/run_v3_replay.sh

@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+# Trace replay for the new unified_v3 (offload-decode) policy.
+#
+# Runs the same trace as run_b3_replay.sh on a single policy:
+#   unified_v3 — prefill on session-affinity host (uses prefix cache),
+#                decode migrated to a low-load target via Mooncake
+#                KV transfer (kv_role=kv_both). Session affinity rotates
+#                to decode_target after migration so next turn lands
+#                where the KV now lives.
+#
+# Applies CT_DR_FIX so the run uses the "best Mooncake state" we have
+# today (post-e3a1d70 + DR sync skipped).
+#
+# Usage: bash run_v3_replay.sh
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+TRACE="${TRACE:-$PROJ_DIR/traces/w600_r0.0015_st30.jsonl}"
+DATE="$(date +%Y%m%d_%H%M)"
+OUTROOT="${OUTROOT:-$PROJ_DIR/outputs/b3_v3_${DATE}}"
+PYTHON="$PROJ_DIR/.venv/bin/python"
+DR_FIX_SCRIPT="$PROJ_DIR/microbench/connector_tax/cache_sweep/apply_direct_read_fix.py"
+VLLM_ROOT="${VLLM_ROOT:-$PROJ_DIR/.venv/lib/python3.12/site-packages/vllm}"
+
+mkdir -p "$OUTROOT"
+echo "=== unified_v3 (offload-decode) trace replay ==="
+echo "Trace : $TRACE"
+echo "Out   : $OUTROOT"
+echo ""
+
+cleanup_all() {
+    pkill -9 -f cache_aware_proxy 2>/dev/null || true
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 5
+    "$PYTHON" "$DR_FIX_SCRIPT" --revert --vllm-root "$VLLM_ROOT" 2>/dev/null || true
+}
+trap cleanup_all EXIT
+cleanup_all
+
+echo "[stage 0] applying CT_DR_FIX (env-gated)"
+"$PYTHON" "$DR_FIX_SCRIPT" --apply --vllm-root "$VLLM_ROOT"
+
+cfg_dir="$OUTROOT/unified_v3"
+mkdir -p "$cfg_dir"
+
+# Activate the DR-fix env-gate (unified_v3 uses Mooncake kv_both)
+export VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC=1
+
+echo ""
+echo "====== unified_v3 ; DR_SYNC_DISABLED=1 ======"
+bash "$PROJ_DIR/scripts/b3_isolated_policy.sh" "unified_v3" "$TRACE" "$cfg_dir" \
+    2>&1 | tee "$cfg_dir/orchestrator.log" | tail -30
+
+pkill -9 -f cache_aware_proxy 2>/dev/null || true
+pkill -9 -f "vllm serve" 2>/dev/null || true
+pkill -9 -f "EngineCore" 2>/dev/null || true
+sleep 5
+
+echo ""
+echo "[stage Z] reverting CT_DR_FIX"
+"$PYTHON" "$DR_FIX_SCRIPT" --revert --vllm-root "$VLLM_ROOT"
+
+echo ""
+echo "Done. Artifacts: $OUTROOT/unified_v3/metrics.jsonl"

microbench/connector_tax/cache_sweep/smoke_nixl_migrate.py

@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+"""Smoke test for Nixl PD-sep migration.
+
+Nixl handshake (vs Mooncake's pre-baked engine_id):
+  1. POST to src with kv_transfer_params={"do_remote_decode": True},
+     max_tokens=1, stream=False.
+  2. src returns kv_transfer_params in the response body containing
+     remote_block_ids, remote_engine_id, remote_host, remote_port,
+     remote_request_id, tp_size.
+  3. POST to dst with the SAME kv_transfer_params dict.
+  4. dst pulls KV via UCX (NVLink intra-node) and decodes.
+
+Verifies migration correctness + measures KV transfer latency on Nixl
+so we can ablate vs Mooncake/RDMA on the same workload.
+"""
+import asyncio, argparse, json, sys, uuid, time
+import httpx
+import random as _r
+
+MODEL = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct"
+
+
+async def send(client, port, prompt, max_tokens, kv_xfer, stream):
+    payload = {
+        "model": MODEL,
+        "prompt": prompt,
+        "max_tokens": max_tokens,
+        "min_tokens": max_tokens if max_tokens == 1 else 1,
+        "temperature": 0.0,
+        "stream": stream,
+    }
+    if kv_xfer is not None:
+        payload["kv_transfer_params"] = kv_xfer
+    if stream:
+        payload["stream_options"] = {"include_usage": True}
+    url = f"http://127.0.0.1:{port}/v1/completions"
+    if not stream:
+        r = await client.post(url, json=payload, timeout=300.0)
+        r.raise_for_status()
+        return r.json()
+    last_with_usage = None; last_any = None
+    async with client.stream("POST", url, json=payload, timeout=300.0) as resp:
+        resp.raise_for_status()
+        buf = ""
+        async for chunk in resp.aiter_bytes():
+            buf += chunk.decode("utf-8", errors="replace")
+            while "\n\n" in buf:
+                line, buf = buf.split("\n\n", 1)
+                if line.startswith("data: "):
+                    s = line[6:].strip()
+                    if s == "[DONE]": continue
+                    try:
+                        d = json.loads(s); last_any = d
+                        if d.get("usage"): last_with_usage = d
+                    except Exception:
+                        pass
+    return last_with_usage or last_any or {}
+
+
+def short(d):
+    if not d: return "no_resp"
+    usage = d.get("usage") or {}
+    details = usage.get("prompt_tokens_details") or {}
+    cached = details.get("cached_tokens", 0) or usage.get("cached_tokens", 0)
+    return (f"cached={cached}/{usage.get('prompt_tokens',0)} "
+            f"completion={usage.get('completion_tokens',0)}")
+
+
+async def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--src-port", type=int, default=8000)
+    p.add_argument("--dst-port", type=int, default=8001)
+    p.add_argument("--n-prefix-tokens", type=int, default=8192)
+    p.add_argument("--n-extension", type=int, default=32)
+    p.add_argument("--decode-tokens", type=int, default=16)
+    p.add_argument("--prefix-base", type=int, default=100)
+    args = p.parse_args()
+
+    rng = _r.Random(f"prefix-{args.prefix_base}")
+    prompt = [rng.randint(1024, 99_999) for _ in range(args.n_prefix_tokens)]
+
+    async with httpx.AsyncClient() as client:
+        # ----- Step 1: src prefills with do_remote_decode=True -----
+        t_src_start = time.monotonic()
+        src_resp = await send(
+            client, args.src_port, prompt, max_tokens=1,
+            kv_xfer={"do_remote_decode": True}, stream=False,
+        )
+        t_src_done = time.monotonic()
+        src_kv = src_resp.get("kv_transfer_params")
+        print(f"[1] src prefill ({(t_src_done-t_src_start)*1000:.0f}ms): {short(src_resp)}")
+        if not src_kv:
+            print(f"    FAIL: src returned no kv_transfer_params")
+            print(f"    response keys: {list(src_resp.keys())}")
+            sys.exit(1)
+        print(f"    src kv_transfer_params keys: {list(src_kv.keys())}")
+        print(f"    remote_block_ids: {len(src_kv.get('remote_block_ids', [[]])[0]) if src_kv.get('remote_block_ids') else 0} blocks")
+
+        # ----- Step 2: dst pulls KV using forwarded kv_transfer_params -----
+        t_dst_start = time.monotonic()
+        dst_resp = await send(
+            client, args.dst_port, prompt, max_tokens=args.decode_tokens,
+            kv_xfer=src_kv, stream=True,
+        )
+        t_dst_done = time.monotonic()
+        dst_total_ms = (t_dst_done - t_dst_start) * 1000
+        n_completion = (dst_resp.get("usage") or {}).get("completion_tokens", 0)
+        print(f"[2] dst decode ({dst_total_ms:.0f}ms, {n_completion} completion tokens): {short(dst_resp)}")
+        print(f"    [TIMING] proto=nixl src_prefill={int((t_src_done-t_src_start)*1000)}ms "
+              f"dst_total={int(dst_total_ms)}ms (KV xfer + {n_completion}-token decode)")
+        print()
+
+        # ----- Step 3: follow-up on dst (no kv_transfer_params) -----
+        ext = [_r.Random(f"ext-{args.prefix_base}").randint(1024, 99_999)
+               for _ in range(args.n_extension)]
+        follow_prompt = prompt + ext
+        fu = await send(client, args.dst_port, follow_prompt, max_tokens=4,
+                        kv_xfer=None, stream=False)
+        print(f"[3] follow-up dst (cache hit test): {short(fu)}")
+
+        usage_fu = fu.get("usage") or {}
+        details_fu = usage_fu.get("prompt_tokens_details") or {}
+        cached_fu = details_fu.get("cached_tokens", 0) or usage_fu.get("cached_tokens", 0)
+        expected_min = int(args.n_prefix_tokens * 0.95)
+        verdict = "PASS" if cached_fu >= expected_min else "FAIL"
+        print(f"\n=== verdict: {verdict} (follow-up cached={cached_fu}, "
+              f"expected >= {expected_min} of {args.n_prefix_tokens}) ===")
+        sys.exit(0 if verdict == "PASS" else 1)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

microbench/connector_tax/cache_sweep/smoke_partial_transfer.py

@@ -0,0 +1,166 @@
+#!/usr/bin/env python3
+"""Smoke test for partial KV transfer (Mechanism B).
+
+Test if vLLM's Mooncake connector actually transfers only the
+NON-OVERLAPPING portion when dst already has prefix cache.
+
+Sequence:
+  step 0: warm dst with prompt P (cold prefill) — dst now has cache for [0, P]
+  step 1: cold prefill on src with prompt P+ext  (src now has [0, P+ext])
+  step 2: migrate src→dst with prompt P+ext
+            - dst local cache should hit [0, P]
+            - only [P, P+ext] needs to come from src (~ext tokens)
+            - dst decode should be fast
+  step 3: control: another migrate src→dst_cold with prompt P+ext
+            - dst_cold has no cache, must pull all P+ext tokens
+            - compare with step 2
+
+If step 2 is dramatically faster than step 3, partial transfer works
+and Mechanism B is viable. If step 2 ~= step 3, partial transfer isn't
+being exploited and we need to dig deeper.
+"""
+import asyncio, argparse, json, sys, uuid, time
+import httpx
+import random as _r
+
+MODEL = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct"
+
+
+async def send(client, port, prompt, max_tokens, kv_xfer, stream):
+    payload = {
+        "model": MODEL,
+        "prompt": prompt,
+        "max_tokens": max_tokens,
+        "min_tokens": max_tokens if max_tokens == 1 else 1,
+        "temperature": 0.0,
+        "stream": stream,
+    }
+    if kv_xfer is not None:
+        payload["kv_transfer_params"] = kv_xfer
+    if stream:
+        payload["stream_options"] = {"include_usage": True}
+    url = f"http://127.0.0.1:{port}/v1/completions"
+    if not stream:
+        r = await client.post(url, json=payload, timeout=300.0)
+        r.raise_for_status()
+        return r.json()
+    last_w = None; last_any = None
+    async with client.stream("POST", url, json=payload, timeout=300.0) as resp:
+        resp.raise_for_status()
+        buf = ""
+        async for chunk in resp.aiter_bytes():
+            buf += chunk.decode("utf-8", errors="replace")
+            while "\n\n" in buf:
+                line, buf = buf.split("\n\n", 1)
+                if line.startswith("data: "):
+                    s = line[6:].strip()
+                    if s == "[DONE]": continue
+                    try:
+                        d = json.loads(s); last_any = d
+                        if d.get("usage"): last_w = d
+                    except Exception:
+                        pass
+    return last_w or last_any or {}
+
+
+async def get_engine_id(client, bp):
+    r = await client.get(f"http://127.0.0.1:{bp}/query")
+    return r.json()["0"]["engine_id"]
+
+
+def cached_of(d):
+    usage = d.get("usage") or {}
+    det = usage.get("prompt_tokens_details") or {}
+    return det.get("cached_tokens", 0) or usage.get("cached_tokens", 0)
+
+
+async def do_migration(client, src_port, dst_port, src_bp, prompt, max_tokens, label):
+    """Perform Mooncake-style PD-sep migration, returns (src_ms, dst_ms, response)."""
+    src_id = await get_engine_id(client, src_bp)
+    transfer_id = f"smoke-xfer-{uuid.uuid4().hex[:8]}"
+    t0 = time.monotonic()
+    src_resp = await send(client, src_port, prompt, max_tokens=1,
+                          kv_xfer={"do_remote_decode": True, "do_remote_prefill": False,
+                                   "transfer_id": transfer_id}, stream=False)
+    t1 = time.monotonic()
+    bootstrap_addr = f"http://127.0.0.1:{src_bp}"
+    dst_resp = await send(client, dst_port, prompt, max_tokens=max_tokens,
+                          kv_xfer={"do_remote_decode": False, "do_remote_prefill": True,
+                                   "remote_bootstrap_addr": bootstrap_addr,
+                                   "remote_engine_id": src_id,
+                                   "transfer_id": transfer_id}, stream=True)
+    t2 = time.monotonic()
+    src_ms = int((t1-t0)*1000); dst_ms = int((t2-t1)*1000)
+    cached = cached_of(dst_resp)
+    print(f"  [{label}] src_prefill={src_ms}ms  dst_total={dst_ms}ms  cached={cached}/{len(prompt)}")
+    return src_ms, dst_ms, dst_resp
+
+
+async def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--src-port", type=int, default=8000)
+    p.add_argument("--src-bp", type=int, default=8998)
+    p.add_argument("--dst-warm-port", type=int, default=8001)  # will be pre-warmed
+    p.add_argument("--dst-warm-bp", type=int, default=8999)
+    p.add_argument("--dst-cold-port", type=int, default=8002)  # cold control
+    p.add_argument("--dst-cold-bp", type=int, default=9000)
+    p.add_argument("--prefix-tokens", type=int, default=32768)
+    p.add_argument("--ext-tokens", type=int, default=512)
+    args = p.parse_args()
+
+    rng = _r.Random("partial-1")
+    prompt_base = [rng.randint(1024, 99_999) for _ in range(args.prefix_tokens)]
+    ext_tokens = [_r.Random("ext-partial").randint(1024, 99_999) for _ in range(args.ext_tokens)]
+    prompt_ext = prompt_base + ext_tokens
+
+    async with httpx.AsyncClient() as client:
+        print(f"\n=== Setup ===")
+        print(f"Prompt prefix: {args.prefix_tokens} tokens, extension: {args.ext_tokens} tokens")
+
+        # Step 0: warm dst_warm with prompt_base (normal request, no kv_transfer)
+        print(f"\n=== Step 0: warm dst_warm (port {args.dst_warm_port}) with prompt_base ===")
+        t0 = time.monotonic()
+        r = await send(client, args.dst_warm_port, prompt_base, max_tokens=1,
+                       kv_xfer=None, stream=False)
+        t1 = time.monotonic()
+        print(f"  cold prefill on dst_warm: {int((t1-t0)*1000)}ms, cached={cached_of(r)}/{args.prefix_tokens}")
+
+        # Sanity: 2nd request to dst_warm hits local cache
+        print(f"\n=== Sanity: 2nd request to dst_warm with same prompt — should hit local cache ===")
+        t0 = time.monotonic()
+        r = await send(client, args.dst_warm_port, prompt_base, max_tokens=1,
+                       kv_xfer=None, stream=False)
+        t1 = time.monotonic()
+        print(f"  warm request on dst_warm: {int((t1-t0)*1000)}ms, cached={cached_of(r)}/{args.prefix_tokens}")
+
+        # Step 1: cold prefill on src with prompt_ext (also caches at src)
+        print(f"\n=== Step 1: cold prefill on src (port {args.src_port}) with prompt_ext ===")
+        t0 = time.monotonic()
+        r = await send(client, args.src_port, prompt_ext, max_tokens=1,
+                       kv_xfer=None, stream=False)
+        t1 = time.monotonic()
+        print(f"  cold prefill on src: {int((t1-t0)*1000)}ms, cached={cached_of(r)}/{len(prompt_ext)}")
+
+        # Step 2: MIGRATE src -> dst_warm (which has cache for prompt_base)
+        print(f"\n=== Step 2: MIGRATE src -> dst_warm (cache-rich) with prompt_ext ===")
+        s_ms_warm, d_ms_warm, _ = await do_migration(
+            client, args.src_port, args.dst_warm_port, args.src_bp,
+            prompt_ext, max_tokens=4, label="cache-rich dst")
+
+        # Step 3: MIGRATE src -> dst_cold (no cache)
+        print(f"\n=== Step 3: MIGRATE src -> dst_cold (port {args.dst_cold_port}, cold) with prompt_ext ===")
+        s_ms_cold, d_ms_cold, _ = await do_migration(
+            client, args.src_port, args.dst_cold_port, args.src_bp,
+            prompt_ext, max_tokens=4, label="cold dst (control)")
+
+        # Verdict
+        print(f"\n=== VERDICT ===")
+        print(f"Cache-rich dst (Mechanism B):  dst_total={d_ms_warm}ms")
+        print(f"Cold dst (full transfer):      dst_total={d_ms_cold}ms")
+        speedup = (d_ms_cold - d_ms_warm) / d_ms_cold * 100 if d_ms_cold > 0 else 0
+        print(f"Δ = {d_ms_cold - d_ms_warm}ms ({speedup:+.1f}% faster with cache-rich dst)")
+        print(f"Partial transfer {'WORKING' if speedup > 30 else 'NOT working / not exploited'}.")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

microbench/connector_tax/cache_sweep/smoke_test_migrate_cache.py

@@ -0,0 +1,255 @@
+#!/usr/bin/env python3
+"""Smoke test: does remote-prefill on dst leave its prefix cache discoverable
+to a follow-up turn on the same instance?
+
+Reproducer for the v3 rotation bug observed in unified_v3 (next-turn at
+decode_target sees cached_tokens=0 despite migration's `cache_blocks`
+supposedly running).
+
+Expects 2 vLLM instances running on 127.0.0.1:8000 and 8001 with
+--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}'
+and Mooncake bootstrap servers on 8998 and 8999.
+
+Run flow:
+  1. Query both bootstrap servers for engine_ids.
+  2. Send migration: src=8000 do_remote_decode (max_tokens=1), then
+     dst=8001 do_remote_prefill (pulls KV via Mooncake) with same prompt.
+  3. Send follow-up: same session prompt + tiny extension, hit 8001
+     directly (no kv_transfer_params), check cached_tokens.
+
+A working migration with prefix-cache visibility would see ~100% cached
+on the follow-up (full prefix hit). The v3 bug shows cached=0.
+"""
+import asyncio
+import argparse
+import json
+import sys
+import uuid
+import httpx
+
+
+async def get_engine_id(client: httpx.AsyncClient, port: int) -> str:
+    url = f"http://127.0.0.1:{port}/query"
+    r = await client.get(url)
+    r.raise_for_status()
+    data = r.json()
+    # data = {"0": {"engine_id": "..."}, ...}
+    return data["0"]["engine_id"]
+
+
+async def send_completion(
+    client: httpx.AsyncClient,
+    host_port: int,
+    prompt: list[int],
+    max_tokens: int,
+    kv_transfer_params: dict | None = None,
+    stream: bool = False,
+) -> dict:
+    payload = {
+        "model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct",
+        "prompt": prompt,
+        "max_tokens": max_tokens,
+        "min_tokens": max_tokens if max_tokens == 1 else 1,
+        "temperature": 0.0,
+        "stream": stream,
+    }
+    if stream:
+        payload["stream_options"] = {"include_usage": True}
+    if kv_transfer_params:
+        payload["kv_transfer_params"] = kv_transfer_params
+    url = f"http://127.0.0.1:{host_port}/v1/completions"
+
+    if not stream:
+        r = await client.post(url, json=payload, timeout=300.0)
+        r.raise_for_status()
+        return r.json()
+    else:
+        # Stream and collect chunks; keep the LAST chunk that contains
+        # `usage` (with include_usage, the very last data: chunk has it).
+        last_with_usage = None
+        last_any = None
+        async with client.stream("POST", url, json=payload, timeout=300.0) as resp:
+            resp.raise_for_status()
+            buffer = ""
+            async for chunk in resp.aiter_bytes():
+                buffer += chunk.decode("utf-8", errors="replace")
+                while "\n\n" in buffer:
+                    line, buffer = buffer.split("\n\n", 1)
+                    if line.startswith("data: "):
+                        data_str = line[6:].strip()
+                        if data_str == "[DONE]":
+                            continue
+                        try:
+                            d = json.loads(data_str)
+                            last_any = d
+                            if d.get("usage"):
+                                last_with_usage = d
+                        except Exception:
+                            pass
+        return last_with_usage or last_any or {}
+
+
+def short(d: dict) -> str:
+    """Pull cached_tokens out of the response usage section."""
+    if not d:
+        return "no_resp"
+    usage = d.get("usage") or {}
+    details = usage.get("prompt_tokens_details") or {}
+    cached = details.get("cached_tokens", 0) or usage.get("cached_tokens", 0)
+    return (
+        f"cached={cached}/{usage.get('prompt_tokens', 0)} "
+        f"completion={usage.get('completion_tokens', 0)} "
+        f"id={d.get('id', '?')[:24]}"
+    )
+
+
+async def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--src-port", type=int, default=8000)
+    p.add_argument("--dst-port", type=int, default=8001)
+    p.add_argument("--src-bp", type=int, default=8998)
+    p.add_argument("--dst-bp", type=int, default=8999)
+    p.add_argument("--n-prefix-tokens", type=int, default=8192,
+                   help="Length of synthetic prompt prefix (tokens)")
+    p.add_argument("--n-extension", type=int, default=32,
+                   help="Tokens added in the follow-up request")
+    p.add_argument("--noise-reqs", type=int, default=0,
+                   help="Number of unrelated requests to send to dst between "
+                        "migration and follow-up (eviction-pressure test)")
+    p.add_argument("--noise-tokens", type=int, default=16384,
+                   help="Tokens per noise request")
+    p.add_argument("--noise-parallel", type=int, default=4,
+                   help="How many noise requests in parallel")
+    p.add_argument("--prefix-base", type=int, default=100,
+                   help="Token-id base for the prompt prefix (use distinct value "
+                        "across iterations to avoid prefix-cache pollution).")
+    p.add_argument("--decode-tokens", type=int, default=16,
+                   help="max_tokens on dst decode request")
+    args = p.parse_args()
+
+    async with httpx.AsyncClient() as client:
+        src_id = await get_engine_id(client, args.src_bp)
+        dst_id = await get_engine_id(client, args.dst_bp)
+        print(f"src engine_id = {src_id}")
+        print(f"dst engine_id = {dst_id}")
+        print()
+
+        # Build a deterministic prompt: a long enough token sequence to be
+        # multiple blocks. Use simple range to avoid tokenizer dependence.
+        # Build deterministic prompt using values in safe vocab range [1024, 100000)
+        # so high prefix-base values don't overflow the tokenizer vocab.
+        import random as _r
+        rng = _r.Random(f"prefix-{args.prefix_base}")
+        prompt = [rng.randint(1024, 99_999) for _ in range(args.n_prefix_tokens)]
+
+        # ----- Step 1: Migration. src does prefill (max_tokens=1, no stream),
+        # then dst pulls KV and decodes. -----
+        transfer_id = f"smoke-xfer-{uuid.uuid4().hex[:8]}"
+        print(f"[1] migration: transfer_id={transfer_id}")
+
+        # First: cold-prefill on src (no Mooncake yet, just establish src has the KV)
+        # The proxy convention: src sees do_remote_decode=True so it will SEND its KV
+        # via Mooncake later. We do this as a single-shot.
+        import time as _t
+        t_src_start = _t.monotonic()
+        src_resp_task = asyncio.create_task(
+            send_completion(
+                client, args.src_port, prompt, max_tokens=1,
+                kv_transfer_params={
+                    "do_remote_decode": True,
+                    "do_remote_prefill": False,
+                    "transfer_id": transfer_id,
+                },
+                stream=False,
+            )
+        )
+
+        # Slight stagger: dst needs to be ready to pull. In production the
+        # proxy waits for src to finish before sending dst. We'll do the
+        # same — await src then send dst.
+        src_resp = await src_resp_task
+        t_src_done = _t.monotonic()
+        print(f"    src prefill resp ({(t_src_done-t_src_start)*1000:.0f}ms): {short(src_resp)}")
+
+        bootstrap_addr = f"http://127.0.0.1:{args.src_bp}"
+        t_dst_start = _t.monotonic()
+        dst_resp = await send_completion(
+            client, args.dst_port, prompt, max_tokens=args.decode_tokens,
+            kv_transfer_params={
+                "do_remote_decode": False,
+                "do_remote_prefill": True,
+                "remote_bootstrap_addr": bootstrap_addr,
+                "remote_engine_id": src_id,
+                "transfer_id": transfer_id,
+            },
+            stream=True,
+        )
+        t_dst_done = _t.monotonic()
+        # dst time = KV transfer + decode of N tokens. Subtract approx decode time
+        # to isolate transfer cost.
+        usage_d = dst_resp.get("usage") or {}
+        n_completion = usage_d.get("completion_tokens", 0)
+        dst_total_ms = (t_dst_done - t_dst_start) * 1000
+        print(f"    dst decode resp ({dst_total_ms:.0f}ms, {n_completion} completion tokens): {short(dst_resp)}")
+        print(f"    [TIMING] proto={args.n_prefix_tokens}p src_prefill={int((t_src_done-t_src_start)*1000)}ms "
+              f"dst_total={int(dst_total_ms)}ms (KV xfer + {n_completion}-token decode)")
+        print()
+
+        # ----- Step 1.5: Noise. Send unrelated requests to dst to test eviction. -----
+        if args.noise_reqs > 0:
+            print(f"[1.5] sending {args.noise_reqs} noise requests "
+                  f"(tokens={args.noise_tokens}, parallel={args.noise_parallel}) to dst")
+            async def noise(idx):
+                rng_n = _r.Random(f"noise-{args.prefix_base}-{idx}")
+                p_n = [rng_n.randint(1024, 99_999) for _ in range(args.noise_tokens)]
+                return await send_completion(
+                    client, args.dst_port, p_n, max_tokens=1,
+                    kv_transfer_params=None, stream=False,
+                )
+            sem = asyncio.Semaphore(args.noise_parallel)
+            async def gated(idx):
+                async with sem:
+                    return await noise(idx)
+            results = await asyncio.gather(*[gated(i) for i in range(args.noise_reqs)])
+            done = sum(1 for r in results if r and r.get("usage"))
+            print(f"    noise: {done}/{args.noise_reqs} completed")
+            print()
+
+        # ----- Step 2: Follow-up. Same session, extended prompt, hit dst directly. -----
+        rng_ext = _r.Random(f"ext-{args.prefix_base}")
+        follow_prompt = prompt + [rng_ext.randint(1024, 99_999) for _ in range(args.n_extension)]
+        print(f"[2] follow-up direct to dst (no kv_transfer_params): "
+              f"prefix_len={args.n_prefix_tokens}, extended_len={len(follow_prompt)}")
+        fu = await send_completion(
+            client, args.dst_port, follow_prompt, max_tokens=4,
+            kv_transfer_params=None,
+            stream=False,
+        )
+        print(f"    follow-up resp: {short(fu)}")
+        print()
+
+        # ----- Step 3: Same prompt twice on dst (sanity) -----
+        print(f"[3] sanity: same prompt again to dst (should see local hit "
+              f"from step 2's just-cached blocks)")
+        sanity = await send_completion(
+            client, args.dst_port, follow_prompt, max_tokens=4,
+            kv_transfer_params=None,
+            stream=False,
+        )
+        print(f"    sanity resp: {short(sanity)}")
+        print()
+
+        # ----- Verdict -----
+        usage_fu = fu.get("usage") or {}
+        details_fu = usage_fu.get("prompt_tokens_details") or {}
+        cached_fu = details_fu.get("cached_tokens", 0) or usage_fu.get("cached_tokens", 0)
+        # Expect ~n_prefix_tokens (minus the last token + alignment)
+        expected_min = int(args.n_prefix_tokens * 0.95)
+        verdict = "PASS" if cached_fu >= expected_min else "FAIL"
+        print(f"=== verdict: {verdict} (follow-up cached={cached_fu}, "
+              f"expected >= {expected_min} of {args.n_prefix_tokens}) ===")
+        sys.exit(0 if verdict == "PASS" else 1)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

microbench/connector_tax/layerwise/DESIGN.md

@@ -0,0 +1,188 @@
+# Layer-wise KV transfer on Mooncake — exploration
+
+Goal: make vLLM's `MooncakeConnector` push KV **per-layer during prefill**
+(write mode) instead of the current **post-hoc full-request transfer**, then
+microbench correctness + whether it hides the transfer behind prefill compute
+(the thing MoRIIO's write mode does on AMD; no NVIDIA connector ships it).
+
+Everything here is isolated in worktree `worktree-mooncake-layerwise`. The
+dash0 venv connector is backed up at `mooncake_connector.py.ORIG_BACKUP`;
+revert = copy the backup back. Opt-in via env `MOONCAKE_LAYERWISE=1`, so with
+the env unset the connector behaves exactly as upstream.
+
+## Baseline flow (post-hoc, what we have)
+
+1. Proxy: prefill on src (`do_remote_decode`, max_tokens=1) → **await done** →
+   decode on dst (`do_remote_prefill`) which pulls.
+2. dst `start_load_kv`→`receive_kv` sends ZMQ `MooncakeXferMetadata` (its block
+   addrs) to src bootstrap.
+3. src `send_kv_to_decode`: waits `send_meta.ready` (set at `request_finished`,
+   i.e. **after full prefill**) → `_build_transfer_params` (all layers) →
+   `_send_blocks` (one big `batch_transfer_sync_write`) → FINISH response.
+
+Measured: this full transfer is on the critical path, runs at ~3 GB/s under
+load (vs ~10 GB/s idle), dominating migration TTFT.
+
+## Layer-wise flow (write mode, this exploration)
+
+Key idea: keep all RDMA + completion on the `sender_loop` thread (clean), but
+issue **one `batch_transfer_sync_write` per layer**, each fired as soon as that
+layer's KV is computed — so writes overlap the remaining prefill compute.
+
+Signaling: `save_kv_layer(layer_name, ...)` (called by vLLM's attention hook
+after each layer's forward, on the main worker thread) records "layer L
+computed" and wakes the sender_loop. `send_kv_to_decode` loops L=0..N-1,
+waits until L is computed, writes layer L's blocks, then sends FINISH.
+
+### Edits to `mooncake_connector.py` (all gated by `_lw_enabled`)
+
+1. **Worker `__init__`**: `_lw_enabled` (env), layer-name→position map,
+   `_lw_computed: dict[transfer_id,int]`, `_lw_active: set[transfer_id]`,
+   wake event, lock.
+2. **`register_kv_caches`**: build `_lw_layer_pos[layer_name]` (0..N-1) and
+   `_lw_addr_idx[pos]` = indices into `kv_caches_base_addr` (×2 if
+   `split_k_and_v`).
+3. **Scheduler `update_state_after_alloc`** (`do_remote_decode` branch): in
+   layer-wise mode capture `blocks.get_block_ids()[0]` and store non-empty in
+   `_reqs_need_send` so the worker learns local block_ids + sets `ready`
+   **before** prefill finishes.
+4. **Worker `note_layer_computed(layer_name)`** (new) called from
+   `MooncakeConnector.save_kv_layer`: bump `_lw_computed[tid]` for active
+   producers, `call_soon_threadsafe(wake.set)`.
+5. **Worker `send_kv_to_decode`**: in layer-wise mode, mark transfer active,
+   loop layers: await `_lw_computed[tid] >= L`, `_send_blocks` for layer L
+   only (subset of `_build_transfer_params`), then send FINISH.
+6. **Worker `_build_layer_transfer_params`** (new): like
+   `_build_transfer_params` but only the addr indices for one layer position.
+
+### Microbench requirements
+
+- Disable chunked prefill (`--max-num-batched-tokens` ≥ prompt) so prefill is a
+  single forward and `save_kv_layer` fires once per layer in order.
+- Dispatch the dst (`do_remote_prefill`) request **first/concurrently** so the
+  ZMQ handshake reaches src during prefill.
+- Correctness: dst follow-up `cached_tokens == prompt_len` (KV landed),
+  identical to baseline.
+- Perf: src prefill wall-clock (does layer-wise slow it?) and dst TTFT (does
+  transfer leave the critical path?), swept over KV size, vs baseline.
+
+## Status
+
+- [x] worktree + connector backup + design
+- [x] modified connector (LAYERWISE.py, +193/-4 lines, env-gated)
+- [x] correctness microbench (mb7_layerwise.py) + launcher (run_mb7.sh)
+- [x] correctness run on dash0 — PASS (KV lands; cached == prompt)
+- [x] perf run + verdict — POSITIVE (transfer hidden behind prefill)
+
+## Results (2-instance, idle, chunked-prefill off, Qwen3-30B-A3B, 48 layers)
+
+Metric: `overhead = total − prefill_only` = the transfer cost left on the
+critical path (TTFT). Baseline = post-hoc full pull (sequential).
+
+| KV size | baseline overhead | **layerwise overhead** | reduction |
+|--------:|------------------:|-----------------------:|----------:|
+| 8192 (0.75 GiB)  | 123 ms | **58 ms** | 2.1× |
+| 16384 (1.5 GiB)  | 202 ms | **58 ms** | 3.5× |
+| 32768 (3.0 GiB)  | 529 ms | **57 ms** | 9.3× |
+
+Key signatures:
+- **Layerwise overhead is ~constant (~58 ms)** regardless of KV size, while
+  baseline grows O(KV size). The 58 ms is handshake + last-layer tail + 1
+  decode; the bulk transfer is hidden behind prefill compute.
+- **Prefill did NOT slow down**: layerwise `t_A` (575/1495/4440 ms) ==
+  `prefill_only` (574/1492/4440 ms). The concurrent RDMA was "free" on idle
+  GPUs — no measurable HBM contention with prefill compute here.
+- Producer logs confirm the transfer itself took 0.39/0.55/4.37 s (grows with
+  size) yet ran *inside* the prefill window, so it left the critical path.
+- **Correctness PASS**: B's follow-up cached == prompt for all sizes; the
+  48-layer / 96-base-addr (split K&V) per-layer addressing is correct.
+
+## Caveats (why this is a proof-of-concept, not a verdict for production)
+
+1. **Idle instances only.** Real migration happens between *busy* instances.
+   Under load both prefill and transfer slow; transfer (even at ~3 GB/s) is
+   still < prefill for big contexts so it should still hide, but receive-side
+   (B) and HBM contention during prefill are untested here. NEXT: rerun with
+   background load on both A and B.
+2. **Chunked prefill disabled.** The monotonic layer counter assumes one
+   forward, layers in order. Production uses chunked prefill (multi-step),
+   which needs per-(chunk,layer) tracking — not implemented.
+3. **Single concurrent producer transfer.** Global counter; real migration is
+   concurrent. Would need per-transfer state.
+4. **Microbench dispatch.** mb7 fires B then A with a 50 ms head start to get
+   the handshake to A before its forward. The real proxy path
+   (`_handle_combined_pd_sep_v2`) dispatches sequentially and would need the
+   write-mode (concurrent) restructure.
+
+## Results under LOAD (bg=16 background decode streams, 8 per instance)
+
+Critical-path transfer overhead (ms), `total − prefill_only`:
+
+| KV size | idle base | idle LW | **load base** | **load LW** |
+|--------:|----------:|--------:|--------------:|------------:|
+| 8k  | 123 | 58 | 158 | **94** |
+| 16k | 202 | 58 | 239 | **83** |
+| 32k | 529 | 57 | **749** | **95** |
+
+The overlap **survives load**: layerwise overhead stays ~constant (~90 ms)
+under load while baseline grows to 749 ms at 32k (7.9× reduction). Prefill did
+not slow (load LW `t_A` == load `prefill_only`); the transfer (0.56/1.46/4.37 s,
+producer logs) ran inside the prefill window even with 16 concurrent decodes.
+Correctness PASS under load.
+
+## FULL 1200-req v3 TRACE re-profile (chunk-safe + concurrent + write-mode)
+
+Hardened connector (per-step incremental shipping, per-transfer state) +
+write-mode proxy (concurrent prefill/decode dispatch). Two passes of
+`w600_r0.0015_st30.jsonl` under `unified_v3`, differing only in transfer mode.
+
+Correctness: layer-wise **1213/1214 success** (1 connection-error on the 128k
+req, not KV corruption); byte-level KV correctness validated on mb7
+(chunked + 3-way concurrent, `cached==prompt`); producer logs confirm
+incremental shipping (e.g. `shipped 7872/7872 blocks`).
+
+Migration sets differ between runs (write-mode timing shifts which requests
+trigger migration; only 4 migrated in both), but are distributionally
+comparable (median new_local/input ≈ 0.42 vs 0.46). **Matched migrations
+all improved**, scaling with the transfer hidden behind prefill:
+
+| request | input | new_local | base TTFT | LW TTFT | Δ |
+|---|--:|--:|--:|--:|--:|
+| 1268630 | 102k | 97k | 41.20 | 33.96 | **−7.23s** |
+| 1334223 | 37k | 14k | 6.04 | 3.23 | −2.81s |
+| 1279412 | 40k | 8k | 5.50 | 2.92 | −2.58s |
+| 1271459 | 8.9k | 8.9k | 37.01 | 36.98 | −0.03s (queue-bound) |
+
+Trace-level TTFT (different sets, directional): overall p90 9.79→9.16 (−6%),
+p99 44.89→42.85 (−5%). **Modest** because (a) migrations are only 25/1214 ≈
+**2%** of requests, and (b) several migrations are queue/contention-bound, not
+transfer-bound — layer-wise removes the transfer component but not the
+control-plane/queue residual (the ~45% from the b3_v3_fullbreak profile).
+
+**Verdict on the trace re-profile:** layer-wise does exactly what the profile
+predicted — it removes the transfer half of migration overhead (matched
+migrations −2.6 to −7.2s, biggest where there's the most prefill to hide
+behind), but the trace-level gain is small because migrations are rare and
+partly queue-bound. It does NOT, on its own, flip migration to a clear win
+over unified for this agentic workload.
+
+## Verdict (microbench)
+
+The mechanism **works and the benefit holds under load**: layer-wise push turns
+migration's KV-transfer cost from O(KV size) on the critical path into a
+near-constant ~90 ms tail, by overlapping it with prefill compute — what
+MoRIIO's write mode does on AMD, now demonstrated on NVIDIA/Mooncake.
+
+**BUT this is single-transfer, non-chunked.** Running the actual 1200-req trace
+correctly needs two more pieces this PoC does NOT have:
+1. **Chunk-safe tracking** — long agentic prompts force chunked prefill;
+   `save_kv_layer` then fires per-chunk and the monotonic counter would ship
+   uncomputed blocks. Needs slot-mapping-aware per-(request,chunk) tracking.
+2. **Concurrent-transfer safety** — the global counter assumes one producer at
+   a time; the trace migrates from busy instances running other forwards.
+
+Also: even with those fixed, layer-wise only removes the **transfer half** of
+the measured migration overhead. The b3_v3_fullbreak profile showed dst-side
+`T_kv_pull` = ~55% RDMA + ~45% control-plane GIL-dispatch stalls; layer-wise
+hides the RDMA half but the control-plane half is orthogonal. So a trace
+re-profile would show roughly the transfer half collapse, not the whole thing.

microbench/connector_tax/layerwise/ES_ABLATION_RESULTS.md

@@ -0,0 +1,77 @@
+# Engine-state ablation: real-time state vs router shadow counters
+
+**Question.** The router (`cache_aware_proxy`) routes on **shadow counters** it
+maintains itself (incremented at dispatch, reconciled to vLLM `/metrics` only
+every 30 s → stale). Does feeding it **real** per-engine state (running/waiting,
+KV-used, pending-prefill, `max_prefill_remaining`) change routing decisions,
+performance, or the policy ranking?
+
+**Setup.** dash1, 8×H20 (TP=1), Qwen3-Coder-30B-A3B, trace
+`w600_r0.0015_st30.jsonl` (1214 reqs / 274 sessions). Each policy run as a
+matched pair: `es0` (shadow only) vs `es1` (real-state feed via
+`file:///dev/shm/...`, published ~50 ms by a scheduler daemon thread, read by
+the proxy via `eff_* = max(shadow, real)`). Only the state source differs.
+Driver: `run_full_ablation.sh`; per-cell freshness via `fresh_sampler.py`;
+comparison via `cmp_es.py`.
+
+## Result — real-time state is NOT the routing lever
+
+It reshuffles 44–76% of routing decisions but **never beats the champion**;
+the cache-affinity champion (`unified+A+B`, es0 p90 **7.62 s**) stays best.
+
+| Policy | how it uses load | inst/session | reroute % | TTFT p90 es0→es1 | mean es0→es1 | verdict |
+|---|---|--:|--:|--:|--:|---|
+| `sticky` | **once at session birth, then pinned** | 1.00 | 44.5% | 13.42 → **9.95 (−26%)** | 4.13→3.65 | **HELPS** |
+| `unified+A+B` | per-req, affinity-dominated | 1.22 | 76.4% | 7.62 → 7.76 (+1.8%) | 3.20→3.24 | wash |
+| `v3_AB_lw` | per-req, affinity-dom + migration | ~1.2 | 71.7% | 9.35 → 9.49 (+1.5%) | 3.34→3.58 | wash* |
+| `unified_kv_both` | per-req, affinity-dom (same picker) | ~1.2 | 73.6% | 6.45 → 9.28 (+44%) | 3.07→3.49 | worse† |
+| `lmetric` | per-req, load×batch | 2.04 | 73.4% | 15.63 → 18.23 (+16.6%) | 5.18→5.80 | HURTS |
+| `load_only` | per-req, pure load | 2.22 | 72.7% | 21.79 → 27.69 (+27%) | 6.65→8.42 | HURTS |
+
+\* v3 real-state migration targeting backfired: migrations 26→32, migrated-req
+mean TTFT 11.99→18.45 s (+54%). Real state does not rescue migration.
+† same picker as `unified`; the 1.8%-vs-44% spread is run-variance (single
+pairs) in which reshuffled routes hit hotspots — sign is consistently ≥ neutral.
+
+## Mechanism — the sign is set by reactivity, not "affinity vs not"
+
+- **One-shot placement (`sticky`) → HELPS.** `pick_instance_sticky` is *not* a
+  stateless hash: the first turn picks `min(eff_num_requests())` (load), then
+  `affinity[session]` pins it for all later turns. State enters at exactly one
+  decision per session; real load → better placement that compounds across the
+  session, locality preserved, no per-request oscillation.
+- **Per-request, affinity-dominated (`unified`/`v3`/`kv_both`) → wash-to-worse.**
+  The hybrid picker mostly obeys affinity; only the ~12% fallback fraction
+  consults load. Net 0…+44%, never helps.
+- **Per-request, pure load (`lmetric`/`load_only`) → HURTS, monotonic in
+  load-purity.** Routing on *instantaneous* load induces **herding** (everyone
+  piles onto whatever momentarily looks idle → transient overload → tail
+  inflation); the stale shadow counter was inadvertently a **dampener**.
+
+## Why the result is trustworthy (not a stale-feed artifact)
+The feed was fresh on every es1 cell: age median **25 ms**, **≤92 ms even
+during 100k-token prefills**, <0.5 % of samples >2 s stale (and those not
+during prefills → reader drops them → shadow fallback). The feared GIL-
+starvation of the publisher during big prefills did **not** materialize.
+
+## Implications
+1. Don't invest in real-time state for per-request routing — it never wins and
+   degrades load-driven policies up to +27 %.
+2. The cache-affinity champion is robust to state source; A+B+RaceFix already
+   handled the staleness that mattered.
+3. **Design insight:** the only place ground-truth state helps is **one-shot
+   session placement** (decide well once on real load, then commit) — not
+   per-request load polling.
+4. All prior shadow-state results stand; the router's approximate state was
+   never the bottleneck. Workload skew + affinity discipline are.
+
+## Reproduce
+```bash
+# per-cell: same proxy, ES=0 (shadow) vs ES=1 (real); see run_v3_trace.sh
+MODE=baseline POLICY=unified AB_FLAGS="--overload-factor 1.3 --lmetric-decode-weight 0.01" \
+  ES=1 TAG=unified_AB_es1 bash run_v3_trace.sh
+# full sweep (waits for the champion es1 marker, then runs the rest):
+bash run_full_ablation.sh
+# compare a pair:
+python cmp_es.py <es0_dir>/unified_v3 <es1_dir>/unified_v3 abl_<tag>_es1.freshness.jsonl
+```

microbench/connector_tax/layerwise/P2_ENGINE_STATE.md

@@ -0,0 +1,82 @@
+# P2: real engine-state feed for migration target selection
+
+Problem: the router (`cache_aware_proxy.py`) decides migration targets from
+**shadow counters** it maintains itself (incremented at dispatch, decremented
+at completion) and reconciles to vLLM `/metrics` only every **30 s**
+(`_reconcile_loop`). So every routing/migration decision is on stale state.
+Worse, the signal that predicts the ~45% control-plane stall — *is the target
+mid-large-prefill?* (a big prefill holds the GIL and starves the mooncake
+receiver_loop) — isn't visible at all, and `/metrics` doesn't expose it either.
+
+Fix: vLLM publishes **real** per-engine state to a shared store ~20 Hz; the
+router reads ground truth and avoids GIL-stall / capacity-wall targets.
+
+## Components (all unit-tested without GPUs)
+
+- `engine_state.py` — canonical `compute_snapshot(scheduler, id)`, `StateWriter`,
+  `StateReader`. Schema per engine: `ts, num_running, num_waiting,
+  gpu_blocks_total/free, gpu_kv_used_frac, pending_prefill_tokens,
+  ongoing_decode_tokens, num_prefilling, max_prefill_remaining`.
+- `instrument_engine_state.py` — vLLM `Scheduler` patch (apply/revert markers
+  `ES_INSTRUMENT_*`): a daemon thread publishes the snapshot every
+  `AGENTIC_ENGINE_STATE_PERIOD_MS` (50 ms) off the forward hot path. Inlined
+  writer (engine process needs no repo import). Coexists with MB5.
+- `migration_target.py` — pure target scorer: avoid `max_prefill_remaining ≥
+  es_big_prefill_threshold` (GIL stall) and `gpu_kv_used_frac ≥ es_kv_wall_frac`
+  (capacity wall), then rank by cache-richness and **real** load.
+- `cache_aware_proxy.WRITEMODE.py` — wired: `InstanceState.real_state`,
+  `_engine_state_poll_loop` (instance i ← `engine_{i}`), `_real_load`/Gate-3 and
+  Mechanism-B now real-state-aware. `--engine-state-uri` flag; off ⇒ identical
+  to before (shadow only).
+
+Transport (`AGENTIC_ENGINE_STATE_URI` / `--engine-state-uri`):
+`file:///dev/shm/agentic_engine_state` (default, zero-dep, single-node) or
+`redis://host:port/0` (multi-node; needs redis-py + server — not installed on
+dash0, so file backend is the working default).
+
+## Tests (no GPU)
+- `compute_snapshot` field math (mock scheduler): running/waiting,
+  max_prefill_remaining, pending, decode, kv_used_frac.
+- writer→reader round-trip + staleness drop (file backend).
+- target scorer: 5 cases incl. *avoid GIL-stall target even when its shadow
+  load is lower*, *real load beats stale shadow*, *cache-rich wins*,
+  *avoid KV wall*, *graceful fallback when feed missing*.
+- end-to-end: publish 8 engines (one mid-130k-prefill) → proxy inlined reader →
+  target selection avoids it.
+
+## Enabling in a GPU run (when free)
+1. `instrument_engine_state.py --apply` on the dash0 venv.
+2. `export AGENTIC_ENGINE_STATE_URI=file:///dev/shm/agentic_engine_state`
+   before the launcher (vLLM instances inherit it; `AGENTIC_WORKER_ID=engine_{i}`
+   already set by `b3_isolated_policy.sh` → publishes as `engine_{i}`).
+3. Proxy: `EXTRA_PROXY_ARGS="--engine-state-uri file:///dev/shm/agentic_engine_state ..."`.
+4. Revert the patch + `rm -rf /dev/shm/agentic_engine_state` after.
+
+## ALL policies now read the real state (update)
+`InstanceState` exposes effective accessors used by **every** picker:
+`eff_num_requests / eff_pending_prefill / eff_ongoing_decode /
+eff_ongoing_tokens` = `max(shadow, real)` when the feed is fresh (real fixes
+the 30s-stale under-count; shadow's atomic pre-await reservation still covers
+the in-flight window, preserving the RaceFix), plus real-only
+`r_max_prefill_remaining / r_kv_used_frac`. Wired into: `load_only`, `lmetric`,
+`sticky`, `pick_instance` (legacy), `pick_instance_unified_hybrid`
+(unified / unified_kv_both), `pick_instance_unified_v3` (gate + Mechanism B),
+and `snapshot_workers` (logged scores now match the decision + real fields).
+Feed off ⇒ `real_state is None` ⇒ accessors return shadow ⇒ byte-identical to
+before. (legacy `unified_v2` left on shadow — retired, not in the ablation.)
+
+## Ablation (when GPU free)
+`run_v3_trace.sh` gains `ES=1` (apply engine-state patch + feed + proxy flag)
+and always deploys the enhanced proxy (dormant when feed/write-mode off).
+`run_ablation_es.sh` runs each config twice (ES=0 vs ES=1) so the only
+difference is the state source. Default decisive set (4 runs): champion
+`unified+A+B` and `unified_v3+A+B+layerwise`, each ES0/ES1. Extend CONFIGS for
+`lmetric` / `unified_kv_both` / `load_only`. Compares per-policy TTFT
+(overall + migrated) and whether the **ranking** changes with ground-truth
+state.
+
+## Status / scope
+- Built + unit-tested (snapshot, round-trip, target scorer, eff_ accessors,
+  end-to-end publish→read→select); NOT yet run against live engines (GPU busy).
+- TP=1 only (one EngineCore/instance → one publisher/engine_id). TP>1 needs
+  per-rank ids.

microbench/connector_tax/layerwise/cmp_es.py

@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+"""Compare an es0 (shadow) vs es1 (real-state) run: TTFT, decision split,
+routing flips, load distribution. Optional 3rd arg = es1 freshness jsonl."""
+import json, sys, statistics, os
+
+def load(d):
+    ms = {}
+    for l in open(os.path.join(d, "metrics.jsonl")):
+        m = json.loads(l); ms[m["request_id"]] = m
+    bd = {x["request_id"]: x for x in json.load(open(os.path.join(d, "breakdown.json")))}
+    return ms, bd
+
+def pct(xs, q):
+    xs = sorted(xs)
+    return xs[min(len(xs) - 1, int(q * len(xs)))] if xs else 0
+
+def chosen(x):
+    return x.get("routed_to", x.get("chosen_idx"))
+
+d0, d1 = sys.argv[1], sys.argv[2]
+m0, b0 = load(d0); m1, b1 = load(d1)
+
+def ttfts(ms):
+    return [m["ttft_s"] for m in ms.values() if not m.get("error") and m.get("ttft_s") is not None]
+
+print("=== overall TTFT ===")
+for tag, ms in [("es0/shadow", m0), ("es1/real ", m1)]:
+    t = ttfts(ms)
+    print(f"{tag}: {len(t)}/{len(ms)} ok  p50={pct(t,.5):.2f} p90={pct(t,.9):.2f} "
+          f"p99={pct(t,.99):.2f} max={max(t):.2f} mean={statistics.mean(t):.2f}")
+
+def byclass(ms, bd):
+    cls = {}
+    for rid, m in ms.items():
+        if m.get("error") or m.get("ttft_s") is None: continue
+        dec = bd.get(rid, {}).get("decision", "?")
+        cls.setdefault(dec, []).append(m["ttft_s"])
+    return cls
+
+print("\n=== decision split (n / p90 / p99) ===")
+for tag, ms, bd in [("es0", m0, b0), ("es1", m1, b1)]:
+    print(f"  [{tag}]")
+    for dec, ts in sorted(byclass(ms, bd).items()):
+        print(f"    {dec:18s} n={len(ts):4d} p90={pct(ts,.9):7.2f} p99={pct(ts,.99):7.2f}")
+
+common = set(b0) & set(b1)
+flips = [r for r in common if chosen(b0[r]) != chosen(b1[r])]
+decflip = [r for r in common if b0[r].get("decision") != b1[r].get("decision")]
+print(f"\n=== routing changes (common reqs={len(common)}) ===")
+print(f"  instance flips    : {len(flips)} ({100*len(flips)/max(1,len(common)):.1f}%)")
+print(f"  decision-type flips: {len(decflip)} ({100*len(decflip)/max(1,len(common)):.1f}%)")
+
+# TTFT of flipped vs non-flipped (es1 side)
+fl_t = [m1[r]["ttft_s"] for r in flips if not m1[r].get("error") and m1[r].get("ttft_s") is not None]
+if fl_t:
+    print(f"  flipped reqs es1 TTFT: p50={pct(fl_t,.5):.2f} p90={pct(fl_t,.9):.2f} mean={statistics.mean(fl_t):.2f}")
+
+def dist(bd):
+    d = {}
+    for x in bd.values():
+        d[chosen(x)] = d.get(chosen(x), 0) + 1
+    return dict(sorted(d.items(), key=lambda kv: str(kv[0])))
+print("\n=== per-instance request count ===")
+print("  es0:", dist(b0))
+print("  es1:", dist(b1))
+
+if len(sys.argv) > 3 and os.path.exists(sys.argv[3]):
+    rows = [json.loads(l) for l in open(sys.argv[3]) if l.strip()]
+    ages = [r["age_s"] for r in rows]
+    busy = [r["age_s"] for r in rows if (r.get("num_prefilling") or 0) > 0]
+    print(f"\n=== es1 feed freshness (full run, n={len(ages)}) ===")
+    if ages:
+        print(f"  age_s med={statistics.median(ages):.3f} p90={pct(ages,.9):.3f} max={max(ages):.3f} "
+              f"stale>2s={sum(1 for a in ages if a>2)}")
+    if busy:
+        print(f"  during-prefill n={len(busy)} med={statistics.median(busy):.3f} max={max(busy):.3f}")

microbench/connector_tax/layerwise/engine_state.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+"""Engine-state store: canonical snapshot + writer/reader, shared schema.
+
+The vLLM scheduler patch (instrument_engine_state.py) inlines a faithful copy
+of `compute_snapshot` + the file/redis writer (engine process needs no repo
+import). The router (cache_aware_proxy) imports `StateReader` here to read the
+real per-engine state instead of its stale shadow counters.
+
+Schema (one record per engine, key = engine_id):
+  ts, engine_id, num_running, num_waiting, gpu_blocks_total, gpu_blocks_free,
+  gpu_kv_used_frac, pending_prefill_tokens, ongoing_decode_tokens,
+  num_prefilling, max_prefill_remaining
+
+Transport URIs:
+  file:///dev/shm/agentic_engine_state   (default; atomic temp+rename)
+  redis://host:port/0                      (optional; needs redis-py)
+"""
+from __future__ import annotations
+
+import json
+import os
+import time
+
+
+def compute_snapshot(scheduler, engine_id: str) -> dict:
+    """Cheap O(batch) read of routing-relevant real state from a live
+    vLLM V1 Scheduler (duck-typed for testability)."""
+    try:
+        pool = scheduler.kv_cache_manager.block_pool
+        total = int(pool.num_gpu_blocks)
+        free = int(pool.get_num_free_blocks())
+    except Exception:
+        total = free = -1
+    n_run = pend = dec = n_pref = max_pref = 0
+    try:
+        for r in scheduler.running:
+            n_run += 1
+            npr = int(getattr(r, "num_prompt_tokens", 0))
+            nct = int(getattr(r, "num_computed_tokens", 0))
+            if nct < npr:
+                rem = npr - nct
+                pend += rem
+                n_pref += 1
+                max_pref = max(max_pref, rem)
+            else:
+                dec += int(getattr(r, "num_tokens", 0))
+    except Exception:
+        pass
+    n_wait = 0
+    try:
+        n_wait = len(scheduler.waiting) + len(getattr(scheduler, "skipped_waiting", []))
+        for r in list(scheduler.waiting):
+            pend += max(0, int(getattr(r, "num_prompt_tokens", 0))
+                        - int(getattr(r, "num_computed_tokens", 0)))
+    except Exception:
+        pass
+    used = ((total - free) / total) if (total and total > 0) else -1.0
+    return {
+        "ts": time.time(),
+        "engine_id": engine_id,
+        "num_running": n_run,
+        "num_waiting": int(n_wait),
+        "gpu_blocks_total": total,
+        "gpu_blocks_free": free,
+        "gpu_kv_used_frac": used,
+        "pending_prefill_tokens": int(pend),
+        "ongoing_decode_tokens": int(dec),
+        "num_prefilling": n_pref,
+        "max_prefill_remaining": int(max_pref),
+    }
+
+
+class StateWriter:
+    def __init__(self, uri: str, engine_id: str):
+        self.engine_id = engine_id
+        self.kind = None
+        if uri.startswith("file://"):
+            self.kind = "file"
+            self.dir = uri[len("file://"):]
+            os.makedirs(self.dir, exist_ok=True)
+            self.path = os.path.join(self.dir, f"{engine_id}.json")
+            self.tmp = self.path + f".tmp.{os.getpid()}"
+        elif uri.startswith("redis://"):
+            self.kind = "redis"
+            import redis
+            self.r = redis.Redis.from_url(uri)
+            self.key = f"engine_state:{engine_id}"
+        else:
+            raise ValueError(f"unsupported engine-state URI: {uri}")
+
+    def publish(self, state: dict):
+        if self.kind == "file":
+            with open(self.tmp, "w") as f:
+                f.write(json.dumps(state))
+            os.replace(self.tmp, self.path)
+        elif self.kind == "redis":
+            self.r.set(self.key, json.dumps(state), ex=5)
+
+
+class StateReader:
+    """Router-side reader. read_all() returns {engine_id: state}, dropping
+    records older than max_age_s (so a dead/hung engine is ignored)."""
+    def __init__(self, uri: str, max_age_s: float = 2.0):
+        self.uri = uri
+        self.max_age_s = max_age_s
+        self.kind = None
+        if uri.startswith("file://"):
+            self.kind = "file"
+            self.dir = uri[len("file://"):]
+        elif uri.startswith("redis://"):
+            self.kind = "redis"
+            import redis
+            self.r = redis.Redis.from_url(uri)
+        else:
+            raise ValueError(f"unsupported engine-state URI: {uri}")
+
+    def read_all(self) -> dict[str, dict]:
+        now = time.time()
+        out: dict[str, dict] = {}
+        try:
+            if self.kind == "file":
+                import glob
+                for p in glob.glob(os.path.join(self.dir, "*.json")):
+                    try:
+                        s = json.load(open(p))
+                    except Exception:
+                        continue
+                    if now - s.get("ts", 0) <= self.max_age_s:
+                        out[s.get("engine_id", os.path.basename(p)[:-5])] = s
+            elif self.kind == "redis":
+                for k in self.r.scan_iter("engine_state:*"):
+                    v = self.r.get(k)
+                    if not v:
+                        continue
+                    s = json.loads(v)
+                    if now - s.get("ts", 0) <= self.max_age_s:
+                        out[s.get("engine_id")] = s
+        except Exception:
+            pass
+        return out

microbench/connector_tax/layerwise/fresh_sampler.py

@@ -0,0 +1,30 @@
+#!/usr/bin/env python3
+"""Sample engine-state feed freshness during the es1 run.
+Writes one jsonl record per engine per tick: age_s = now - state.ts.
+Stops when the DONE marker appears (run finished + /dev/shm wiped) or after 90min.
+"""
+import json, os, time, sys, glob
+
+esdir, outpath, donemarker = sys.argv[1], sys.argv[2], sys.argv[3]
+deadline = time.time() + 90 * 60
+with open(outpath, "a") as out:
+    while not os.path.exists(donemarker) and time.time() < deadline:
+        now = time.time()
+        if os.path.isdir(esdir):
+            for f in sorted(glob.glob(os.path.join(esdir, "engine_*.json"))):
+                try:
+                    s = json.load(open(f))
+                    rec = {
+                        "now": round(now, 3),
+                        "engine": os.path.basename(f)[:-5],
+                        "age_s": round(now - s.get("ts", 0), 4),
+                        "num_running": s.get("num_running"),
+                        "num_prefilling": s.get("num_prefilling"),
+                        "max_prefill_remaining": s.get("max_prefill_remaining"),
+                        "kv_used": s.get("gpu_kv_used_frac"),
+                    }
+                    out.write(json.dumps(rec) + "\n")
+                except Exception:
+                    pass
+            out.flush()
+        time.sleep(5)

microbench/connector_tax/layerwise/instrument_engine_state.py

@@ -0,0 +1,234 @@
+#!/usr/bin/env python3
+"""Patch vLLM V1 scheduler to publish REAL engine state to a shared store,
+so the global router reads ground truth instead of its own stale shadow
+counters (reconciled only every 30s).
+
+Published per engine (key = AGENTIC_ENGINE_ID), throttled ~20 Hz from a
+daemon thread (off the forward hot path):
+
+  {ts, num_running, num_waiting, gpu_blocks_total, gpu_blocks_free,
+   gpu_kv_used_frac, pending_prefill_tokens, ongoing_decode_tokens,
+   num_prefilling, max_prefill_remaining}
+
+`max_prefill_remaining` is the key signal /metrics does NOT expose: the
+largest in-progress prefill on the engine. A big in-progress prefill holds
+the GIL and stalls the mooncake receiver_loop — so the router should avoid
+migrating KV to such an instance (P2).
+
+Transport (env AGENTIC_ENGINE_STATE_URI):
+  file:///dev/shm/agentic_engine_state   (default; atomic temp+rename)
+  redis://host:port/0                      (optional; needs redis-py + server)
+
+Self-contained (inlined writer) so the engine process needs no repo import.
+Apply/revert markers: # ES_INSTRUMENT_START / # ES_INSTRUMENT_END.
+
+Usage:
+  python instrument_engine_state.py --apply  [--venv PATH]
+  python instrument_engine_state.py --revert [--venv PATH]
+  python instrument_engine_state.py --check  [--venv PATH]
+"""
+from __future__ import annotations
+
+import argparse
+import re
+from pathlib import Path
+
+DEFAULT_VENV = Path("/home/admin/cpfs/wjh/agentic-kv/.venv")
+TARGET_REL = "lib/python3.12/site-packages/vllm/v1/core/sched/scheduler.py"
+START = "# ES_INSTRUMENT_START"
+END = "# ES_INSTRUMENT_END"
+
+# ---- Patch 1: header (writer + publisher thread), before class Scheduler ----
+HEADER_ANCHOR = "class Scheduler(SchedulerInterface):"
+HEADER = f'''{START}
+import json as _es_json
+import os as _es_os
+import threading as _es_threading
+import time as _es_time
+
+_ES_URI = _es_os.environ.get("AGENTIC_ENGINE_STATE_URI", "")
+_ES_ID = _es_os.environ.get("AGENTIC_ENGINE_ID") or _es_os.environ.get(
+    "AGENTIC_WORKER_ID", f"engine_{{_es_os.getpid()}}")
+_ES_PERIOD_S = float(_es_os.environ.get("AGENTIC_ENGINE_STATE_PERIOD_MS", "50")) / 1000.0
+
+
+class _ESWriter:
+    """Pluggable state writer: file:// (atomic temp+rename) or redis://."""
+    def __init__(self, uri: str, engine_id: str):
+        self.engine_id = engine_id
+        self.kind = None
+        if uri.startswith("file://"):
+            self.kind = "file"
+            self.dir = uri[len("file://"):]
+            _es_os.makedirs(self.dir, exist_ok=True)
+            self.path = _es_os.path.join(self.dir, f"{{engine_id}}.json")
+            self.tmp = self.path + f".tmp.{{_es_os.getpid()}}"
+        elif uri.startswith("redis://"):
+            self.kind = "redis"
+            import redis  # lazy
+            self.r = redis.Redis.from_url(uri)
+            self.key = f"engine_state:{{engine_id}}"
+
+    def publish(self, state: dict):
+        try:
+            if self.kind == "file":
+                with open(self.tmp, "w") as f:
+                    f.write(_es_json.dumps(state))
+                _es_os.replace(self.tmp, self.path)  # atomic
+            elif self.kind == "redis":
+                self.r.set(self.key, _es_json.dumps(state), ex=5)
+        except Exception:
+            pass
+
+
+def _es_compute_snapshot(scheduler) -> dict:
+    """Cheap O(batch) state read from the live scheduler."""
+    try:
+        kvm = scheduler.kv_cache_manager
+        pool = kvm.block_pool
+        total = int(pool.num_gpu_blocks)
+        free = int(pool.get_num_free_blocks())
+    except Exception:
+        total = free = -1
+    n_run = 0
+    pend = 0
+    dec = 0
+    n_pref = 0
+    max_pref = 0
+    try:
+        for r in scheduler.running:
+            n_run += 1
+            npr = int(getattr(r, "num_prompt_tokens", 0))
+            nct = int(getattr(r, "num_computed_tokens", 0))
+            if nct < npr:  # still prefilling
+                rem = npr - nct
+                pend += rem
+                n_pref += 1
+                if rem > max_pref:
+                    max_pref = rem
+            else:  # decoding
+                dec += int(getattr(r, "num_tokens", 0))
+    except Exception:
+        pass
+    n_wait = 0
+    try:
+        n_wait = len(scheduler.waiting) + len(getattr(scheduler, "skipped_waiting", []))
+        for r in list(scheduler.waiting):
+            pend += max(0, int(getattr(r, "num_prompt_tokens", 0))
+                        - int(getattr(r, "num_computed_tokens", 0)))
+    except Exception:
+        pass
+    used_frac = ((total - free) / total) if (total and total > 0) else -1.0
+    return {{
+        "ts": _es_time.time(),
+        "engine_id": _ES_ID,
+        "num_running": n_run,
+        "num_waiting": int(n_wait),
+        "gpu_blocks_total": total,
+        "gpu_blocks_free": free,
+        "gpu_kv_used_frac": used_frac,
+        "pending_prefill_tokens": int(pend),
+        "ongoing_decode_tokens": int(dec),
+        "num_prefilling": n_pref,
+        "max_prefill_remaining": int(max_pref),
+    }}
+
+
+class _ESPublisher:
+    def __init__(self, scheduler):
+        self._sched = scheduler
+        self._writer = _ESWriter(_ES_URI, _ES_ID)
+        self._stop = _es_threading.Event()
+        self._t = _es_threading.Thread(target=self._loop, daemon=True)
+        self._t.start()
+
+    def _loop(self):
+        while not self._stop.is_set():
+            try:
+                self._writer.publish(_es_compute_snapshot(self._sched))
+            except Exception:
+                pass
+            _es_time.sleep(_ES_PERIOD_S)
+{END}
+
+
+'''
+
+# ---- Patch 2: start the publisher at the end of Scheduler.__init__ ----------
+# Anchor on the existing agentic step-log block tail in __init__.
+INIT_ANCHOR = """        _step_path = _os.environ.get("AGENTIC_STEP_LOG_PATH")"""
+INIT_INSERT = f"""        {START}
+        if _ES_URI:
+            try:
+                self._es_publisher = _ESPublisher(self)
+                logger.info("agentic engine-state publisher: uri=%s id=%s",
+                            _ES_URI, _ES_ID)
+            except Exception as _e:
+                logger.warning("engine-state publisher disabled (%r)", _e)
+        {END}
+        _step_path = _os.environ.get("AGENTIC_STEP_LOG_PATH")"""
+
+PATCHES = [
+    ("header", HEADER_ANCHOR, HEADER + HEADER_ANCHOR),
+    ("init", INIT_ANCHOR, INIT_INSERT),
+]
+
+
+def find_target(venv: Path) -> Path:
+    for c in (venv / TARGET_REL, DEFAULT_VENV / TARGET_REL):
+        if c.is_file():
+            return c
+    raise FileNotFoundError(f"cannot find {TARGET_REL} under {venv}")
+
+
+def is_patched(t: str) -> bool:
+    return START in t
+
+
+def apply(target: Path):
+    text = target.read_text()
+    if is_patched(text):
+        print(f"[es-instr] already patched: {target}")
+        return
+    new = text
+    for name, src, dst in PATCHES:
+        if src not in new:
+            raise RuntimeError(f"patch {name!r}: anchor not found in {target}")
+        new = new.replace(src, dst, 1)
+    target.write_text(new)
+    print(f"[es-instr] applied {len(PATCHES)} patches -> {target}")
+
+
+def revert(target: Path):
+    text = target.read_text()
+    if not is_patched(text):
+        print(f"[es-instr] not patched: {target}")
+        return
+    pat = re.compile(r"[ \t]*" + re.escape(START) + r".*?" + re.escape(END) + r"\n",
+                     flags=re.DOTALL)
+    new = pat.sub("", text)
+    new = re.sub(r"\n{3,}class Scheduler\(", "\n\nclass Scheduler(", new)
+    target.write_text(new)
+    print(f"[es-instr] reverted: {target}")
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--apply", action="store_true")
+    p.add_argument("--revert", action="store_true")
+    p.add_argument("--check", action="store_true")
+    p.add_argument("--venv", type=Path, default=DEFAULT_VENV)
+    a = p.parse_args()
+    t = find_target(a.venv)
+    if a.apply:
+        apply(t)
+    elif a.revert:
+        revert(t)
+    elif a.check:
+        print(f"[es-instr] {'PATCHED' if is_patched(t.read_text()) else 'CLEAN'}: {t}")
+    else:
+        p.error("specify --apply/--revert/--check")
+
+
+if __name__ == "__main__":
+    main()

microbench/connector_tax/layerwise/mb7_layerwise.py

@@ -0,0 +1,294 @@
+#!/usr/bin/env python3
+"""MB7: correctness + perf of layer-wise KV push vs post-hoc transfer.
+
+Two 2-instance modes against A (src/producer) and B (dst/consumer):
+
+  baseline  : prefill A (await) -> THEN B pulls (post-hoc full transfer).
+              T_total = T_prefill + T_xfer  (sequential)
+  layerwise : dispatch B's remote-prefill (handshake) and A's prefill
+              CONCURRENTLY, so A pushes each layer as it computes it.
+              If overlap works, T_total ~= max(T_prefill, T_xfer) ~= T_prefill.
+
+Reference: T_prefill_only = a plain prefill on A with no transfer.
+
+Correctness: after the transfer, a plain follow-up to B on the same prompt
+must report cached_tokens >= ~prompt_len (the KV actually landed on B).
+
+The connector mode is selected by the launcher (run_mb7.sh): baseline uses the
+stock connector; layerwise deploys mooncake_connector.LAYERWISE.py +
+MOONCAKE_LAYERWISE=1. This script just drives the requests and measures.
+
+Usage:
+  python mb7_layerwise.py --mode layerwise --sizes 8192,32768,65536 --repeats 3 \
+      --src-port 8000 --dst-port 8001 --src-bp 8998 --dst-bp 8999 --out mb7.json
+"""
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import statistics
+import time
+import uuid
+from pathlib import Path
+
+import httpx
+
+MODEL = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct"
+KV_PER_TOK = 98304
+
+
+def synth_prompt(seed: int, n: int) -> list[int]:
+    import random
+    rng = random.Random(seed)
+    return [rng.randint(100, 150000) for _ in range(n)]
+
+
+async def get_engine_id(client, host, bp):
+    r = await client.get(f"http://{host}:{bp}/query")
+    r.raise_for_status()
+    return r.json()["0"]["engine_id"]
+
+
+async def completion(client, host, port, prompt, max_tokens, ktp=None):
+    payload = {
+        "model": MODEL, "prompt": prompt, "max_tokens": max_tokens,
+        "min_tokens": max_tokens if max_tokens == 1 else 1,
+        "temperature": 0.0, "stream": False,
+    }
+    if ktp:
+        payload["kv_transfer_params"] = ktp
+    t0 = time.perf_counter()
+    r = await client.post(f"http://{host}:{port}/v1/completions",
+                          json=payload, timeout=600.0)
+    dt = time.perf_counter() - t0
+    r.raise_for_status()
+    return dt, r.json()
+
+
+def cached_of(resp) -> int:
+    usage = resp.get("usage") or {}
+    det = usage.get("prompt_tokens_details") or {}
+    return det.get("cached_tokens", 0) or usage.get("cached_tokens", 0) or 0
+
+
+async def _stream_completion(client, host, port, prompt, max_tokens):
+    payload = {"model": MODEL, "prompt": prompt, "max_tokens": max_tokens,
+               "min_tokens": 1, "temperature": 0.0, "stream": True}
+    async with client.stream("POST", f"http://{host}:{port}/v1/completions",
+                             json=payload, timeout=600.0) as r:
+        r.raise_for_status()
+        async for _ in r.aiter_bytes():
+            pass
+
+
+class BackgroundLoad:
+    """Hold N concurrent long-decode streams across endpoints to keep busy."""
+    def __init__(self, client, endpoints, n, prompt_tokens=2000, out_tokens=6000):
+        self.client, self.endpoints, self.n = client, endpoints, n
+        self.pt, self.ot = prompt_tokens, out_tokens
+        self._stop = asyncio.Event()
+        self._tasks = []
+
+    async def _w(self, idx):
+        host, port = self.endpoints[idx % len(self.endpoints)]
+        seed = 800000 + idx
+        while not self._stop.is_set():
+            try:
+                await _stream_completion(self.client, host, port,
+                                         synth_prompt(seed, self.pt), self.ot)
+            except Exception:
+                await asyncio.sleep(0.5)
+            seed += 1
+
+    def start(self):
+        self._tasks = [asyncio.create_task(self._w(i)) for i in range(self.n)]
+
+    async def stop(self):
+        self._stop.set()
+        for t in self._tasks:
+            t.cancel()
+        await asyncio.gather(*self._tasks, return_exceptions=True)
+
+
+async def num_running(client, host, port):
+    try:
+        r = await client.get(f"http://{host}:{port}/metrics", timeout=5.0)
+        for line in r.text.splitlines():
+            if line.startswith("vllm:num_requests_running"):
+                return int(float(line.split()[-1]))
+    except Exception:
+        pass
+    return -1
+
+
+async def prefill_only(client, host, port, prompt):
+    """Reference: plain prefill cost on A, no transfer."""
+    dt, _ = await completion(client, host, port, prompt, max_tokens=1)
+    return dt
+
+
+async def measure_baseline(client, A, B, src_eid, src_bp_addr, prompt, seed):
+    tid = uuid.uuid4().hex
+    t0 = time.perf_counter()
+    t_pf, _ = await completion(client, *A, prompt, 1,
+                               ktp={"do_remote_decode": True, "transfer_id": tid})
+    t_xfer, _ = await completion(client, *B, prompt, 1,
+                                 ktp={"do_remote_prefill": True, "transfer_id": tid,
+                                      "remote_engine_id": src_eid,
+                                      "remote_bootstrap_addr": src_bp_addr})
+    t_total = time.perf_counter() - t0
+    # correctness: B follow-up should hit cache
+    _, fr = await completion(client, *B, prompt, 1)
+    return {"t_prefill_s": t_pf, "t_xfer_s": t_xfer, "t_total_s": t_total,
+            "cached": cached_of(fr)}
+
+
+async def measure_layerwise(client, A, B, src_eid, src_bp_addr, prompt, seed):
+    """Dispatch B handshake + A prefill concurrently => layer-wise overlap."""
+    tid = uuid.uuid4().hex
+    t0 = time.perf_counter()
+
+    async def run_B():
+        return await completion(client, *B, prompt, 1,
+                                ktp={"do_remote_prefill": True, "transfer_id": tid,
+                                     "remote_engine_id": src_eid,
+                                     "remote_bootstrap_addr": src_bp_addr})
+
+    async def run_A():
+        # small head start for B's handshake to reach A before A's forward
+        await asyncio.sleep(0.05)
+        return await completion(client, *A, prompt, 1,
+                                ktp={"do_remote_decode": True, "transfer_id": tid})
+
+    b_task = asyncio.create_task(run_B())
+    a_task = asyncio.create_task(run_A())
+    (t_b, _), (t_a, _) = await asyncio.gather(b_task, a_task)
+    t_total = time.perf_counter() - t0
+    _, fr = await completion(client, *B, prompt, 1)
+    return {"t_A_s": t_a, "t_B_s": t_b, "t_total_s": t_total,
+            "cached": cached_of(fr)}
+
+
+async def main_async(a):
+    sizes = [int(s) for s in a.sizes.split(",")]
+    A = (a.src_host, a.src_port)
+    B = (a.dst_host, a.dst_port)
+    limits = httpx.Limits(max_connections=64, max_keepalive_connections=64)
+    async with httpx.AsyncClient(limits=limits, trust_env=False) as client:
+        src_eid = await get_engine_id(client, a.src_host, a.src_bp)
+        src_bp_addr = f"http://{a.src_host}:{a.src_bp}"
+        print(f"[mb7] mode={a.mode} bg_load={a.bg_load} src_eid={src_eid[:16]}...")
+
+        loader = None
+        if a.bg_load > 0:
+            loader = BackgroundLoad(client, [A, B], a.bg_load)
+            loader.start()
+            print(f"[mb7] ramping background load ({a.bg_load}) ...")
+            for _ in range(40):
+                await asyncio.sleep(1.0)
+                na = await num_running(client, *A)
+                nb = await num_running(client, *B)
+                if na >= 1 and nb >= 1:
+                    print(f"[mb7] busy: A_run={na} B_run={nb}")
+                    break
+
+        # --- concurrent correctness mode: fire N transfers at once ----------
+        if a.concurrent > 1 and a.mode == "layerwise":
+            print(f"[mb7] CONCURRENT correctness: {a.concurrent} simultaneous "
+                  f"transfers per size (src=A stresses concurrent producing)")
+            all_ok = True
+            for sz in sizes:
+                tasks = [
+                    asyncio.create_task(measure_layerwise(
+                        client, A, B, src_eid, src_bp_addr,
+                        synth_prompt(sz * 1000 + j, sz), sz * 1000 + j))
+                    for j in range(a.concurrent)
+                ]
+                rows = await asyncio.gather(*tasks)
+                oks = [r["cached"] >= int(sz * 0.9) for r in rows]
+                all_ok = all_ok and all(oks)
+                print(f"  sz={sz:>6} x{a.concurrent}: cached="
+                      f"{[r['cached'] for r in rows]} correct={oks}")
+            print(f"[mb7] CONCURRENT correctness: "
+                  f"{'ALL PASS' if all_ok else 'FAILURE'}")
+            if loader:
+                await loader.stop()
+            return
+
+        results = []
+        for sz in sizes:
+            for rep in range(a.repeats):
+                prompt = synth_prompt(sz * 100 + rep, sz)
+                # reference prefill-only cost (fresh prompt, different seed so no cache)
+                t_pf_only = await prefill_only(
+                    client, *A, synth_prompt(sz * 100 + rep + 555, sz))
+                if a.mode == "baseline":
+                    row = await measure_baseline(client, A, B, src_eid, src_bp_addr,
+                                                 prompt, sz * 100 + rep)
+                else:
+                    row = await measure_layerwise(client, A, B, src_eid, src_bp_addr,
+                                                  prompt, sz * 100 + rep)
+                row.update({"mode": a.mode, "size": sz, "rep": rep,
+                            "t_prefill_only_s": t_pf_only,
+                            "kv_gib": sz * KV_PER_TOK / 2**30,
+                            "correct": row["cached"] >= int(sz * 0.9)})
+                results.append(row)
+                extra = (f"xfer={row.get('t_xfer_s', 0)*1000:.0f}ms"
+                         if a.mode == "baseline"
+                         else f"tA={row.get('t_A_s',0)*1000:.0f}ms tB={row.get('t_B_s',0)*1000:.0f}ms")
+                print(f"  sz={sz:>6} rep={rep} pf_only={t_pf_only*1000:6.0f}ms "
+                      f"total={row['t_total_s']*1000:7.0f}ms {extra} "
+                      f"cached={row['cached']}/{sz} correct={row['correct']}")
+
+        if loader:
+            await loader.stop()
+
+    # summary
+    print(f"\n=== {a.mode} (bg={a.bg_load}) summary ===")
+    print(f"{'size':>7} {'n':>2} {'pf_only_ms':>11} {'total_ms':>9} "
+          f"{'overhead_ms':>12} {'correct':>8}")
+    summary = []
+    for sz in sizes:
+        rs = [r for r in results if r["size"] == sz]
+        if not rs:
+            continue
+        pf = statistics.median(r["t_prefill_only_s"] for r in rs) * 1000
+        tot = statistics.median(r["t_total_s"] for r in rs) * 1000
+        allok = all(r["correct"] for r in rs)
+        # overhead = total - prefill_only = the part NOT hidden behind prefill
+        overhead = tot - pf
+        summary.append({"size": sz, "n": len(rs), "pf_only_ms": pf,
+                        "total_ms": tot, "overhead_ms": overhead,
+                        "all_correct": allok})
+        print(f"{sz:>7} {len(rs):>2} {pf:>11.0f} {tot:>9.0f} {overhead:>12.0f} "
+              f"{str(allok):>8}")
+
+    Path(a.out).write_text(json.dumps(
+        {"mode": a.mode, "model": MODEL, "raw": results, "summary": summary}, indent=2))
+    print(f"\n[mb7] wrote {a.out}")
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--mode", choices=["baseline", "layerwise"], required=True)
+    p.add_argument("--src-host", default="127.0.0.1")
+    p.add_argument("--dst-host", default="127.0.0.1")
+    p.add_argument("--src-port", type=int, default=8000)
+    p.add_argument("--dst-port", type=int, default=8001)
+    p.add_argument("--src-bp", type=int, default=8998)
+    p.add_argument("--dst-bp", type=int, default=8999)
+    p.add_argument("--sizes", default="8192,32768,65536")
+    p.add_argument("--repeats", type=int, default=3)
+    p.add_argument("--bg-load", type=int, default=0,
+                   help="N concurrent background decode streams across A+B")
+    p.add_argument("--concurrent", type=int, default=1,
+                   help="layerwise: fire N simultaneous transfers to test "
+                        "concurrent-producing correctness")
+    p.add_argument("--out", default="mb7_result.json")
+    args = p.parse_args()
+    asyncio.run(main_async(args))
+
+
+if __name__ == "__main__":
+    main()

microbench/connector_tax/layerwise/migration_target.py

@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+"""P2: real-state-aware migration target selection.
+
+Pure helpers (no proxy deps) so they're unit-testable. The router calls
+`rank_migration_targets` to pick the decode target, using REAL engine state
+(from the engine-state store) when available, falling back to shadow counters.
+
+Key fix over the shadow-only Mechanism B: deprioritise targets that are
+mid-large-prefill (`max_prefill_remaining` high) — those hold the GIL and
+stall the mooncake receiver_loop, which is the ~45% control-plane residual
+that layer-wise transfer does NOT fix. Also avoid targets near the KV
+capacity wall (`gpu_kv_used_frac` high).
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass
+class TargetCandidate:
+    idx: int
+    cache_hit: int                 # estimated transfer bytes saved (tokens)
+    shadow_num_req: int            # proxy shadow counter (fallback)
+    ongoing_tokens: int            # shadow tertiary
+    real_state: dict | None = None # engine-state record, or None if stale/missing
+
+
+def real_load(c: TargetCandidate) -> float:
+    """Effective load: prefer real (running + waiting); else shadow."""
+    rs = c.real_state
+    if rs is not None:
+        return float(rs.get("num_running", 0) + rs.get("num_waiting", 0))
+    return float(c.shadow_num_req)
+
+
+def big_prefill_remaining(c: TargetCandidate) -> int:
+    """Largest in-progress prefill on the candidate (GIL-stall predictor).
+    0 when unknown (no real state) so we don't over-penalise blind."""
+    rs = c.real_state
+    return int(rs.get("max_prefill_remaining", 0)) if rs is not None else 0
+
+
+def kv_used_frac(c: TargetCandidate) -> float:
+    rs = c.real_state
+    if rs is not None:
+        f = rs.get("gpu_kv_used_frac", -1.0)
+        return float(f) if f is not None and f >= 0 else 0.0
+    return 0.0
+
+
+def target_sort_key(
+    c: TargetCandidate,
+    big_prefill_threshold: int = 16000,
+    kv_wall_frac: float = 0.90,
+):
+    """Sort key (lower = better). Ordering of concerns:
+      1. NOT mid-large-prefill (avoid the GIL-stall dst)         [bool]
+      2. NOT near the KV capacity wall                            [bool]
+      3. most cache-rich  (fewest transfer bytes)  -> -cache_hit
+      4. lowest real load
+      5. lowest ongoing_tokens (shadow tertiary tie-break)
+    """
+    stalls = 1 if big_prefill_remaining(c) >= big_prefill_threshold else 0
+    near_wall = 1 if kv_used_frac(c) >= kv_wall_frac else 0
+    return (stalls, near_wall, -c.cache_hit, real_load(c), c.ongoing_tokens)
+
+
+def rank_migration_targets(
+    candidates: list[TargetCandidate],
+    big_prefill_threshold: int = 16000,
+    kv_wall_frac: float = 0.90,
+) -> TargetCandidate | None:
+    """Return the best candidate, or None if the list is empty."""
+    if not candidates:
+        return None
+    return min(
+        candidates,
+        key=lambda c: target_sort_key(c, big_prefill_threshold, kv_wall_frac),
+    )

microbench/connector_tax/layerwise/results/mb7_baseline.json

@@ -0,0 +1,140 @@
+{
+  "mode": "baseline",
+  "model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct",
+  "raw": [
+    {
+      "t_prefill_s": 0.5736213000018324,
+      "t_xfer_s": 0.36388630099827424,
+      "t_total_s": 0.9375749369974073,
+      "cached": 8176,
+      "mode": "baseline",
+      "size": 8192,
+      "rep": 0,
+      "t_prefill_only_s": 1.0551288530004967,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 0.5740011439993395,
+      "t_xfer_s": 0.12374231500143651,
+      "t_total_s": 0.6978207100000873,
+      "cached": 8176,
+      "mode": "baseline",
+      "size": 8192,
+      "rep": 1,
+      "t_prefill_only_s": 0.5743715360003989,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 0.5732713990000775,
+      "t_xfer_s": 0.10885842400239198,
+      "t_total_s": 0.6821924389987544,
+      "cached": 8176,
+      "mode": "baseline",
+      "size": 8192,
+      "rep": 2,
+      "t_prefill_only_s": 0.5745713680007611,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 1.4892208660021424,
+      "t_xfer_s": 0.2091717740004242,
+      "t_total_s": 1.6984740270017937,
+      "cached": 16368,
+      "mode": "baseline",
+      "size": 16384,
+      "rep": 0,
+      "t_prefill_only_s": 1.4990949730017746,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 1.4885207330007688,
+      "t_xfer_s": 0.2010940889995254,
+      "t_total_s": 1.6896768289989268,
+      "cached": 16368,
+      "mode": "baseline",
+      "size": 16384,
+      "rep": 1,
+      "t_prefill_only_s": 1.4898170189990196,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 1.4895933570005582,
+      "t_xfer_s": 0.2026357979993918,
+      "t_total_s": 1.6922962099997676,
+      "cached": 16368,
+      "mode": "baseline",
+      "size": 16384,
+      "rep": 2,
+      "t_prefill_only_s": 1.4907751430000644,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 4.438586502998078,
+      "t_xfer_s": 0.37847799000155646,
+      "t_total_s": 4.817142683001293,
+      "cached": 32752,
+      "mode": "baseline",
+      "size": 32768,
+      "rep": 0,
+      "t_prefill_only_s": 4.437922253000579,
+      "kv_gib": 3.0,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 4.4350325649975275,
+      "t_xfer_s": 0.5313337980005599,
+      "t_total_s": 4.966431269000168,
+      "cached": 32752,
+      "mode": "baseline",
+      "size": 32768,
+      "rep": 1,
+      "t_prefill_only_s": 4.437473922000208,
+      "kv_gib": 3.0,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 4.436279826000828,
+      "t_xfer_s": 0.6335160570015432,
+      "t_total_s": 5.069869226001174,
+      "cached": 32752,
+      "mode": "baseline",
+      "size": 32768,
+      "rep": 2,
+      "t_prefill_only_s": 4.440119222999783,
+      "kv_gib": 3.0,
+      "correct": true
+    }
+  ],
+  "summary": [
+    {
+      "size": 8192,
+      "n": 3,
+      "pf_only_ms": 574.5713680007611,
+      "total_ms": 697.8207100000873,
+      "overhead_ms": 123.24934199932613,
+      "all_correct": true
+    },
+    {
+      "size": 16384,
+      "n": 3,
+      "pf_only_ms": 1490.7751430000644,
+      "total_ms": 1692.2962099997676,
+      "overhead_ms": 201.52106699970318,
+      "all_correct": true
+    },
+    {
+      "size": 32768,
+      "n": 3,
+      "pf_only_ms": 4437.922253000579,
+      "total_ms": 4966.431269000168,
+      "overhead_ms": 528.5090159995889,
+      "all_correct": true
+    }
+  ]
+}

microbench/connector_tax/layerwise/results/mb7_baseline_bg16.json

@@ -0,0 +1,140 @@
+{
+  "mode": "baseline",
+  "model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct",
+  "raw": [
+    {
+      "t_prefill_s": 0.5868483350022871,
+      "t_xfer_s": 0.19584889299949282,
+      "t_total_s": 0.7827702419999696,
+      "cached": 8176,
+      "mode": "baseline",
+      "size": 8192,
+      "rep": 0,
+      "t_prefill_only_s": 0.5920699099988269,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 0.5875704979989678,
+      "t_xfer_s": 0.1554814909977722,
+      "t_total_s": 0.7431365060001554,
+      "cached": 8176,
+      "mode": "baseline",
+      "size": 8192,
+      "rep": 1,
+      "t_prefill_only_s": 0.5814537600017502,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 0.5852241569991747,
+      "t_xfer_s": 0.15129724399957922,
+      "t_total_s": 0.7365909610016388,
+      "cached": 8176,
+      "mode": "baseline",
+      "size": 8192,
+      "rep": 2,
+      "t_prefill_only_s": 0.5846994370003813,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 1.498547145001794,
+      "t_xfer_s": 0.2475714690008317,
+      "t_total_s": 1.7462187470009667,
+      "cached": 16368,
+      "mode": "baseline",
+      "size": 16384,
+      "rep": 0,
+      "t_prefill_only_s": 1.5670790190015396,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 1.5025789940009417,
+      "t_xfer_s": 0.24532966799961287,
+      "t_total_s": 1.7479741930001182,
+      "cached": 16368,
+      "mode": "baseline",
+      "size": 16384,
+      "rep": 1,
+      "t_prefill_only_s": 1.5008903820016712,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 1.5021674179988622,
+      "t_xfer_s": 0.24640760400143336,
+      "t_total_s": 1.7486415580024186,
+      "cached": 16368,
+      "mode": "baseline",
+      "size": 16384,
+      "rep": 2,
+      "t_prefill_only_s": 1.509417139001016,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 4.444555983998725,
+      "t_xfer_s": 0.4227471090016479,
+      "t_total_s": 4.86737214599998,
+      "cached": 32752,
+      "mode": "baseline",
+      "size": 32768,
+      "rep": 0,
+      "t_prefill_only_s": 4.4467717689985875,
+      "kv_gib": 3.0,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 4.442135782999685,
+      "t_xfer_s": 0.7519038230020669,
+      "t_total_s": 5.194113359000767,
+      "cached": 32752,
+      "mode": "baseline",
+      "size": 32768,
+      "rep": 1,
+      "t_prefill_only_s": 4.445541313998547,
+      "kv_gib": 3.0,
+      "correct": true
+    },
+    {
+      "t_prefill_s": 4.439772993999213,
+      "t_xfer_s": 0.7855456319994119,
+      "t_total_s": 5.225392060998274,
+      "cached": 32752,
+      "mode": "baseline",
+      "size": 32768,
+      "rep": 2,
+      "t_prefill_only_s": 4.442906365002273,
+      "kv_gib": 3.0,
+      "correct": true
+    }
+  ],
+  "summary": [
+    {
+      "size": 8192,
+      "n": 3,
+      "pf_only_ms": 584.6994370003813,
+      "total_ms": 743.1365060001554,
+      "overhead_ms": 158.43706899977406,
+      "all_correct": true
+    },
+    {
+      "size": 16384,
+      "n": 3,
+      "pf_only_ms": 1509.417139001016,
+      "total_ms": 1747.9741930001182,
+      "overhead_ms": 238.5570539991022,
+      "all_correct": true
+    },
+    {
+      "size": 32768,
+      "n": 3,
+      "pf_only_ms": 4445.541313998547,
+      "total_ms": 5194.113359000767,
+      "overhead_ms": 748.57204500222,
+      "all_correct": true
+    }
+  ]
+}

microbench/connector_tax/layerwise/results/mb7_layerwise.json

@@ -0,0 +1,140 @@
+{
+  "mode": "layerwise",
+  "model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct",
+  "raw": [
+    {
+      "t_A_s": 0.5749198459998297,
+      "t_B_s": 0.6508419569981925,
+      "t_total_s": 0.6509377910006151,
+      "cached": 8176,
+      "mode": "layerwise",
+      "size": 8192,
+      "rep": 0,
+      "t_prefill_only_s": 1.0447357020020718,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_A_s": 0.574626908000937,
+      "t_B_s": 0.6306310719992325,
+      "t_total_s": 0.6307087300010608,
+      "cached": 8176,
+      "mode": "layerwise",
+      "size": 8192,
+      "rep": 1,
+      "t_prefill_only_s": 0.5731983850018878,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_A_s": 0.5756587910000235,
+      "t_B_s": 0.6316753270002664,
+      "t_total_s": 0.6317471290021786,
+      "cached": 8176,
+      "mode": "layerwise",
+      "size": 8192,
+      "rep": 2,
+      "t_prefill_only_s": 0.5737888650000968,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_A_s": 1.4953326409995498,
+      "t_B_s": 1.5502465710014803,
+      "t_total_s": 1.5503262860001996,
+      "cached": 16368,
+      "mode": "layerwise",
+      "size": 16384,
+      "rep": 0,
+      "t_prefill_only_s": 1.5000705940001353,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_A_s": 1.493850356000621,
+      "t_B_s": 1.5505031290012994,
+      "t_total_s": 1.5505791659998067,
+      "cached": 16368,
+      "mode": "layerwise",
+      "size": 16384,
+      "rep": 1,
+      "t_prefill_only_s": 1.4924546469992492,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_A_s": 1.4979969070009247,
+      "t_B_s": 1.554968774002191,
+      "t_total_s": 1.5551903560008213,
+      "cached": 16368,
+      "mode": "layerwise",
+      "size": 16384,
+      "rep": 2,
+      "t_prefill_only_s": 1.4914496510027675,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_A_s": 4.4403588690001925,
+      "t_B_s": 4.496483378999983,
+      "t_total_s": 4.4965666819989565,
+      "cached": 32752,
+      "mode": "layerwise",
+      "size": 32768,
+      "rep": 0,
+      "t_prefill_only_s": 4.440080869000667,
+      "kv_gib": 3.0,
+      "correct": true
+    },
+    {
+      "t_A_s": 4.44209005599987,
+      "t_B_s": 4.499940814999718,
+      "t_total_s": 4.500021006002498,
+      "cached": 32752,
+      "mode": "layerwise",
+      "size": 32768,
+      "rep": 1,
+      "t_prefill_only_s": 4.440225810998527,
+      "kv_gib": 3.0,
+      "correct": true
+    },
+    {
+      "t_A_s": 4.437084657998639,
+      "t_B_s": 4.496842522999941,
+      "t_total_s": 4.496926485000586,
+      "cached": 32752,
+      "mode": "layerwise",
+      "size": 32768,
+      "rep": 2,
+      "t_prefill_only_s": 4.439449855002749,
+      "kv_gib": 3.0,
+      "correct": true
+    }
+  ],
+  "summary": [
+    {
+      "size": 8192,
+      "n": 3,
+      "pf_only_ms": 573.7888650000968,
+      "total_ms": 631.7471290021786,
+      "overhead_ms": 57.958264002081705,
+      "all_correct": true
+    },
+    {
+      "size": 16384,
+      "n": 3,
+      "pf_only_ms": 1492.4546469992492,
+      "total_ms": 1550.5791659998067,
+      "overhead_ms": 58.124519000557484,
+      "all_correct": true
+    },
+    {
+      "size": 32768,
+      "n": 3,
+      "pf_only_ms": 4440.080869000667,
+      "total_ms": 4496.926485000586,
+      "overhead_ms": 56.845615999918664,
+      "all_correct": true
+    }
+  ]
+}

microbench/connector_tax/layerwise/results/mb7_layerwise_bg16.json

@@ -0,0 +1,140 @@
+{
+  "mode": "layerwise",
+  "model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct",
+  "raw": [
+    {
+      "t_A_s": 0.5905098549992545,
+      "t_B_s": 0.6900827390018094,
+      "t_total_s": 0.6904724189989793,
+      "cached": 8176,
+      "mode": "layerwise",
+      "size": 8192,
+      "rep": 0,
+      "t_prefill_only_s": 0.5852864849985053,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_A_s": 0.5897548109969648,
+      "t_B_s": 0.6827381169969158,
+      "t_total_s": 0.6828304180016858,
+      "cached": 8176,
+      "mode": "layerwise",
+      "size": 8192,
+      "rep": 1,
+      "t_prefill_only_s": 0.5890174580017629,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_A_s": 0.5850713190011447,
+      "t_B_s": 0.6744917560026806,
+      "t_total_s": 0.6745770380002796,
+      "cached": 8176,
+      "mode": "layerwise",
+      "size": 8192,
+      "rep": 2,
+      "t_prefill_only_s": 0.5943713950000529,
+      "kv_gib": 0.75,
+      "correct": true
+    },
+    {
+      "t_A_s": 1.5030149390004226,
+      "t_B_s": 1.596173029000056,
+      "t_total_s": 1.597060264000902,
+      "cached": 16368,
+      "mode": "layerwise",
+      "size": 16384,
+      "rep": 0,
+      "t_prefill_only_s": 1.5130829510017065,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_A_s": 1.499876754998695,
+      "t_B_s": 1.5940461120007967,
+      "t_total_s": 1.5948001770011615,
+      "cached": 16368,
+      "mode": "layerwise",
+      "size": 16384,
+      "rep": 1,
+      "t_prefill_only_s": 1.5024838620010996,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_A_s": 1.5068977490009274,
+      "t_B_s": 1.5950395179970656,
+      "t_total_s": 1.59571184500237,
+      "cached": 16368,
+      "mode": "layerwise",
+      "size": 16384,
+      "rep": 2,
+      "t_prefill_only_s": 1.5303227439981129,
+      "kv_gib": 1.5,
+      "correct": true
+    },
+    {
+      "t_A_s": 4.4503932609986805,
+      "t_B_s": 4.538851200999488,
+      "t_total_s": 4.539281312001549,
+      "cached": 32752,
+      "mode": "layerwise",
+      "size": 32768,
+      "rep": 0,
+      "t_prefill_only_s": 4.446753306998289,
+      "kv_gib": 3.0,
+      "correct": true
+    },
+    {
+      "t_A_s": 4.44226107799841,
+      "t_B_s": 4.551636377997056,
+      "t_total_s": 4.552389411001059,
+      "cached": 32752,
+      "mode": "layerwise",
+      "size": 32768,
+      "rep": 1,
+      "t_prefill_only_s": 4.44538704000297,
+      "kv_gib": 3.0,
+      "correct": true
+    },
+    {
+      "t_A_s": 4.440309538000292,
+      "t_B_s": 4.539836316998844,
+      "t_total_s": 4.540553365997766,
+      "cached": 32752,
+      "mode": "layerwise",
+      "size": 32768,
+      "rep": 2,
+      "t_prefill_only_s": 4.443476915999781,
+      "kv_gib": 3.0,
+      "correct": true
+    }
+  ],
+  "summary": [
+    {
+      "size": 8192,
+      "n": 3,
+      "pf_only_ms": 589.0174580017629,
+      "total_ms": 682.8304180016858,
+      "overhead_ms": 93.8129599999229,
+      "all_correct": true
+    },
+    {
+      "size": 16384,
+      "n": 3,
+      "pf_only_ms": 1513.0829510017065,
+      "total_ms": 1595.71184500237,
+      "overhead_ms": 82.62889400066342,
+      "all_correct": true
+    },
+    {
+      "size": 32768,
+      "n": 3,
+      "pf_only_ms": 4445.38704000297,
+      "total_ms": 4540.553365997766,
+      "overhead_ms": 95.16632599479635,
+      "all_correct": true
+    }
+  ]
+}