MB5 PD ablation v2 results: concurrency axis + reuse 3-way writeup

- fig3_conc32k.json + fig3_concurrency_axis.png: agentic-corner concurrency
  sweep (in=32768, reuse=0.984, out=128), N 8->128, PD capped 600s / colo
  uncapped. colo completes 100% at every N (graceful, E2E 2.4s->81s); every
  static PD split collapses, earlier as N rises (viable only N<=16; <27% by N=32).
- analysis/mb5_pd_ablation/README.md: self-contained v2 writeup. Reuse axis
  3-way (A=d2048/o256, C=d2048/o128, B=d1024/o128) decomposes shape: output
  ~negligible, delta (real prefill/turn) dominant; crossover to colo at reuse
  ~90-95% robust. Run on dash2 (dash0 NICs faulted for Mooncake).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.gitignore

@@ -3,7 +3,14 @@ __pycache__/
 .venv/
 *.egg-info/
 outputs/
-traces/
+traces/*
+# ship the anonymized sampled trace + its provenance (metadata only, no cleartext)
+!traces/w600_r0.0015_st30.jsonl
+!traces/README.md
 *.log
 .claude/
 # third_party/vllm tracked in git for patch management
+!traces/w600_r0.0015_st30_first600s.jsonl
+# + time_to_parent_chat annotation (for --dispatch-mode thinktime); same anon data
+!traces/w600_r0.0015_st30_ttp.jsonl
+!traces/w600_r0.0015_st30_first600s_ttp.jsonl

MEETING.md

@@ -6,18 +6,26 @@
 
 ## 1. 关键洞察：Dispatch Coupling
 
-Chatbot：turn 间有人类 think-time，系统快慢 ⊥ 下一 turn 到达率。
-Agentic：turn 间只有 tool-call 返回 (≈0)，**系统跑慢 → session 停留长 → 并发多 → KV pool 紧 → 更慢**。
+每个 turn 间有一段外部 gap `T_external`（chatbot 是人类读+想+打字；agentic 是 tool 执行）。**Little's Law `L = Λ · N · (W_turn + T_external)`** 在两种 workload 下都成立 —— 差异在于 `T_external` 的分布相对于 `W_turn` 的位置：
+- `T_external ≫ W_turn` → 开环 regime：scheduler 退一步不动 L
+- `T_external ≲ W_turn` → 闭环 regime：`W_turn(L)` 因 KV 竞争耦合到 L，反馈环把 scheduler 的 ε 退步放大几倍
 
-Little's Law 隐式方程：
+**Production trace 实测 `T_external` 分布**（next.start − prev.end，formatted session 链作 ground truth）：
 
-```
-L = Λ · N · W_turn(L)        # agentic, T_human≈0
-```
+![](figs/f3a_inter_turn_gap.png)
 
-小扰动分析：amplification = `1 / (1 − Λ·N·W'(L*))`，系统接近 KV 饱和时发散。
+| | Agentic | Chatbot |
+|---|---:|---:|
+| p50 | **1.6s** | **7.2s** |
+| gap < 1s | **39%** | 4% |
+| gap < 5s | 67% | 29% |
+| p99 | 738s | 43s |
 
-**实测**：lmetric 跑 600s trace 用 49 min wall-clock = **8x amplification**。同硬件 unified 比 lmetric session 清空速度 ~3x。**per-turn W 的小差异被放大成 wall-clock 数量级差距** —— 这意味着 locality 不是 nice-to-have，是 dominant lever。
+两个分布形状完全不同：chatbot unimodal 集中在 5–15s（人类节奏）；agentic bimodal —— **39% 的 gap 在 sub-second 里（autonomous tool-call mode）**，外加 13% > 30s 的长尾。**Agentic 的 sub-second mass 是 chatbot 没有的**，正是 dispatch coupling 激活的来源。
+
+**实测 regime**：在 unified（TTFT p90 = 7.3s）下，**73% 的 agentic turn 把系统推进闭环**（W_turn > T_external），chatbot 仅 32%。在 lmetric（15.7s）下 agentic 80%、**chatbot 也到 88%** —— lmetric 把 chatbot 自己也拖进闭环，这就是它在两种 workload 都 underperform 的根因。
+
+**结果**：lmetric 跑 600s trace 用 49 min wall-clock = **8x amplification**。**per-turn W 的小差异被放大成 wall-clock 数量级差距** —— locality 不是 nice-to-have，是 dominant lever。
 
 ---
 
@@ -27,7 +35,7 @@ L = Λ · N · W_turn(L)        # agentic, T_human≈0
 |---|---|---|
 | KV reuse 几乎只在 session 内 | intra 93.2% / cross 5.7% / shared 1.1% | ![](figs/f2a_reuse_topology.png) |
 | Session 极度偏斜 | production trace 上 top 1% / 5% / 10% / 25% / 50% = **46.5% / 66.5% / 74.6% / 87.5% / 96.0%** input mass | ![](figs/f2b_session_skew.png) |
-| 单请求 KV footprint 已经很大 | p99 = 11.8 GiB ≈ H20 12% | ![](figs/f2c_kv_footprint_cdf.png) |
+| 单请求 KV footprint 大，单 instance KV pool 很快被占满 | per-instance KV pool ≈ **38 GiB**（0.4 × 96 GiB H20，剩 50% params + 10% activation）；p99 req 11.5 GiB → 一个 instance 只装 **3 个 p99 decode**；4P+4D 让系统 decode 容量直接减半 | ![](figs/f2c_kv_footprint_cdf.png) |
 
 理论 APC 上界 = intra-session 79.6% / any-session 80.3%，差 <1pp。**任何不 affinity 的调度都丢绝大部分 reuse。**
 
@@ -41,6 +49,8 @@ L = Λ · N · W_turn(L)        # agentic, T_human≈0
 
 LMetric 56.9%、load_only 54.1% APC，远低于 79.6% 上界。23pp 缺口直接来自跨 instance 路由丢的 intra-session hit。
 
+注意 LMetric 比 load_only 只好 **+3.3pp**：LMetric 的 score = `(pending_prefill + input − cache_hit) × num_requests`，cache_hit 只作 cost-model 减项，但 score 是**乘性**的 —— 一个有 affinity 的 instance 因 num_requests 高被乘式吃掉 cache 收益，LMetric 仍然会选冷 instance。sticky 把 cache 作硬约束直接拉到 77.2%。**结论：cache-aware-load routing 不够 —— affinity 必须是独立路由路径，不能折叠进 load cost 里**。
+
 ### 静态 PD-disagg：D 侧 KV 容量墙
 
 ![](figs/f4b_pdsep_kv_wall.png)

PAPER_OUTLINE.md

@@ -1,375 +1,362 @@
-# EAR: Elastic Affinity Routing for Agentic LLM Serving
+# GPU-Hit-First: Serving Agentic LLM Workloads by Keeping the Working Set in HBM
 
-> **One-liner**: Agentic LLM workload 的 KV reuse 93% 是 intra-session 的，且 turn 间 tool-call 反馈耦合把单 request 的延迟差放大成 throughput 差距 —— locality 因此成为主导调度杠杆；现有 load-balance 丢 locality、静态 PD-disagg 撞 D 侧 KV 墙、pure session-sticky 造 hot pin，我们提出 session-affinity routing + hot-instance 触发 session migration 的调度器 **EAR (Elastic Affinity Router)**，单一方案同时拿到 locality 和 balance。
+> **Thesis (one-liner)**: 对 agentic LLM 负载，用户感受到的端到端 metric 是 **request latency / TPS / GPU
+> utilization**，而它们由一件事主导 —— **KV cache 命中是否发生在 GPU HBM 上**。Agentic 的 KV reuse 93% 在
+> session 内、且活跃 working set 小到一个节点就能常驻 HBM；命中层级 `GPU ≫ CPU-local > remote-RDMA-store ≫
+> recompute` 的代价差随 context 拉大。由此得到一条统一原则 —— **GPU-hit-first**：把活跃 working set 留在 HBM，
+> 而不是建深的 CPU/storage hierarchy 去追长尾。三个推论分别修复现有系统的三处失配：(3.1) 让 PD-colocation 重新
+> 成为默认；(3.2) 在全局路由里做 biased KV-cache-awareness；(3.3) 用 KV migration 而非 replication 做跨实例
+> GPU 去重。
+
+> **Framing note (2026-05-30)**：本 outline 取代早期的 "EAR: Elastic Affinity Routing" 版本（保留在 git
+> 历史里）。原 EAR 的 dispatch-coupling 形式化在此 **降级为 §2 的 metric 论证**（解释"为什么是 request latency
+> 而不是 TTFT/TPOT"），不再是 headline；headline 升格为 GPU-hit-first 原则，affinity routing / migration
+> 成为该原则的两个推论（§3.2 / §3.3）。
 
 ---
 
-## 📊 Validation Status (2026-05-27)
+## 📊 Validation Status (2026-05-30)
 
-| 部分 | 现有数据 | 待补 |
-|---|---|---|
-| §2 Workload characterization | ✅ 完整 (3 张图复用) | — |
-| §3.1 Load-balance 丢 locality | ✅ 完整 (`f4a`) | — |
-| §3.2 静态 PD-disagg 撞 KV 墙 | ✅ 完整 (`f4b`) | — |
-| §3.3 Sticky 造 hot pin | ✅ 完整 (`f4c`, `f4d`) | — |
-| §4.1-2 Affinity routing | ✅ 已实现（current `unified` 算法）| — |
-| `kv_both` substrate cost | ✅ **VERIFIED net-positive** (2026-05-27, commit `ef9e010`) | TTFT p90 −18.6% w/o DR-fix, −36.6% w/ DR-fix |
-| §4.3 Migration mechanism (e2e) | 🚧 **PARTIAL** | substrate 已通；e2e trigger + target selection 实验未跑 |
-| §5.2 End-to-end | ⚠️ 5/6 baseline 有数据 (`f6`) | 缺 static PD-disagg；EAR 列待 migration |
-| §5.3 Ablation | 🚧 **PARTIAL DEFER** | 仅 affinity-only 现可做，full 待 migration |
-| §5.4 Dispatch coupling validation | 🚧 **NEW DATA NEEDED** | 5 baseline wall-clock 重跑（Phase 1 patch 后）|
-| §5.5 Sensitivity | 🚧 **PARTIAL DEFER** | λ/skew/KV pool 可做；`T_hot`/`T_cool` 待 migration |
-| §5.6 Migration microbench | 🚧 **FULL DEFER** | 完全依赖 migration validation |
-
-**前提背景**：team 之前 4 次尝试 migration 都因 transfer overhead 被还原（见 `analysis/unified_routing_fix_review.md`）；2026-05-27 的 trace-replay A/B/C（`microbench/connector_tax/cache_sweep/REPORT_TRACE_REPLAY.md`）证明 `kv_both` substrate 已经反转 —— 不仅 +45% penalty obsolete，substrate 本身就是 net positive（TTFT p90 −18.6% vs plain，DR-fix 后 −36.6%）。**之前 4 次 migration revert 的最大根因消失，但 e2e migration 策略层（trigger + target selection 在反馈环里的真实收益）仍未直接验证 ——  EAR 的 migration 部分实验已无 substrate 风险，只剩策略层风险。**
+| 章节 | 论点 | 证据 | 状态 |
+|---|---|---|---|
+| §1 | 背景：PD-colo / PD-disagg / KV storage hierarchy | — | 写作 |
+| §2 metric | request latency 而非 TTFT/TPOT；TPS；GPU util | dispatch coupling + amplification；`bench_report.py` | 🟡 论证全，wall-clock sweep 待补 |
+| §2.1 | KV hit 普遍且关键 | `f2a` intra 93.2%、APC 上界 79.6%、C1/C2 | ✅ |
+| **§2.2** | **GPU hit > CPU hit > RDMA-store hit ≫ miss** | **`v2/figs/exp_a_tier_latency.png`（四层实测）** | ✅ **NEW** |
+| **§2.2 Ev#1** | **GPU 足以常驻"有价值的" working set** | **`v2/figs/exp_b_capacity_knee.png`（knee）+ working_set + cluster-scale 校正** | ✅ **NEW** |
+| §3.1 | Make PD-colocation great again | `PD_DISAGG_RESULTS`、`crossover_pd_advantage`、MB1/MB2、C2/C3 | ✅ |
+| §3.2 | Biased KV-aware global routing | LPWL（TTFT p90 −31%）、LMetric 乘性稀释、sticky hot-pin、ES ablation | ✅ |
+| §3.3 | GPU dedup via migration not replication | substrate net-positive（−18.6%/−36.6%）；correctness smoke tests | 🟡 substrate 通，policy e2e 待验证 |
+| §4 | 集成系统端到端 eval | 散落 mb1/mb2/mb5/crossover/lpwl，需统一 | 🚧 |
+| §5 | Related work（含 storage hierarchy 正面回应）| — | 写作 |
 
 ---
 
-## §1 Introduction
+## §1 Background and System Setup
 
-Agentic LLM workload —— 由 LLM 通过 tool call 自驱、多 turn 完成任务 —— 已经成为推理系统的主导负载，但现有为 chatbot 设计的调度策略在 agentic 下普遍失败。本文先刻画 agentic 与 chatbot 的本质区别，然后说明为什么三类主流调度都不够，最后给出 EAR 设计。
+### §1.1 LLM 与 KV cache
 
-**Contributions**:
+Transformer 自回归推理分两段：**prefill**（一次性算完 prompt 的全部 KV，compute-bound）与 **decode**（逐 token
+生成，memory-bandwidth-bound）。每个 token 的 KV 常驻 GPU HBM 才能被后续 attention 复用。Prefix caching（APC）让
+相同 prompt 前缀直接命中已算好的 KV，省掉重复 prefill —— 这是本文全部优化的物理基础。
 
-- **C1 Dispatch coupling 论证**：我们形式化一个 agentic workload 独有的反馈环 —— 单 turn 服务时间通过 Little's Law 隐式方程影响并发 session 数，从而把 per-request 延迟差放大成 throughput 差距。实测：load-balance baseline 在 600s trace 上跑出 **8x** wall-clock amplification；EAR 跑出 **TBDx**。
-- **C2 EAR 设计**：两个 pillar 的调度器 —— affinity-default routing 抓 intra-session locality，hot-instance 触发的 session migration 在 hotspot 出现时把整个 session 的 KV 搬到更轻的 instance，避免 hot pin。
-- **C3 评估**：在真实 Qwen3-Coder agentic trace 上，EAR 同时 dominate 5 个 baseline 的 TTFT、TPOT、APC、worst-worker p90、wall-clock 五个维度。
+> Qwen3-Coder-30B-A3B（GQA, 48 层, 4 KV heads, head_dim 128, bf16）：**KV = 96 KiB/token**，1 GiB = 10,923
+> token，block(16 tok) = 1.573 MB。
 
-**Figure 1: Teaser — wall-clock vs trace-time across schedulers** — `figs/f1_teaser.png` **🚧 TBD (NEW DATA NEEDED)**
-> Needs Phase 3 measurements: 5 baselines × 3 runs of trace replay, extract `amplification = wall_clock_s / trace_span_s` from each summary (Phase 1 patch already exposes the field). Plot as bar chart with y=1 reference line. EAR row 暂为 TBD（待 migration validation）。
+### §1.2 Agentic workflow
+
+Agentic 负载 = LLM 通过 tool-call 自驱、多 turn 完成任务。与 chatbot 的本质差异：每个 turn 由上一个 turn 的
+tool-call 结果触发（无人类 think-time），prefill-dominated（input/output ≈ 75×）。
+
+**但它是一个 mixture，不是"全多轮"**（C1, `figs/workload_chars/c1_session_mixture.png`）：
+
+- **90.3%** 的 session 是单轮（mean 1.62 turns）；但多轮 session（9.7%）= **44.2% 的请求**、**66.9% 的
+  prefill 质量**。
+- Continuation hazard（Lindy）：turn1→2 仅 **10.2%**，turn5→6 87%，turn12→13 **94.3%** —— heaviness 在
+  cold-start 几乎不可预测（corr(turn1_input, n_turns) = **0.04**）。
+
+> **Routing 含义（贯穿 §3.2）**：heaviness 在 session 起点不可预测 → 必须 **reactive**（观测累计负载），不能
+> proactive 预判；单轮海洋与深尾的最优策略相反（前者 load-balance、后者 affinity-pin），且 turn-1 无法区分 →
+> 唯一可行的策略是"人人 load-balanced 起步、随 turn 累积变 sticky" —— 正是 LPWL 的 emergent 行为。
+
+### §1.3 Serving agents in the wild
+
+- **PD colocation（8C）**：每个 instance 对称，prefill 与 decode 在同一 GPU 上由 chunked-prefill +
+  continuous batching 交错。弹性 KV 池，无静态分区。
+- **PD disaggregation**：把 instance 静态分成 prefill 池（P）与 decode 池（D），物理隔离两个阶段
+  （DistServe / Splitwise）。
+- **KV cache storage hierarchy**：GPU HBM → CPU DRAM → 远端 pooled store（RDMA/SSD，如 Mooncake Store /
+  LMCache）。把被淘汰/跨实例的 KV 下沉到更慢但更大的层，用传输换重算。
+
+> 三者各被本文一节回应：PD-colo 在 §3.1 被"复活"为默认；PD-disagg 在 §3.1 被证否（agentic regime）；storage
+> hierarchy 在 §2.2 被定量地"限位"（GPU 命中远胜下层，且活跃 working set 本就装得下）。
 
 ---
 
-## §2 Background and Workload Characterization
+## §2 GPU memory hit is the key to serving agents
 
-### §2.1 Agentic Workload Primer
+### §2.0 正确的 metric：request latency / TPS / GPU utilization（不是 TTFT/TPOT）
 
-Agentic workload 与 chatbot 的三个本质差异：
+**为什么不是 per-request TTFT/TPOT**：agentic 的 turn 之间有反馈环，单 turn 的延迟会跨 turn **复利**成 session
+端到端时间与系统吞吐差距。只有 **request/session latency、tokens-per-second、GPU utilization** 能 capture 这件事。
 
-- **Multi-turn, programmatic continuation**：每个 turn 由上一个 turn 的 tool-call 结果触发，没有人类 think-time
-- **Prefill-dominated**：input/output token ratio **75x**，98% 计算在 prefill 阶段（chatbot 为 1-10x）
-- **Skewed sessions**（来自 Qwen3 production trace，n=1.3M session / 2.1M req / 7200s）：top 1% 贡献 **46.5%** input token，top 5% **66.5%**，top 10% **74.6%**，top 25% **87.5%**，top 50% **96.0%** —— 半数 session 几乎占满全部 input mass
+**Dispatch coupling（降级为本节论证）**。每个 turn 间有外部 gap `T_external`（chatbot 是人在读/想/打字；agentic
+是 tool 执行）。Little's Law：`L = Λ · N · (W_turn(L) + T_external)`。当 `W_turn ≳ T_external` 时
+`W_turn(L)` 经 KV 竞争耦合到并发 L，scheduler 的 ε 退步被反馈环放大成 wall-clock 数倍差。
 
-平均 session 长度 TBD turn、TBD 输入 token；p99 单请求 KV 占用 **11.49 GiB**（H20 96GB HBM 的 12%）。
+production trace 实测 `T_external` CDF（`figs/f3a_inter_turn_gap.png`）：
 
-### §2.2 KV Cache Reuse Topology
+| | Agentic | Chatbot |
+|---|---:|---:|
+| p50 | **1.6 s** | 7.2 s |
+| gap < 1 s | **39%** | 4% |
+| gap < 5 s | 67% | 29% |
+| p99 | 738 s | 43 s |
 
-Trace 上 KV reuse 的分解：
+agentic 有一段 chatbot 没有的 **sub-second tool-call mass（39% vs 4%）**，几乎天然 `W_turn ≫ T_external` → 闭环。
+**实测**：lmetric 跑 600s trace 用 49 min wall-clock = **8× amplification**。**结论：per-turn 延迟的小差被放大成
+端到端数量级差 → 必须用 request latency / TPS / GPU util 衡量。**
 
-| Class | Share |
-|---|---|
-| Intra-session | **93.2%** |
-| Cross-session | 5.7% |
-| Shared prefix | 1.1% |
+- **TPS / GPU util** 的工具：`microbench/fresh_setup/bench_report.py`（TTFT/TPOT/E2E 全分位 + TPS +
+  per-worker GPU util）。PD stall 时 GPU ~0% util vs colo 34%（§3.1）即是 GPU-util 作为 metric 的直接体现。
 
-理论 APC 上界：any-session **80.3%**，intra-session-only **79.6%**，差距 <1pp。**cache 本质上是 session-local 的**；任何不保留 session affinity 的调度都丢掉绝大部分 reuse 机会。
+> **🚧 待补**：5 baseline × ≥3 runs 的 wall-clock amplification sweep（replayer 已输出 `amplification` 字段），
+> 钉死本节实证 closure。优先级高。
 
-**Figure 2: Workload characterization (3 panels)** — 现有数据可复用：
+### §2.1 KV$ hit is common and critical
 
-![F2a Reuse topology — intra 93.2% / cross 5.7% / shared 1.1%](figs/f2a_reuse_topology.png)
+Trace 上 KV reuse 的分解（`figs/f2a_reuse_topology.png`）：**intra-session 93.2% / cross-session 5.7% /
+shared-prefix 1.1%**。理论 APC 上界：intra-only **79.6%** vs any-session 80.3%，差 <1pp —— **cache 本质上是
+session-local 的**。
 
-![F2b Session input-token mass CDF — production trace top 1%/5%/10%/25%/50% = 46.5%/66.5%/74.6%/87.5%/96.0% (replay window overlaid for sanity)](figs/f2b_session_skew.png)
+per-turn 视角（C2, `figs/workload_chars/c2_work_amortization.png`）：resident context 11k→56k+ token 增长而
+new-prefill 从 2.7k 坍缩到 ~200 token，per-turn reuse 爬到 **99.6%**，resident/new（"PD tax"）到 turn 12 ≈
+250×、turn 30 ≈ 450×。**绝大部分 prefill 工作是可被命中省掉的**；命中与否直接决定 TTFT。
 
-![F2c KV footprint CDF — p99 = 11.8 GiB ≈ 12% of H20](figs/f2c_kv_footprint_cdf.png)
+### §2.2 Hits on GPU is more important than the CPU
 
-> 📝 Layout TBD：三张拼成 1×3 还是分散到 §2.1/§2.2/§2.4 各一张。
+**命中层级的代价是实测的，不是断言的**（Qwen3-Coder-30B-A3B / H20）。TTFT(s, p50) 服务一段长 L 的复用前缀，
+来自每个 KV 层：
 
-### §2.3 Dispatch Coupling — Why Locality Dominates
+![四层命中延迟：GPU < CPU-local < remote-RDMA-store ≪ miss，差距随 context 拉大](v2/figs/exp_a_tier_latency.png)
 
-这是本文最依赖直觉的论证，单独成节。
 
-**直觉**：chatbot 里每个 turn 后人要读、想、打字，**外部时钟**控制下一个 turn 何时到达；agentic 里 LLM 一拿到 tool-call 结果立刻发下一个 request，**系统自己的速度决定下一个 turn 何时到达**。所以一个慢策略不仅让单请求变慢，还让 session 在系统里停留更久 → 并发 session 更多 → KV 竞争更激烈 → 每个 turn 更慢 —— 反馈环。
+| prefix L | miss(recompute) | **remote RDMA store** | CPU-local(DRAM,PCIe) | GPU(HBM) | miss/RDMA | RDMA/CPU | CPU/GPU |
+|---:|---:|---:|---:|---:|---:|---:|---:|
+| 8k  | 0.588 | 0.151 | 0.076 | 0.053 | 3.9× | 2.0× | 1.5× |
+| 16k | 1.547 | 0.262 | 0.105 | 0.063 | 5.9× | 2.5× | 1.7× |
+| 32k | 4.604 | 0.680 | 0.158 | 0.080 | 6.8× | 4.3× | 2.0× |
+| **64k** | **15.23** | **0.97** | **0.27** | **0.11** | **15.8×** | **3.6×** | **2.4×** |
 
-**具体例子**：一个 coding agent 跑 20 turn 的任务。
+- **GPU hit ~flat**（42→111 ms / 1k→64k）：命中即整段前缀在 HBM，只重算最后一个 token。
+- **CPU-local hit** transfer-bound（PCIe H2D 实测 ~54 GB/s）；CPU-hit ≈ GPU-hit + KV/PCIe + ~0.15s 开销。
+  （native KV offload，命中经 `vllm:external_prefix_cache_hits` 100% 验证。）
+- **remote RDMA-store hit** = Mooncake-Store 机制（实测：两 instance，B 用 `do_remote_prefill` 经 RDMA 从 A 拉取
+  缓存前缀而非重算；`mb2_kv_transfer.py` / `v2/.../run_rdma.sh`）。对 recompute 是大赢（**最高 16×**，与 blog 的
+  46× 同向），但付 **NIC 税**（有效 ~5–7 GB/s，cf. MB2 raw ~9.7 GB/s；multi-NIC pooling 可抬高），故比 CPU-local
+  慢 3.6×、比 GPU 慢 ~9×（64k），**代价差随 context 拉大**。
+- **结论 —— 层级严格且随 context 拉大：`GPU < CPU-local < remote-RDMA-store ≪ miss`**。global KV store 确实
+  有用（这也是该路线存在的理由），但每靠近 GPU 一层就再省 1.4–4× TTFT。**最值钱的复用是 GPU-resident 的那种。**
 
-- 快策略：每 turn 2s，session 共 40s，平均并发 10 个 session
-- 慢策略（线性估算）：每 turn 3s，session 共 60s，应该并发 15 个
-- 慢策略（实际）：15 个并发让每 turn 被推到 4s，session 80s，并发 20 个，turn 再推到 5s …… 直到撞墙或落到一个远更糟的新平衡
+#### Evidence #1：GPU is sufficient to hold most KV requests
 
-对照 chatbot：每 turn 后人读 30s。turn 从 2s 变 3s，session 从 32s 变 33s，3% 差距，几乎无反馈。
+**realized APC 与 latency 在很小的 GPU 容量就饱和**（closed-loop 多轮负载，并发 4，扫 GPU KV 容量）：
 
-**形式化**。记 `Λ` = session 到达率（trace 给定），`N` = 每 session 的 turn 数，`W_turn(L)` = 单 turn 服务时间，是当前并发 session 数 `L` 的递增函数（并发越多、KV 竞争越激烈、`W_turn` 越大）。
+![容量 knee：APC 与 TTFT p90 在 3.6 GB（=活跃 working set）饱和，之后 dead-flat](v2/figs/exp_b_capacity_knee.png)
 
-Chatbot 的 Little's Law:
-```
-L = Λ · N · (W_turn(L) + T_human)
-```
-被大常数 `T_human` 主导，`W_turn(L)` 的扰动几乎不动 `L`。
+| GPU KV (GB) | realized APC | TTFT p90 |
+|---:|---:|---:|
+| 1.2 | 7.4% | 13.00 s |
+| 2.4 | 36.3% | 4.62 s |
+| **3.6** | **80.3%** | **0.53 s** |
+| 9.7 | 72.9% | 0.65 s |
+| 14.5| 72.9% | 0.65 s |
 
-Agentic 的 Little's Law（`T_human ≈ 0`）:
-```
-L = Λ · N · W_turn(L)
-```
-这是关于 `L` 的隐式方程。设策略变化让 `W_turn` 整体放大 `(1+ε)` 倍，小扰动分析得到：
-```
-dL/dε|_{ε=0} = L* / (1 − Λ · N · W'_turn(L*))
-```
-**分母接近 0** 时（系统接近 KV 饱和），放大系数发散。这就是为什么 lmetric 在 600s trace 上跑出 8x wall-clock 放大。
+**Knee 出现在 3.6 GB = 恰好 = 活跃 working set（4 session × 0.91 GB）**：APC 饱和到上界、TTFT p90 从 13.0s 坍缩
+到 0.53s，之后 dead-flat。**超过 working set 的 HBM 买不到额外收益；为追长尾而建的 CPU/storage tier 同理 ≈ 0。**
 
-**Figure 3: Dispatch coupling schematic** — `figs/f3_coupling_schematic.png` **🚧 TBD (CUSTOM DRAW)**
-> 需要新画一张示意图：上半 chatbot timeline（`system → T_human → system → T_human → ...`），下半 agentic timeline（`system → ε → system → ε → ...`），右侧叠一个反馈环箭头 `W_turn → Λ → L → W_turn`。适合用 TikZ / draw.io / matplotlib annotate。
+**Cluster-scale 校正（关键）**：working_set 分析（`analysis/working_set/`，`figs/working_set/`）显示"装下整个
+2h cluster 的全部 reuse 尾巴需 ~14 节点"，**这不构成 CPU offload 的动机** —— 那是用 1 个 replica 容量去装**整个
+cluster** 的 reuse；产出该 trace 的真实 cluster 远不止 14 节点（trace 是 cluster 聚合，见
+`project-trace-is-cluster-level`）。在 per-cluster 的 HBM 总量下，活跃 working set 本就 GPU-resident（live KV
+533–1157 GB ≪ 单节点 1528 GB）。**knee 位置随并发线性增长 = 随 cluster GPU 数增长**，而 cluster 本就提供了它。
 
-### §2.4 Takeaway
+> ⚠ **Scope**：本小节"装得下"指的是**活跃 working set 产生的近期高价值复用**，不是"全部 reuse 尾巴"。冷
+> session 长 gap 后回来的深尾命中（既低价值/byte 又贵于 fetch）正是 storage-hierarchy 派追的东西；本文论点是
+> 在 agentic 下这条尾巴不值得为之建深层级。§5 正面回应该派。
 
-三个性质 —— intra-session locality dominant (§2.2)、long context + prefill-heavy (§2.1)、dispatch coupling (§2.3) —— 共同决定了 agentic workload 的调度必须以 **locality 为主导**，并能容忍 skew 带来的 instance 间负载不均。
+### §2.3 Takeaway
+
+正确的 metric（§2.0）+ 命中集中在 GPU 才便宜（§2.2）+ 活跃 working set 装得下 HBM（Ev#1）⇒ **GPU-hit-first**：
+设计目标是最大化活跃 working set 的 **GPU 常驻 + 命中**，而非建深 CPU/storage hierarchy。§3 给出三个推论。
 
 ---
 
-## §3 Why Existing Schedulers Don't Fit
+## §3 Optimizing agent serving with the GPU-Hit-First Principle
 
-三类现有调度各自撞上 §2 三个性质中的一个：
+### §3.1 Make PD-colocation great again
 
-### §3.1 Load-balanced routing 丢 locality
+静态 PD-disaggregation 对 chatbot 有效，对 agentic 结构性失败 —— colocation 才应是默认。
 
-Round-robin 和 load-aware routing（如 LMetric, OSDI'26）最大化 instance 利用率，但忽略 session affinity。**实测 APC 跌到 56.9%**（vs 上界 79.6%），23pp 的差距直接来自丢失的 intra-session cache hit。违反 §2.2。
+**端到端证据**（`microbench/fresh_setup/PD_DISAGG_RESULTS.md`，8×H20，trace replay）：**没有任何静态 P/D 比能赢
+8-way colocation（8C）**，且失败模式随比例移动：
 
-### §3.2 静态 PD-disaggregation 撞 D 侧 KV 墙
+| Metric | **8C** | 6P+2D | 4P+4D | 2P+6D |
+|---|---:|---:|---:|---:|
+| completion | **100%** | 100% | 100% | **9%** 💀 |
+| wall-clock | **2994 s** | 3419 | 4171 | 5762 |
+| prefix-cache hit | **19.4%** | 0% | 0% | 0% |
+| TTFT p50 | **7.0 s** | 41.0 | 56.4 | 23.6 |
+| E2E p90 | **83.3 s** | 91.8 | 157.1 | 499 |
 
-静态把 instance 分成 P pool 和 D pool 对 chatbot 有效，对 agentic 失败：agentic 请求平均 33.6k token，需要 **3.3GB** KV；4D 方案下 p90 请求占 D 侧 KV pool **69%**，p99 直接 **溢出 138%**。结果：**TTFT p50 暴涨 62-72x**，成功率从 99.5% 跌至 **52-68%**。违反 §2.1（prefill-dominant + 长 context）。
+- **D-heavy（4P+4D）**：decode 池饱和 **97.5%**、prefill 池 ~30% —— 半个 cluster 的 KV 被困在错的一侧；agentic
+  请求大（p99 KV **11.5 GiB**），4D 让系统 decode 容量直接减半（24→12 并发，`figs/f2c_kv_footprint_cdf.png`、
+  `figs/f4b_pdsep_kv_wall.png`）。
+- **P-heavy（2P+6D）**：prefill 池 jam 99.7%，872 请求堆积，**91% 永不完成**。
+- **更聪明的路由救不了**（§6.x）：给 P 侧加 session-affinity 反而更差（4P+4D completion 100%→36%），GPU ~0%
+  util，cluster 卡在 KV-transfer 协调而非 compute —— 复现 producer hot-pinning。
 
-### §3.3 Pure session-sticky 的真正失败：全员被 hot session 拖累
+**为什么 colo 赢（正确论证，C2/C3 支撑）**：
 
-硬 session-instance 绑定恢复 locality（APC **77.2%**，达到上界 97%），但**绝对 worker latency** 全员被拖累 —— 是 pure sticky 的真正失败模式。
+- **时变 P:D 需求**：agentic 同时在 roofline 两侧有实质工作 —— compute-bound prefill（~30% 时间）+
+  memory-bound decode（**~70% 时间**，C3 token≠time 校正，`figs/workload_chars/c3_prefill_decode_balance.png`）。
+  colo 的弹性池吸收当下热的那一相；静态分区让 P-instance 带宽闲、D-instance 算力闲。
+- **resident KV 本地化**（C2）：下一 turn 的 prefix = [prevPrompt+prevAnswer] 横跨 P/D 两侧，disagg 必须
+  gather/transfer，colo 免费本地保留。
+- **transfer 不便宜且拓扑无关**（MB2，`figs/mb2_transfer_time_compare.png`）：Mooncake `batch_transfer_sync_write`
+  恒走 RDMA NIC（~9.7 GB/s），intra ≈ inter；PD-disagg 的 per-request transfer 税无法靠拓扑买回。
+- **phase-isolation 是 disagg 唯一的真赢面但被压倒**（MB1，`figs/mb1_interference.png`：32k prefill 让
+  per-stream TPOT 退化 52×，131k → 183×）—— 但被 D 侧容量天花板压倒（见上）。
+
+**边界（不 overclaim）**：crossover sweep（`analysis/crossover/`，`figs/crossover_pd_advantage.png`）给出 colo
+停止占优的 input 长度 —— colo 在 agentic 工作点赢，且我们知道边界在哪。
+
+### §3.2 Biased KV-cache-awareness in global routing
+
+GPU-hit-first 在路由层 = **把 cache-awareness 作为带偏置的独立路由路径**，而不是折叠进 load cost。
+
+**反例：load-balanced / 朴素 cache-aware-load 丢 locality**（`figs/f4a_apc_loss.png`）。LMetric（OSDI'26）打分
+`P = pending_prefill + (input − cache_hit)`，`score = P × num_requests` —— cache 只作 cost-model **减项**，而 score
+是**乘性**的，有 affinity 的 instance 因 num_requests 高被乘式吃掉 cache 收益：
+
+| 策略 | APC | vs load_only | cache 处理方式 |
+|---|---:|---:|---|
+| load_only | 53.9% | — | 无 |
+| LMetric | 57.2% | **+3.3pp** | cost-model 减项（被稀释）|
+| sticky | 77.7% | **+23.8pp** | 硬约束 |
+| unified | 78.7% | **+24.8pp** | 硬+软混合 |
+
+`load_only→LMetric` 的 +3.3pp 几乎可忽略；**+20.5pp 的回报来自把 cache 作独立路由路径**。
+
+**本文方法：LPWL（least-prefill-work，parameter-free）**（`project-lpwl-policy`、`analysis/lpwl_5policy_600s.md`）：
+按 `new_uncached ≈ input − cache_hit` 路由 —— `new_uncached≈input`（冷/新 session）自动按负载分散，`new_uncached≈0`
+（暖 session）自动 stick。**零旋钮**，在 600s trace 上击败 tuned unified+A+B（TTFT p90 **−31%**），full w600 上打平。
+这正是 C1 mixture 要的形状：无需 classifier 自动切分单轮海洋与深尾。
+
+**两条被否的杠杆（节省读者时间）**：
+
+- **real-time engine state 不是路由杠杆**（ES ablation，`project-es-ablation-sweep`）：只对一次性 placement 有用
+  （sticky −26%），对 per-req load-chasing 有害（load_only +27%）。
+- **derived-κ decode 项 net-negative**：decode-awareness 是错的杠杆。
+
+**Pure sticky 的失败用绝对 per-worker latency 衡量**（`figs/f4c_per_worker_ttft.png`，不能用 max/median 比值）：
 
 | | median worker TTFT p90 | max worker | system e2e p90 |
 |---|---:|---:|---:|
-| `sticky` | **20.3 s** | 55.4 s | **34.6 s** |
-| `unified` (affinity + LMetric fallback) | **10.3 s** | 37.7 s | **18.0 s** |
-| `lmetric` | 14.0 s | 31.3 s | 24.8 s |
+| sticky | **20.3 s** | 55.4 s | **34.6 s** |
+| unified | **10.3 s** | 37.7 s | **18.0 s** |
 
-机制：production trace 上 top 1% session 占 46.5% input mass、top 5% 占 66.5%，hot session 数量远大于 instance 数（8）；sticky 的 hash 绑定让 **每个 worker 都自己承接一份 hot session**，median worker 也被拖慢到 20s 量级。unified 用 LMetric fallback 把 cold/new session 重路由到非 hot worker，保留 7/8 worker 的速度。系统 p90 由大多数请求决定，所以 unified 在 e2e p90 上 ~2x 快于 sticky。
+hot session 数 ≫ instance 数（8），sticky 的 hash 绑定让**每个 worker 都自接一份 hot session**，median 也被拖慢；
+biased 路由把 cold/new 重路由到非 hot worker，保 7/8 worker 速度 → e2e p90 ~2× 快。**这引出 §3.3：sticky 的残余
+hot-pin 需要 migration 解。**
 
-**§3.3 sub-finding**：hot pin failure 必须用 **per-worker absolute latency**（median + max）衡量，**不能用 normalized ratio**。`max/median` 在 unified 这样的"affinity + escape"方案上会反向惩罚 —— sticky 的 ratio 2.73 比 unified 的 3.67 低，但 sticky 的 median 也高（20.3s vs unified 10.3s），ratio 越低反而越糟。本文 paper 中所有 worker 平衡相关的比较一律用 (median, max) 双指标，不用单一比值。
+### §3.3 GPU KV-cache dedup with migration instead of replication
 
-违反 §2.4 的 skew 容忍要求。
+**视角**：多个 instance 各自缓存同一段 prefix = GPU 容量被 **replication** 浪费；GPU-hit-first 要求**全局只留一份 +
+把 session 搬到那一份**（migration/dedup），既保 GPU 命中又均衡负载、修复 §3.2 的残余 hot-pin。
 
-**Figure 4: Three baselines, three failure modes** — 拆成三个子图，分别放在 §3.1/§3.2/§3.3：
+- **Trigger**：host 的 `pending_prefill > T_hot` 且 session 在 `T_cool` 内未迁移过。
+- **Target**：用 **observable pending prefill tokens** 选最轻 instance，**不用** cost-model 预测（mooncake cost
+  model 误差 10–21×，by construction 绕过）。
+- **Mechanism**：当前 request 重定向到 target，session KV 经 Mooncake `kv_connector` 迁移；binding 更新；后续
+  turn 按 affinity 路由到新 host。迁移成本在该 session 剩余 turn 上摊销。
+- **Thrashing prevention**：per-session cooldown `T_cool`。
 
-§3.1 — APC 实测 vs 理论上界 79.6% (lmetric 56.9%, load_only 54.1%, sticky 77.2%, unified 79.4%)：
-![F4a APC loss](figs/f4a_apc_loss.png)
-
-§3.2 — D 侧 KV pool 占用 vs per-request KV footprint，4P+4D 和 6P+2D 在 agentic regime 都穿过 90% 内存墙：
-![F4b PD-sep KV memory wall](figs/f4b_pdsep_kv_wall.png)
-
-§3.3 — Per-worker TTFT p90 across 8 instances × 5 policies。sticky 的所有 worker 都被拖慢（median 20.3s），unified 把伤害集中在 e4 上、其他 worker 快（median 10.3s）：
-![F4c Per-worker TTFT p90 distribution](figs/f4c_per_worker_ttft.png)
-
-
-> 📝 可选支撑图 — Prefill-decode 干扰（同 GPU 8k prefill 让 TPOT 退化 66x），放 §3.3 支撑 sticky 的 interference 论证：
-![F4d PD interference](figs/f4d_pd_interference.png)
-
-### §3.4 Takeaway
-
-**问题不是任何单一 baseline 太弱，而是没有一个方案同时满足 §2 的三个性质**：保留 locality、尊重 D 侧 KV 容量、容忍 skew 带来的负载不均。EAR 是据我们所知第一个三件事同时做到的调度器。
+> **状态**：**substrate 已验证 net-positive**（`kv_both` connector 在 trace replay 上 TTFT p90 **−18.6%**，
+> DR-fix 后 **−36.6%** vs plain；之前认为是 blocker 的 transfer overhead 已不存在，4 次历史 revert 的主因消失）。
+> migration correctness smoke tests 通过。**policy 层 e2e（trigger + target 在反馈环里的真实收益）仍未直接验证**
+> —— 这是全文最弱的一环，独立风险是"决策错误 + cooldown thrashing"。affinity-only pillar（§3.2 LPWL）已独立成立，
+> 即便 migration 仍 marginal，paper 也有 strong-routing 主线。
 
 ---
 
-## §4 Design: EAR
+## §4 System Evaluation
 
-### §4.1 Architecture
+> **🚧 关键缺口**：目前证据散落在 mb1/mb2/mb5/crossover/lpwl/v2；§4 需要一个**集成系统**（colocation +
+> biased routing + dedup-migration，统一命名）跑端到端、用 §2.0 的新 metric（request latency / TPS / GPU util）
+> 评测，并把 §3.1/§3.2/§3.3 做成 ablation。
 
-EAR 是位于 N 个同质 instance 之上的 router。每个 instance 是对称的 PD-colocated，没有静态 P/D 分区。每个 session 在 router 内维护一个 **host binding** —— 当前持有该 session KV 状态的 instance。Binding 在常态下稳定，仅在 hotspot 触发时通过 migration 改变。
+### §4.1 Setup
+- **Trace**：真实 Qwen3-Coder agentic trace（cluster-level，见 `project-trace-is-cluster-level`）；日常迭代
+  `w600_r0.0015_st30`（~850 req / ~13 min / peak QPS ~1.6 / APC headroom ~76%）。
+- **Hardware**：8× H20 (96 GB)；vLLM 0.18.1 (V1, chunked-prefill) + Mooncake。
+- **Baselines**：① round-robin ② LMetric (OSDI'26) ③ kvcache-aware+load 线性混合 ④ sticky ⑤ static PD-disagg
+  (4P/4D) ⑥ **本文系统**（colo + LPWL + dedup-migration）。
+- **Metrics**：request latency (mean/p50/p90/p99)、**TPS**、**GPU util (median/max worker)**、APC、
+  wall-clock amplification（不单看 TTFT/TPOT，§2.0）。
 
-**Figure 5: EAR architecture and request flow** — `figs/f5_architecture.png` **🚧 TBD (CUSTOM DRAW)**
-> 组件图：router (含 session→host table) → N 个 symmetric instances；affinity 路径实线，migration path 虚线。适合 TikZ / draw.io。
+### §4.2 End-to-end
+- `figs/f6_e2e_latency_bars.png` / `f6_e2e_latency_full_grid.png`（现有 4–5 baseline；🚧 补 static PD-disagg
+  列 + 本文系统列）。
 
-### §4.2 Pillar 1: Affinity-Default Routing
+### §4.3 Ablation（GPU-hit-first 三推论各关一个）
+- −colocation（→ static PD-disagg，§3.1）
+- −biased routing（→ load-balance / LMetric，§3.2）
+- −dedup-migration（→ pure sticky，§3.3）
+- 🚧 migration-only / full 待 policy e2e 验证。
 
-- **Cold start**：新 session 到达时，router 用 load-balance（选 pending prefill tokens 最少的 instance）分配初始 host
-- **Warm path**：已建立 session 的后续每个 turn 一律路由到当前 host
-- **效果**：intra-session KV reuse 被构造性保留，APC 接近 §2.2 的上界 79.6%
+### §4.4 Microbench 支撑（已有，复用）
+- §2.2 四层命中：`v2/figs/exp_a_tier_latency.png`
+- §2.2 容量 knee：`v2/figs/exp_b_capacity_knee.png`
+- §3.1 PD-disagg 失败：`figs/f4b`、`PD_DISAGG_RESULTS`、`crossover_pd_advantage`
+- transfer cost：`figs/mb2_*`；phase interference：`figs/mb1_interference.png`
 
-### §4.3 Pillar 2: Hot-Triggered Session Migration 🚧 PARTIAL VALIDATION
-
-避免 Pillar 1 退化成 pure sticky 的关键 mechanism。
-
-> **状态（2026-05-27 更新）**：
-> - **Substrate 验证 PASS**（commit `ef9e010`）：`kv_both` connector 在 trace replay 上 net positive（TTFT p90 −18.6%），DR-fix 后再 −22%。之前认为是 migration blocker 的 transfer overhead 已不存在。
-> - **策略层 e2e 验证 PENDING**：trigger 阈值 + target selection 在 agentic 反馈环里的真实收益仍未直接测。之前 4 次 migration 尝试（`6b255fa`, `e991960/5772149`, `cc6e562`, `4c583f2`）被还原的主因（substrate overhead）已消失，但 trigger 决策错误 + cooldown thrashing 是独立风险，需新一轮 e2e 实验确认。
-
-#### §4.3.1 Trigger signal
-
-EAR 实时监控每个 instance 的 **pending prefill tokens**。新 request 到达且按 affinity 应该路由到 host H 时，router 先检查：
-- `H.pending_prefill > T_hot`？（hotspot 检测）
-- session 在过去 `T_cool` 秒内未发生过 migration？（thrashing prevention，§4.3.4）
-
-两个条件同时满足才考虑触发 migration。`T_hot` 和 `T_cool` 的取值见 §5.5 sensitivity。
-
-#### §4.3.2 Target selection
-
-候选集：所有 instance 中 (a) 剩余 KV 容量能装下 session 现有 context、(b) `pending_prefill` 严格小于 H 的。选 `pending_prefill` 最低者。
-
-**关键设计点**：我们用 **observable current load** 而不是 **predicted transfer time** 排序。文献和 colleague 数据均显示 mooncake cost model 的预测误差达 10-21x；而 pending prefill tokens 是 router 直接观察到的数值，accuracy by construction。
-
-若候选集为空（所有其他 instance 都装不下，或都比 H 更忙），EAR 保留当前 binding，继续在 H 上处理请求 —— **migration 是 opportunistic，不是 mandatory**。
-
-#### §4.3.3 Mechanism
-
-Migration 触发时：
-
-1. 当前 request 直接重定向到 target instance T
-2. session 累计的 KV 状态从 source H 通过 Mooncake `kv_connector` 传输到 T
-3. session 的 host binding 更新为 T；后续 turn 按 affinity 自动路由到 T
-
-KV transfer 发生在触发该 migration 的 request 的 critical path 上，但被该 session 剩余的 TBD turn 摊销。
-
-#### §4.3.4 Thrashing prevention
-
-每个 session 维护 `last_migration_timestamp`。在 cooldown `T_cool` 内被禁止再次 migrate。Cooldown 把 migration 行为限制在 O(session_lifetime / T_cool) 量级。
-
-### §4.4 Implementation
-
-基于 vLLM 0.18.1 + Mooncake (vanilla kv_connector)。EAR 是一个 router 进程，~TBD LoC。Session→host 表用 TBD（in-memory dict / Redis）维护。
+### §4.5 Sensitivity
+- 到达率 λ / skew(Zipf α) / KV pool size（不依赖 migration，可做）；`T_hot`/`T_cool`（依赖 migration，deferred）。
 
 ---
 
-## §5 Evaluation
+## §5 Related Work
 
-### §5.1 Setup
-
-- **Trace**: 真实 Qwen3-Coder agentic trace，TBD requests / TBD seconds / r=0.0015 st=0.3，peak QPS ~1.6，APC headroom ~76%
-- **Hardware**: TBD × H20 (96GB HBM)
-- **Engine**: vLLM 0.18.1 + Mooncake `kv_connector`
-- **Baselines** (6 个):
-  1. `load-balance` —— round-robin
-  2. `LMetric` —— OSDI'26 load-aware routing
-  3. `kvcache-aware + load-balance` —— linear combination of cache score and load score
-  4. `sticky` —— 硬 session-instance pinning
-  5. `static PD-disagg` —— 4P / 4D 静态分区
-  6. `EAR` —— 本文
-- **Metrics**: TTFT (mean/p50/p90/p99)、TPOT (同上)、E2E、APC、worker TTFT p90 (median + max)、wall-clock vs trace-time
-
-### §5.2 End-to-end Performance
-
-**Figure 6: End-to-end performance** — ✅ (PARTIAL，缺 PD-disagg 列)
-
-![F6 E2E latency bars — 5 policies](figs/f6_e2e_latency_bars.png)
-
-> **🚧 TBD (NEW DATA)**：上图缺 `static PD-disagg` 那一列；EAR 列也是 TBD（需 migration validation）。要再补一张同样格式但包含全 6 个 baseline 的图。
-
-| Scheduler | TTFT p50 | TTFT p90 | TPOT p90 | APC | Hotspot idx | Wall-clock factor |
-|---|---|---|---|---|---|---|
-| load-balance | TBD | TBD | TBD | TBD | TBD | TBD |
-| LMetric | TBD | TBD | TBD | 56.9% | 6.53 | ~8x |
-| kvcache+load | TBD | TBD | TBD | TBD | TBD | TBD |
-| sticky | TBD | 18.02s | TBD | 77.2% | 13.65 | TBD |
-| static PD-disagg | 62.8s | TBD | TBD | TBD | TBD | TBD |
-| **EAR** | TBD | **7.35s** | TBD | **79.4%** | TBD | TBD |
-
-(粗体数字来自现有 "unified" 原型测量。)
-
-### §5.3 Ablation 🚧 PARTIAL DEFER
-
-我们独立关闭两个 pillar:
-
-- **EAR (affinity only)**: 等价于 pure sticky；衡量 locality 单独贡献
-- **EAR (migration only)**: cold-balance + reactive migration，无 affinity；衡量 migration 能否独立成立
-- **EAR (full)**: 两个 pillar 都开
-
-**Figure 7: Ablation** — `figs/f7_ablation.png` **🚧 TBD — DEFERRED (BLOCKED ON MIGRATION VALIDATION)**
-> 完整 ablation 需要 migration-only / both / affinity-only 三个配置。Migration-only 和 both 都依赖 migration 重测。现阶段可先做 affinity-only vs load-balance 的两点对比（已有数据：unified 79.4% APC vs lmetric 56.9% APC）。
-
-预期结论：affinity-only 拿到 locality 但 interference 翻倍；migration-only 抓不住 locality；两者都必须。
-
-### §5.4 Dispatch Coupling Validation
-
-闭环 §2.3 的论证。对每个 baseline 测量：
-
-- 单 turn 平均服务时间 `W_turn`（x 轴）
-- Wall-clock / trace-time amplification（y 轴）
-
-**Figure 8: Wall-clock amplification vs per-turn service time** — `figs/f8_coupling_measured.png` **🚧 TBD (NEW DATA)**
-> 散点：x = 平均 per-turn `W_turn`（从 per-request metrics 算 TTFT + decode_time），y = amplification (`wall_clock / trace_span`，Phase 1 patch 已暴露)。每个 baseline 一个点。理论曲线 `L*/(1 − Λ·N·W'(L*))` 叠加（可选）。这是 §2.3 论证的实证 closure，**优先级最高**。
-
-预期：EAR 在 `W_turn` 最小且放大系数最低的角上。
-
-### §5.5 Sensitivity
-
-| 参数 | 范围 | 检验 |
-|---|---|---|
-| 到达率 λ | TBD | EAR 在低/高负载下是否稳定 dominate |
-| Skew 程度 (Zipf α) | TBD | sticky 与 EAR 的差距是否随 skew 拉开 |
-| KV pool size | TBD | static PD-disagg 撞墙边界 |
-| `T_hot` (migration threshold) | TBD | 触发太宽 → thrash，太严 → 错过 |
-| `T_cool` (cooldown) | TBD | 同上 |
-
-**Figure 9: Sensitivity heatmaps** — `figs/f9_sensitivity.png` **🚧 TBD (NEW DATA, PARTIAL DEFER)**
-> Arrival rate / skew / KV pool size 这三轴可现在做（不依赖 migration）；`T_hot` / `T_cool` 两轴依赖 migration validation，deferred。
-
-### §5.6 Migration Microbenchmark 🚧 FULL DEFER
-
-刻画 EAR 内部 migration 行为：
-
-- Migration 触发率（% of requests）
-- 平均 KV transfer 时间
-- Migration accuracy：迁移后 target instance 在接下来 TBD 个 turn 内保持非 hot 的比例
-- Thrashing rate：cooldown 窗口内多次迁移的 session 占比（应为 0）
-
-**Figure 10: Migration timeline** — `figs/f10_migration_timeline.png` **🚧 TBD — DEFERRED (BLOCKED ON MIGRATION VALIDATION)**
-> 时间轴上每个 instance 的 pending prefill tokens heatmap，migration 事件以箭头标出。完全依赖 migration 重测。
+- **Serving systems**：vLLM, Mooncake, SGLang, DistServe, Splitwise。本文基于 vLLM+Mooncake，与
+  DistServe/Splitwise 不同在于**不做静态 P/D 分区**（§3.1 给出 agentic regime 的证否 + crossover 边界）。
+- **Cache-aware routing**：LMCache, Production-Stack, LMetric (OSDI'26)。本文指出 cache 信号不能折叠进乘性 load
+  score（§3.2 LMetric 稀释分析），须作独立 biased 路由路径。
+- **KV storage hierarchy / offload（主要对手，正面回应）**：Mooncake Store, LMCache, AttentionStore 等把 KV 下沉
+  到 CPU/SSD/远端 pool。**本文不否认其对 recompute 的收益**（§2.2 实测 remote-RDMA 命中比 recompute 快达 16×，
+  与 Mooncake-Store blog 的 46× 同向）；但论证在 **agentic** 下 (i) 命中层级 GPU ≫ CPU ≫ RDMA-store（§2.2），
+  (ii) 活跃 working set 本就 GPU-resident（§2.2 Ev#1）⇒ 深层级的边际收益低，应优先 GPU 常驻而非建深 hierarchy。
+- **Stateful migration**：Pollux, Gandiva（RL training 的 migration-as-rebalancing）。本文把该思路迁到 LLM KV
+  cache 的 dedup（§3.3）。
 
 ---
 
-## §6 Discussion and Limitations
+## §6 Conclusion
 
-- **Extreme skew**: 若单个 session 自己就把任意 instance 撑成 hot，EAR 退化为 sticky。我们未在该 regime 做 stress test。
-- **Cost model accuracy**: EAR 用 observable load 绕过了预测误差问题。但未来若引入 predictive admission control，需要解决 mooncake cost model 10-21x 误差。
-- **Heterogeneous hardware / multi-model**: EAR 假设 instance 同质。混合模型 / 混合 GPU 池需要扩展 binding 模型。
-- **Per-instance batch tuning (future)**: 动态调整 `max_batched_tokens` 可能进一步降低 instance 内部 prefill-decode 干扰，留作 future work。
-
----
-
-## §7 Related Work
-
-- **LLM serving systems**: vLLM, Mooncake, SGLang, DistServe, Splitwise. EAR 基于 vLLM + Mooncake 实现，与 DistServe/Splitwise 不同之处在于不做静态 P/D 分区。
-- **Cache-aware routing**: LMCache, Production-Stack, LMetric (OSDI'26)。这些工作最小化 cross-instance cache miss，但不迁移状态。
-- **Stateful service migration**: Pollux, Gandiva (RL training)。EAR 借鉴 migration-as-rebalancing 思路，将其迁移到 LLM inference 的 KV cache 场景。
-
----
-
-## §8 Conclusion
-
-对 agentic LLM workload，locality 是主导调度杠杆。EAR 用 session-affinity routing 抓住它，用 hot-triggered session migration 保护它，单一方案在 TTFT、APC、worst-worker p90、wall-clock throughput 四个维度同时 dominate 五个 baseline。
+对 agentic LLM 负载，用户感受的 request latency / TPS / GPU util 由 **KV 命中是否在 GPU HBM 上** 主导。命中层级
+`GPU ≫ CPU > RDMA-store ≫ recompute` 且代价差随 context 拉大（§2.2），而活跃 working set 小到本就 GPU-resident
+（§2.2 Ev#1）。**GPU-hit-first** 原则由此统一三件事：复活 PD-colocation（§3.1）、在路由里做 biased
+KV-awareness（§3.2）、用 migration 而非 replication 做 GPU 去重（§3.3）。
 
 ---
 
 ## Work Plan
 
 ### ✅ Done
+- §1/§2 背景与 metric 论证；dispatch-coupling 数学；inter-turn gap CDF（`f3a`）
+- §2.1 reuse topology / C1–C3（`f2a`、`workload_chars/*`）
+- **§2.2 四层命中实测（`v2/exp_a_tier_latency`：GPU/CPU/RDMA/miss）**
+- **§2.2 Ev#1 容量 knee（`v2/exp_b_capacity_knee`）+ working_set + cluster-scale 校正**
+- §3.1 PD-disagg 证否（`PD_DISAGG_RESULTS`、crossover、MB1/MB2）
+- §3.2 LPWL（−31%）、LMetric 稀释、sticky hot-pin、ES ablation
+- §3.3 migration substrate net-positive + correctness smoke tests
 
-- [x] §1 anchor sentence + contribution bullets
-- [x] §2 outline + reuse existing characterization figures (`f2a`/`f2b`/`f2c`)
-- [x] §3.1/§3.2/§3.3 outline + reuse existing baseline failure figures (`f4a`/`f4b`/`f4c`/`f4d`)
-- [x] §4 design description (§4.3 待实证)
-- [x] §5.2 partial figure (`f6` 5/6 baselines)
-- [x] `replayer/replay.py` patched to emit `trace_span_s` + `amplification` in summary
+### 🟢 不依赖 migration，现在可做
+- §2.0 wall-clock amplification sweep（5 baseline × ≥3 runs）— **优先级最高**
+- §4 集成系统命名 + 端到端 baseline 矩阵（含 static PD-disagg 列）
+- §4.5 λ / skew / KV pool sensitivity
+- 草拟 §1–§3 正文（证据/图已齐）
 
-### 🟢 Can do without migration (paper writing now possible)
+### 🚧 Deferred（待 migration policy e2e）
+- §3.3 migration trigger + target 的反馈环收益验证
+- §4.3 full / migration-only ablation
+- §4.5 `T_hot` / `T_cool` sensitivity
 
-- [ ] Draft §1-§4 正文（数据全有，figures 已 copy 完）
-- [ ] §2.3 dispatch coupling 那一节的正文 draft（数学已经在 conversation 里推完）
-- [ ] §3 三个失败模式正文 draft
-- [ ] §5.4 wall-clock amplification 实测（5 baseline × ≥3 runs）— **优先级最高**，这是 §2.3 的实证 closure
-- [ ] §5.2 把 static PD-disagg 补进 `f6` 那张图（重跑或合并现有 PD-sep 数据）
-- [ ] §5.5 sensitivity 的 λ / skew / KV pool 三轴
-- [ ] §3 三张子图各自独立的 latex/markdown layout 决定
+### 🎨 待画
+- §1.3 storage-hierarchy 示意（GPU HBM → CPU DRAM → RDMA store）
+- §2.0 dispatch-coupling schematic（chatbot vs agentic timeline + 反馈环）
+- 集成系统 architecture 图
 
-### 🚧 Deferred (待 migration validation)
-
-- [ ] §4.3 migration mechanism e2e 验证：substrate 已通（commit `ef9e010`），缺 trigger + target selection 的策略层实验
-- [ ] §5.3 full ablation (migration-only + both 两个配置)
-- [ ] §5.5 `T_hot` / `T_cool` 两轴 sensitivity
-- [ ] §5.6 migration microbench 全部
-- [ ] §1 teaser 图 (`f1`) EAR 那一列
-- [ ] §5.2 表里 EAR 那一行
-- [ ] §4.3.1 / §4.3.4 的 `T_hot` 和 `T_cool` 取值
-
-### 🎨 Custom drawings (paper-writing 阶段)
-
-- [ ] `f3_coupling_schematic.png` —— chatbot vs agentic timeline + 反馈环
-- [ ] `f5_architecture.png` —— EAR 组件图
-
-### ❓ Open design decisions
-
-- [ ] §4.4 session→host 表的存储介质（in-memory dict vs Redis）
-- [ ] §5.1 instance 数量、trace 总长度的最终定稿
+### ❓ Open
+- 集成系统最终命名（GPU-hit-first 是原则；系统名待定）
+- §4 instance 数 / trace 总长定稿

README.md

@@ -0,0 +1,77 @@
+# agentic-kv
+
+Serving agentic LLM workloads by keeping the KV working set in GPU HBM
+(GPU-hit-first). Research outline: [`PAPER_OUTLINE.md`](PAPER_OUTLINE.md).
+Evidence + experiments: [`v2/`](v2/).
+
+---
+
+## ⚠️ Benchmarking methodology — read this first
+
+> **Replay agentic traces with `--dispatch-mode thinktime`, not the default
+> `tracets`.** It is the faithful, more realistic load — and the dispatch mode
+> materially changes the performance you measure.
+
+The replayer offers two ways to time each turn:
+
+| mode | turn-k dispatched at | what it models |
+|---|---|---|
+| `tracets` (default) | `max(prev_turn_finished, trace_ts)` | absolute production schedule |
+| **`thinktime` (use this)** | `prev_turn_finished + time_to_parent_chat` | real closed-loop agent pacing |
+
+**Why it matters.** `tracets` collapses the inter-turn think-time to ~0 whenever
+the system falls behind (it fires the next turn immediately because the trace
+timestamp is already in the past). That manufactures **artificial request
+bursts** — spiking instantaneous concurrency → KV-pool pressure → preemption →
+inflated tail latency and wasted throughput. `thinktime` keeps each turn's real
+gap (tool-exec + agent think), so the offered load is what a real agent produces.
+
+**Measured (w600 first-300s window, 8×H20, round-robin, 100% completion):**
+
+| metric (N=8) | `tracets` (Mode 1) | **`thinktime` (Mode 2)** | Δ |
+|---|---:|---:|---:|
+| E2E p90 | 102.8 s | **73.5 s** | **−28%** |
+| E2E p99 | 245 s | **227 s** | −7% |
+| TTFT p90 | 56.1 s | **39.7 s** | **−29%** |
+| system TPS | 111.8 | **119.3** | **+7%** |
+| wall-clock | 967 s | **787 s** | −19% |
+| TPOT p90 | 0.174 s | 0.188 s | ~flat |
+
+So under realistic capacity, `tracets` makes the system look **~30% worse on
+tail latency** than it actually is. Tell-tale: scaling 6→8 instances barely helped
+`tracets` (975→967 s — its bursts re-saturate regardless of capacity) but helped
+`thinktime` a lot (1125→787 s). Under heavy saturation (N=6) the two converge
+(E2E p90 ≈ 118–120 s), since there is no slack for bursts to harm. Decode (TPOT)
+is dispatch-independent everywhere.
+
+**Recommendation:** benchmark with `--dispatch-mode thinktime`; use `tracets`
+only as an explicit bursty stress case. Full ablation:
+[`v2/exp_c_dispatch_ablation/`](v2/exp_c_dispatch_ablation/).
+
+### How to use it
+
+```bash
+# 1. annotate a trace with the real per-turn gap (one-time; scans the raw trace)
+python scripts/add_time_to_parent.py traces/w600_r0.0015_st30.jsonl traces/w600_ttp.jsonl
+
+# 2. replay closed-loop with faithful think-time
+python -m replayer --trace traces/w600_ttp.jsonl --endpoint <eps> \
+    --model <model> --dispatch-mode thinktime
+```
+
+`time_to_parent_chat = this_turn.request_ready_time_ms − parent_turn.request_end_time_ms`,
+computed from the raw trace and stored per request; turn-1 has none (fires at its
+trace arrival). Traces without the field fall back to `tracets`.
+
+---
+
+## Project map
+
+- [`PAPER_OUTLINE.md`](PAPER_OUTLINE.md) — GPU-hit-first paper outline (the thesis).
+- [`v2/`](v2/) — evidence experiments:
+  - `exp_a_tier_latency/` — KV-hit cost by tier (GPU < CPU-local < remote-RDMA < miss).
+  - `exp_b_capacity_knee/` — realized APC / latency knee vs GPU capacity.
+  - `exp_c_dispatch_ablation/` — the replay-mode study above.
+- `replayer/` — trace replayer (`--dispatch-mode`, closed-loop think-time).
+- `scripts/add_time_to_parent.py` — trace annotation for `thinktime`.
+- `microbench/`, `analysis/` — PD-disagg, routing, workload characterization.

RESULTS_SUMMARY.md

@@ -0,0 +1,134 @@
+# 目前已成立的结论（2026-05-27）
+
+EAR 项目目前能用实测数据支撑的论点汇总。每条都标了对应的图/数据路径。
+
+---
+
+## 1. Workload 性质（§2）
+
+Production trace = Qwen3-Coder agentic，1.3 M sessions / 2.1 M reqs / 7200 s。
+
+| 性质 | 数据 | 实证 |
+|---|---|---|
+| **KV 复用几乎全在 session 内** | intra 93.2% / cross 5.7% / shared 1.1%；理论 APC 上界 79.6% | `figs/f2a_reuse_topology.png` |
+| **Session 极度偏斜** | top 1%/5%/10%/25%/50% = 46.5%/66.5%/74.6%/87.5%/**96.0%** input mass | `figs/f2b_session_skew.png` |
+| **单请求 KV 已经很大** | p50 1.8 GiB / p90 8.0 / p95 9.6 / **p99 11.5 GiB**；KV pool 38 GiB/instance（0.4 × H20 96 GiB）→ p99 req 只能装 **3 个/instance** | `figs/f2c_kv_footprint_cdf.png` |
+
+**结论**：cache 是 session-local 的，scheduling 必须保留 session affinity；单 request KV 接近 pool 上限，**PD-disagg 4P+4D 让系统 decode 容量直接减半**。
+
+## 2. Dispatch Coupling（§2.3）
+
+| 数据 | Agentic (Qwen3-Coder) | Chatbot (qwen3-max) |
+|---|---:|---:|
+| Inter-turn `T_external` p50 | **1.6 s** | 7.2 s |
+| `gap < 1 s` 比例 | **39%** | 4% |
+| `gap < 5 s` 比例 | 67% | 29% |
+| p99 | 738 s | 43 s |
+
+参考图：`figs/f3a_inter_turn_gap.png`。
+
+**结论**：agentic 有一段 chatbot 没有的 **sub-second tool-call mode**（39% vs 4%）。当 `W_turn ≳ T_external`（任何 W_turn > 1 s 的 scheduler 在 agentic 上都满足这条件），Little's Law `L = Λ · N · (W_turn(L) + T_external)` 进入闭环 regime，scheduler 的 ε 退步通过 KV 竞争反馈环被放大成 wall-clock 数倍差距。**实测**：lmetric 跑 600 s trace 用 49 min wall-clock = **8x amplification**。
+
+## 3. 现有调度的三类失败（§3）
+
+| Baseline | 失败模式 | 数据 |
+|---|---|---|
+| **load-balance / LMetric** | 丢 locality | lmetric APC **56.9%**（vs 上界 79.6%）；LMetric 比 load_only 只好 +3.3pp，因为 cache 信号在乘性 score `(pending+input−hit) × num_req` 里被 num_req 吞掉 |
+| **静态 PD-disagg** | D 侧 KV 容量墙 + transfer 成本 | 见 §4 cost-vs-benefit |
+| **Pure sticky** | 全员被 hot session 拖累，不是单一热点 | sticky median worker 20.3 s vs unified 10.3 s；system e2e p90 sticky 34.6 s vs unified 18.0 s（**用 max/median ratio 衡量是误导**，§3.3 用 absolute per-worker latency）|
+
+参考图：`figs/f4a_apc_loss.png`、`figs/f4b_pdsep_kv_wall.png`、`figs/f4c_per_worker_ttft.png`、`figs/f6_e2e_latency_bars.png`、`figs/f6_e2e_latency_full_grid.png`。
+
+## 4. Static PD-disagg 为什么失败（§3.2）—— 容量问题，不是 cost-benefit 问题
+
+⚠ **2026-05-27 纠正**：本节前一版本论证"PD-disagg 因为 transfer cost > phase isolation benefit 而失败"。**这个论证算错了**。正确的 phase-isolation benefit 是**每个 prefill 事件 × D 个 concurrent stream** 的总和（≈ `D × T_prefill`），不是单个 request 的 decode 时长。用正确公式，PD-disagg 在 phase-isolation 这一维上**赢 colo 一两个数量级**。Static PD-disagg 在 agentic 上失败的**真正根因是 D 侧 KV pool 容量**，不是这一维。
+
+### 4.1 真正的失败模式：D 侧 KV 容量天花板
+
+| | 8C colo | 4P+4D PD-disagg |
+|---|---:|---:|
+| Per-D-instance KV pool（0.4 × 96 GiB） | 38 GiB | 38 GiB |
+| 系统总 decode 容量（D 实例数 × 单池） | 8 × 38 = **304 GiB** | 4 × 38 = **152 GiB** |
+| p99 单请求 KV = 11.5 GiB → 最多并发 decode | 24 | **12（减半）** |
+
+Colleague 4P+4D 实测：TTFT p50 0.91 s → **62.8 s（62×）**、success rate **99.5% → 52%**。失败模式：**D 池溢出 + 排队**，不是 transfer 延迟。
+
+参考图：`figs/f4b_pdsep_kv_wall.png`（pdf 版本是高质量 paper figure）。
+
+### 4.2 MB2 — KV transfer cost（per-request 一次性成本，**不**是 dominant cost）
+
+dash1 GPU 0+1（intra）和 dash1 ↔ dash2（inter, 200 Gbps RoCE）扫 9 个 size × 5 reps。
+
+| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求 11.5 GiB transfer |
+|---|---|---|
+| Intra-node | **9.7 GB/s** | p50 **1.9 s** · max 10 s |
+| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · max 9.2 s |
+
+**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC**，不走 NVLink。200 Gbps NIC 是天花板。**PD-disagg transfer cost 与拓扑无关**。
+
+参考图：`figs/mb2_transfer_time_compare.png`、doc `analysis/mb2/README.md`。
+
+### 4.3 MB1 — Phase interference（PD-disagg 的潜在 benefit 上界）
+
+dash1 GPU 0 单 instance（无 kv_connector），chunked-prefill 默认开启，D × P 扫描。D=8 结果：
+
+| Prefill | T_prefill | per-stream TPOT during | penalty |
+|---|---:|---:|---:|
+| 2k tok | 143 ms | 32 ms | 4× |
+| 32k tok | 4.5 s | 388 ms | **52×** |
+| 131k tok | 57 s | 1419 ms | **183×** |
+
+**Decode 在 prefill 期间被几乎完全 halted**，单 stream 损失 ≈ `T_prefill` 的时间。**每个 prefill event 总 decode 损失 ≈ `D × T_prefill`**。
+
+参考图：`figs/mb1_interference.png`、doc `analysis/mb1/README.md`。
+
+### 4.4 联合 cost-benefit（per-prefill event）
+
+| Prefill (KV size) | T_prefill | Cost = T_transfer | Benefit = D × T_prefill (D=8) | Cost / Benefit |
+|---:|---:|---:|---:|---:|
+| 2k tok (192 MiB) | 0.14 s | 8 ms | 1.1 s | **0.7%** |
+| 33k tok (3 GiB, trace mean) | 4.5 s | 0.32 s | 36 s | **0.9%** |
+| 125k tok (12 GiB, ~p99) | 57 s | 1.9 s | 456 s | **0.4%** |
+
+**PD-disagg 在 phase-isolation 这一维赢 100×–250×**。但**这不是 §3.2 该用的论证**，因为 §3.2 真正的 dominant failure 是 §4.1 的 D 池容量天花板（颠覆了上面的全部数学）。
+
+**总结**：
+- D 侧 KV 容量天花板（§4.1）→ PD-disagg 在 agentic 上**结构性失败**。
+- MB1 + MB2 的合计 cost-benefit 在 phase isolation 维度上 PD-disagg 是赢的，**但这件事被容量天花板压倒**。
+- Paper §3.2 论证应该聚焦"D 池装不下"，MB1/MB2 数据用作 supporting context（per-request transfer charge 量化、phase isolation benefit 量化）而不是 main argument。
+
+> ✅ **2026-05-30 更新 — 干净栈三轴 ablation 证实本节、并加 regime 细化。**
+> 本节的容量论点（D 池容量天花板 / decode 减半）在修复 `e13391e` 污染后的 clean stack
+> 上**得到确认**：concurrency 轴 N=64 时 PD 倾覆，**producer APC 从 71% 崩到 1.4%、TPS −30%**，
+> 而 colo 线性 scale（Fig 3）。**细化**：PD 并非"在 agentic 上一律失败"——它在
+> *低负载 / decode-heavy / 低复用* 区间**赢** colo；真正失败的是 agentic corner（高复用 +
+> 短输出 + 大上下文 + 高并发）——静态 P:D split 无法同时给出复用所需的 producer 容量
+> *和* decode 容量，而 colo 的弹性池两者兼得。
+> **另注**：旧 MB5 文档（`PD_DISAGG_RESULTS.md` §6）"session-affinity 救不了 PD / PD 复用=0%"
+> 的结论是 `e13391e`（producer 每次 KV 传输后 evict prefix）的**污染产物，已撤回**；
+> 干净栈上 session 路由的 producer APC 与 colo 持平（71–82%）。
+> 图：[`figs/mb5_pd_ablation/`](figs/mb5_pd_ablation/)。
+
+## 5. EAR 设计的实证状态（§4）
+
+| Pillar | 已实证 | 待实证 |
+|---|---|---|
+| **Affinity-default routing** (Pillar 1) | ✅ Current `unified` 算法 = LMetric + high-cache affinity；APC **79.4%**（达到 79.6% 上界 97%），TTFT p90 **7.3 s**，median worker p90 **10.3 s** | — |
+| **Hot-triggered session migration** (Pillar 2) | substrate 已通：`kv_both` connector 在 trace replay 上 net positive（TTFT p90 −18.6%，DR-fix 后 −36.6%），原 elastic_migration_v2 paper 的 "+45% kv_both penalty" obsolete | e2e 策略层（trigger 阈值 + target selection 在反馈环里）未直接验证 |
+
+## 6. 已经能写的 paper 主张（按 confidence 排序）
+
+1. **Agentic vs chatbot 在调度上是不同 regime**（dispatch coupling + sub-second tool-call mass）—— 实证完整
+2. **三类现有调度 baseline 各自的失败模式**（load-balance / static PD-disagg / pure sticky）—— 实证完整
+3. **Static PD-disagg 在 agentic 下失败的 dominant 根因是 D 侧 KV 容量**（不是 phase-isolation cost-benefit）—— 实证完整（`f4b` + colleague 4P+4D 数据）
+4. **Mooncake transfer cost 拓扑无关**（intra ≈ inter，~9.7 GB/s NIC 上限）—— 实证完整（MB2）
+5. **Phase isolation interference 在 chunked-prefill on 下仍然显著**（per-stream TPOT during prefill 15×–2000× baseline）—— 实证完整（MB1）。**注意**：这条数据本身不直接论证 "PD-disagg 失败"，因为算正确账后 PD-disagg 反而在这一维上赢；它的用途是给 §3.2 提供 phase-isolation benefit 上界的量化。
+6. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整
+7. **Hot-triggered migration 修复 sticky 的 hot pin** —— design 完整、e2e 待验证
+
+## 7. 待做
+
+- **MB3-5**（end-to-end PD-disagg deployment）：D-pool runtime occupancy、cache reuse × PD interaction、PD ratio sweep。这些是 §5 完整实验矩阵的事
+- **EAR Pillar 2 migration e2e validation**（在 connector_tax DR-fix 之上重测）
+- **§5.4 wall-clock amplification sweep**（5 baseline × 3 runs，钉死 dispatch coupling 论证的实证 closure）
+- **Scale-out 验证**（dash1+dash2 = 16 GPU，等 dash0 + 3-node 可用时扩到 80 GPU）

analysis/characterization/render_window1_figures.py

@@ -89,6 +89,48 @@ def fig_b3_latency_bars(comp: dict, out: Path) -> None:
     plt.close(fig)
 
 
+def fig_b3_latency_full_grid(results_dir: Path, out: Path) -> None:
+    """4 rows (mean / p50 / p90 / p99) × 3 cols (TTFT / TPOT / E2E) per policy.
+
+    Reads per-policy metrics.summary.json caches under raw_stats/, which
+    expose mean alongside the percentiles (b3_policy_comparison.json does
+    not record mean).
+    """
+    raw_dir = results_dir / "raw_stats"
+    pols = [p for p in POLICY_ORDER if (raw_dir / f"{p}.json").exists()]
+    if not pols:
+        return
+    stats = {p: json.loads((raw_dir / f"{p}.json").read_text()) for p in pols}
+
+    rows = [("mean", "mean"), ("p50", "p50"), ("p90", "p90"), ("p99", "p99")]
+    cols = [
+        ("TTFT (s)", "ttft", 1.0),
+        ("TPOT (ms)", "tpot", 1000.0),
+        ("E2E (s)", "e2e", 1.0),
+    ]
+    fig, axes = plt.subplots(len(rows), len(cols), figsize=(11, 11), sharex=True)
+    for i, (row_label, agg_key) in enumerate(rows):
+        for j, (col_label, metric_key, scale) in enumerate(cols):
+            ax = axes[i][j]
+            vals = [stats[p][metric_key][agg_key] * scale for p in pols]
+            ax.bar(pols, vals,
+                   color=[POLICY_COLOR.get(p, "gray") for p in pols],
+                   edgecolor="black", linewidth=0.5)
+            for k, v in enumerate(vals):
+                ax.text(k, v, f"{v:.1f}", ha="center", va="bottom", fontsize=8)
+            if j == 0:
+                ax.set_ylabel(row_label, fontsize=11)
+            if i == 0:
+                ax.set_title(col_label, fontsize=11)
+            ax.grid(alpha=0.3, axis="y")
+            ax.tick_params(axis="x", rotation=20, labelsize=9)
+            ax.margins(y=0.18)
+    fig.suptitle("B3 latencies per policy — mean / p50 / p90 / p99")
+    fig.tight_layout()
+    fig.savefig(out, dpi=120)
+    plt.close(fig)
+
+
 def fig_b3_apc_vs_upper(comp: dict, upper: dict, out: Path) -> None:
     by = {r["policy"]: r for r in comp["rows"]}
     pols = [p for p in POLICY_ORDER if p in by]
@@ -266,19 +308,67 @@ def fig_reuse_decomposition(reuse: dict, out: Path) -> None:
 
 
 def fig_kv_footprint_cdf(kv: dict, out: Path) -> None:
+    """How many concurrent decodes fit per percentile, under three deployments.
+
+    KV pool assumption: 96 GiB H20 HBM split ~50% model params (Qwen3-Coder-
+    30B-A3B in bf16 + headroom), ~10% runtime activations, leaving ~40% for
+    the KV cache pool — i.e. ~38.4 GiB per instance.
+
+    For each request-size percentile, we report system-wide concurrent
+    decode capacity = N_D × floor(KV_pool / req_size_MiB) under three 8-GPU
+    deployments: all-combined, 4P+4D, 6P+2D. The point is that going from
+    combined 8C to 4P+4D halves the system's decode population at the
+    same per-request KV pressure.
+    """
     s = kv.get("kv_mib_per_request") or {}
-    vals = [s.get(k) for k in ("p50", "p90", "p95", "p99")]
-    labels = ["p50", "p90", "p95", "p99"]
-    fig, ax = plt.subplots(figsize=(6, 3.5))
-    ax.bar(labels, vals, color="#1f77b4", edgecolor="black", linewidth=0.5)
-    for i, v in enumerate(vals):
-        ax.text(i, v, f"{v:.0f} MiB", ha="center", va="bottom", fontsize=9)
-    ax.axhline(95 * 1024, color="red", linestyle="--", alpha=0.5,
-                label="H20 ~95 GiB usable")
-    ax.set_ylabel("KV bytes per request (MiB)")
-    ax.set_title("B1' Per-request KV footprint (Qwen3-Coder-30B-A3B, 98304 B/token)")
-    ax.legend()
+    pct_keys = ["p50", "p90", "p95", "p99"]
+    req_mib = [float(s.get(k, 0.0)) for k in pct_keys]
+    req_gib = [v / 1024 for v in req_mib]
+
+    hbm_gib = 96.0
+    kv_pool_frac = 0.40
+    kv_pool_mib = hbm_gib * kv_pool_frac * 1024  # ≈ 39322 MiB per instance
+
+    deploys = [
+        ("Combined 8C",     8, "#2ca02c"),
+        ("PD-disagg 4P+4D", 4, "#ff7f0e"),
+        ("PD-disagg 6P+2D", 2, "#d62728"),
+    ]
+
+    import numpy as _np
+    x = _np.arange(len(pct_keys))
+    bar_w = 0.26
+
+    fig, ax = plt.subplots(figsize=(9, 5.2))
+    for i, (label, n_d, color) in enumerate(deploys):
+        per_inst = [int(kv_pool_mib // r) if r > 0 else 0 for r in req_mib]
+        sys_cap = [n_d * pi for pi in per_inst]
+        bars = ax.bar(x + (i - 1) * bar_w, sys_cap, bar_w,
+                      label=f"{label} (N_D={n_d})",
+                      color=color, edgecolor="black", linewidth=0.5)
+        for j, (b, n) in enumerate(zip(bars, sys_cap)):
+            ax.text(b.get_x() + b.get_width() / 2, n, str(n),
+                    ha="center", va="bottom", fontsize=9, color="#333")
+
+    # Annotate per-request KV size and per-instance fit just above the x-axis
+    per_inst_combined = [int(kv_pool_mib // r) if r > 0 else 0 for r in req_mib]
+    annot = [
+        f"{pct}\n{rg:.1f} GiB / req\nfits {pi}/inst"
+        for pct, rg, pi in zip(pct_keys, req_gib, per_inst_combined)
+    ]
+    ax.set_xticks(x)
+    ax.set_xticklabels(annot, fontsize=10)
+
+    ax.set_ylabel("System-wide concurrent decodes")
+    ax.set_title(
+        f"Per-instance KV pool ≈ {kv_pool_mib / 1024:.1f} GiB "
+        f"(0.4 × H20 96 GiB; remaining 0.5 model + 0.1 activation)\n"
+        f"PD-disagg halves the decode population at p90+ "
+        f"(Qwen3-Coder-30B-A3B, 98304 B/token)"
+    )
+    ax.legend(loc="upper right")
     ax.grid(alpha=0.3, axis="y")
+    ax.margins(y=0.15)
     fig.tight_layout()
     fig.savefig(out, dpi=120)
     plt.close(fig)
@@ -307,6 +397,9 @@ def main() -> None:
 
     fig_b3_apc_vs_hotspot(comp, upper, args.out_dir / "fig_b3_apc_vs_hotspot.png")
     fig_b3_latency_bars(comp, args.out_dir / "fig_b3_latency_bars.png")
+    fig_b3_latency_full_grid(
+        args.results_dir, args.out_dir / "fig_b3_latency_full_grid.png"
+    )
     fig_b3_apc_vs_upper(comp, upper, args.out_dir / "fig_b3_apc_vs_upper.png")
     fig_b3_failure_breakdown(comp, args.out_dir / "fig_b3_failure_breakdown.png")
     fig_b3_per_worker_ttft(args.results_dir, comp,

analysis/characterization/window_1_results/raw_stats/lmetric.json

@@ -0,0 +1,23 @@
+{
+  "ttft": {
+    "count": 1214.0,
+    "mean": 5.111546324698484,
+    "p50": 0.9387824369769078,
+    "p90": 15.671339168207492,
+    "p99": 53.56683189840049
+  },
+  "tpot": {
+    "count": 1214.0,
+    "mean": 0.01757124870168204,
+    "p50": 0.008854518407308914,
+    "p90": 0.02122720699121469,
+    "p99": 0.18280341184277568
+  },
+  "e2e": {
+    "count": 1214.0,
+    "mean": 9.518126648903337,
+    "p50": 2.754255389008904,
+    "p90": 24.8209177934099,
+    "p99": 80.59924928059091
+  }
+}

analysis/characterization/window_1_results/raw_stats/load_only.json

@@ -0,0 +1,23 @@
+{
+  "ttft": {
+    "count": 1214.0,
+    "mean": 6.268620166597892,
+    "p50": 1.2609447415161412,
+    "p90": 20.197147866390882,
+    "p99": 52.84285237012196
+  },
+  "tpot": {
+    "count": 1214.0,
+    "mean": 0.02406975794215626,
+    "p50": 0.009231464695980247,
+    "p90": 0.026851662550158716,
+    "p99": 0.3211630676943426
+  },
+  "e2e": {
+    "count": 1214.0,
+    "mean": 11.702793988628443,
+    "p50": 3.58568156149704,
+    "p90": 33.459180271782685,
+    "p99": 93.95083751494239
+  }
+}

analysis/characterization/window_1_results/raw_stats/sticky.json

@@ -0,0 +1,23 @@
+{
+  "ttft": {
+    "count": 1214.0,
+    "mean": 5.55315460854824,
+    "p50": 0.5415176274836995,
+    "p90": 18.021296651283045,
+    "p99": 74.09429564891524
+  },
+  "tpot": {
+    "count": 1214.0,
+    "mean": 0.027834537397398284,
+    "p50": 0.008952101894096181,
+    "p90": 0.03641285916619554,
+    "p99": 0.35152006935195085
+  },
+  "e2e": {
+    "count": 1214.0,
+    "mean": 12.109200157184377,
+    "p50": 2.081947358994512,
+    "p90": 34.62592205510591,
+    "p99": 139.68334607904353
+  }
+}

analysis/characterization/window_1_results/raw_stats/unified.json

@@ -0,0 +1,23 @@
+{
+  "ttft": {
+    "count": 1213.0,
+    "mean": 3.2790960856202394,
+    "p50": 0.4997710260213353,
+    "p90": 7.345769894809922,
+    "p99": 42.34170345296613
+  },
+  "tpot": {
+    "count": 1213.0,
+    "mean": 0.012493800538265787,
+    "p50": 0.008079791456705824,
+    "p90": 0.017110194704198407,
+    "p99": 0.12655874612209597
+  },
+  "e2e": {
+    "count": 1213.0,
+    "mean": 6.961301470549104,
+    "p50": 1.7495028690318577,
+    "p90": 18.033410895219994,
+    "p99": 68.80023987947489
+  }
+}

analysis/crossover/d1_i1024_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.1385939740576,
+      "amplification": 1.0013425037685975,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.08144469193556772,
+        "p50": 0.07862715201918036,
+        "p90": 0.08015060934703797,
+        "p99": 0.0875979653932154
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005001699398049616,
+        "p50": 0.004988961030788246,
+        "p90": 0.005045765990923557,
+        "p99": 0.005062779263327164
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.3968209869372152,
+        "p50": 0.393534954986535,
+        "p90": 0.39730903680901974,
+        "p99": 0.40925762055674536
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.1604231200181,
+      "amplification": 1.0014153316291596,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.09946277569807849,
+        "p50": 0.09600010397844017,
+        "p90": 0.10452785079833121,
+        "p99": 0.11205230774357905
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005007447102661814,
+        "p50": 0.004987124730611131,
+        "p90": 0.005003212126977151,
+        "p99": 0.005478902989961502
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.415208436744531,
+        "p50": 0.41056320699863136,
+        "p90": 0.4200975856045261,
+        "p99": 0.44871115096379066
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.2032543020323,
+      "amplification": 1.0015582278907484,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.10561635944505095,
+        "p50": 0.10468761203810573,
+        "p90": 0.11257308297790587,
+        "p99": 0.12065987563692024
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005328901365752947,
+        "p50": 0.005144592110318915,
+        "p90": 0.005990574603515958,
+        "p99": 0.006688486758013448
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.4416300980896939,
+        "p50": 0.42991435900330544,
+        "p90": 0.4854830394731835,
+        "p99": 0.5404117306252005
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d1_i16384_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 303.05708941399644,
+      "amplification": 1.0110793839624128,
+      "ttft": {
+        "count": 1167,
+        "mean": 1.674437926279444,
+        "p50": 1.5353219069947954,
+        "p90": 2.0787689138029237,
+        "p99": 3.039117059087727
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.035498316425319934,
+        "p50": 0.02951206674587743,
+        "p90": 0.085249871320677,
+        "p99": 0.15422643764865662
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 3.9111985703531236,
+        "p50": 3.392241114997887,
+        "p90": 7.760864628604043,
+        "p99": 11.30427318874542
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 1136,
+      "completion_rate": 0.9734361610968295,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 866.9596620649972,
+      "amplification": 2.8924089607535697,
+      "ttft": {
+        "count": 1136,
+        "mean": 65.09021699308856,
+        "p50": 66.22900710900285,
+        "p90": 112.5535424454938,
+        "p99": 124.55262411334482
+      },
+      "tpot": {
+        "count": 1136,
+        "mean": 0.005710658520121912,
+        "p50": 0.005725543936557461,
+        "p90": 0.005750613698356098,
+        "p99": 0.0058447879207267845
+      },
+      "e2e": {
+        "count": 1136,
+        "mean": 65.4504681098121,
+        "p50": 66.59053339700768,
+        "p90": 112.9150809329949,
+        "p99": 124.91415351489852
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 307.42712411200046,
+      "amplification": 1.0256589867656205,
+      "ttft": {
+        "count": 1167,
+        "mean": 3.6233640342625417,
+        "p50": 3.255483777000336,
+        "p90": 6.0935156565916255,
+        "p99": 7.349482456580735
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.006360297526341433,
+        "p50": 0.006324973206372104,
+        "p90": 0.007198417158741947,
+        "p99": 0.007942749238420567
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 4.024414356593621,
+        "p50": 3.6796783979953034,
+        "p90": 6.510242249601289,
+        "p99": 7.7530036393977895
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.6563838903170522,
+          "pd_advantage": 1.0,
+          "n_slo": 766
+        },
+        "4P+4D": {
+          "attainment": 0.0,
+          "pd_advantage": 0.0,
+          "n_slo": 0
+        },
+        "6P+2D": {
+          "attainment": 0.20565552699228792,
+          "pd_advantage": 0.3133159268929504,
+          "n_slo": 240
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.7095115681233933,
+          "pd_advantage": 1.0,
+          "n_slo": 828
+        },
+        "4P+4D": {
+          "attainment": 0.012853470437017995,
+          "pd_advantage": 0.018115942028985508,
+          "n_slo": 15
+        },
+        "6P+2D": {
+          "attainment": 0.7746358183376179,
+          "pd_advantage": 1.0917874396135265,
+          "n_slo": 904
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d1_i2048_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.1927249849541,
+      "amplification": 1.0015230992770423,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.1367582682356152,
+        "p50": 0.1337889520218596,
+        "p90": 0.13568343156948687,
+        "p99": 0.1417779766954481
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005039774583944486,
+        "p50": 0.005025791809968059,
+        "p90": 0.005088228446667984,
+        "p99": 0.0051380019550329985
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.45452996390628697,
+        "p50": 0.45110413199290633,
+        "p90": 0.4555703255813569,
+        "p99": 0.4688122991356063
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.2166367470054,
+      "amplification": 1.0016028753010615,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.1660798433322169,
+        "p50": 0.16653224604669958,
+        "p90": 0.17306958099361508,
+        "p99": 0.1832630861806683
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005046875948133808,
+        "p50": 0.0050270736812510425,
+        "p90": 0.0050664691831029595,
+        "p99": 0.005532995437730161
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.48433690918382055,
+        "p50": 0.4845748710213229,
+        "p90": 0.4916891472181305,
+        "p99": 0.5235905526624989
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.2855537990108,
+      "amplification": 1.001832800991369,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.17076528707394278,
+        "p50": 0.16959826194215566,
+        "p90": 0.17923957279417663,
+        "p99": 0.18792458378709853
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005377765550956012,
+        "p50": 0.005177637602808693,
+        "p90": 0.006078562941697855,
+        "p99": 0.007001848420689976
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.5098518045703324,
+        "p50": 0.4961305959150195,
+        "p90": 0.5569328068522736,
+        "p99": 0.6314893902023319
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d1_i32768_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 951,
+      "completion_rate": 0.8149100257069408,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 567.1572082909988,
+      "amplification": 1.8921879104611408,
+      "ttft": {
+        "count": 951,
+        "mean": 156.08624654169503,
+        "p50": 159.80228465000982,
+        "p90": 263.44082942500245,
+        "p99": 267.3189407904938
+      },
+      "tpot": {
+        "count": 951,
+        "mean": 0.5120599246480225,
+        "p50": 0.5265210192539223,
+        "p90": 0.5403215261268621,
+        "p99": 0.5436686666190138
+      },
+      "e2e": {
+        "count": 951,
+        "mean": 188.3470215559188,
+        "p50": 192.9057134770119,
+        "p90": 294.68771972299146,
+        "p99": 300.2205298934932
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 596,
+      "completion_rate": 0.5107112253641817,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 789.8321366070013,
+      "amplification": 2.63509093833936,
+      "ttft": {
+        "count": 596,
+        "mean": 240.99040966441748,
+        "p50": 259.1066791820049,
+        "p90": 379.7789027979961,
+        "p99": 381.22047349565617
+      },
+      "tpot": {
+        "count": 596,
+        "mean": 0.006412497327210905,
+        "p50": 0.006444117785865692,
+        "p90": 0.0064974606745333095,
+        "p99": 0.006660929446938789
+      },
+      "e2e": {
+        "count": 596,
+        "mean": 241.39516468475196,
+        "p50": 259.5132920899996,
+        "p90": 380.1822811405,
+        "p99": 381.620277955458
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 712,
+      "completion_rate": 0.6101113967437874,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 880.7267912379903,
+      "amplification": 2.938339780290167,
+      "ttft": {
+        "count": 712,
+        "mean": 175.69983718378106,
+        "p50": 172.43730882649834,
+        "p90": 250.6456608123961,
+        "p99": 594.014642834873
+      },
+      "tpot": {
+        "count": 712,
+        "mean": 0.007946936024875445,
+        "p50": 0.007568808706382119,
+        "p90": 0.010293392149296345,
+        "p99": 0.011275891952114068
+      },
+      "e2e": {
+        "count": 712,
+        "mean": 176.20114515147316,
+        "p50": 172.91078094949626,
+        "p90": 251.09661155799986,
+        "p99": 594.4257701966026
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        },
+        "4P+4D": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        },
+        "6P+2D": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        },
+        "4P+4D": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        },
+        "6P+2D": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d1_i4096_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.32634208898526,
+      "amplification": 1.0019688816195438,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.2628433253752162,
+        "p50": 0.2604120689211413,
+        "p90": 0.2637787788407877,
+        "p99": 0.2976114549371393
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005143964614959416,
+        "p50": 0.005124552334318795,
+        "p90": 0.005180289564166395,
+        "p99": 0.0052467077378080115
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.5871811069348818,
+        "p50": 0.5839827890740708,
+        "p90": 0.5883447280153632,
+        "p99": 0.6408826243295329
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.3759389320621,
+      "amplification": 1.0021343499332585,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.30861241175426407,
+        "p50": 0.3091614870354533,
+        "p90": 0.3227507559815422,
+        "p99": 0.3547600568598136
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005148936446188915,
+        "p50": 0.0051316127147791645,
+        "p90": 0.005184003428393413,
+        "p99": 0.00547781713853871
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.6332931331915089,
+        "p50": 0.6365591660141945,
+        "p90": 0.6472898541716858,
+        "p99": 0.7053035433730102
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.4041277950164,
+      "amplification": 1.0022283955081221,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.31527804771431056,
+        "p50": 0.31678270106203854,
+        "p90": 0.3284846659982577,
+        "p99": 0.3402980738645419
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.0055213440500518985,
+        "p50": 0.005325577683776381,
+        "p90": 0.006278165857056304,
+        "p99": 0.0070092806818761975
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.6634175265073216,
+        "p50": 0.6494167000055313,
+        "p90": 0.7135140666039661,
+        "p99": 0.7741130576422437
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d1_i8192_goodput.json

@@ -0,0 +1,142 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.66455180699995,
+      "amplification": 1.0030972395602922,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.584800394000121,
+        "p50": 0.5817365200055065,
+        "p90": 0.5870239008043427,
+        "p99": 0.6088061455392736
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005585180932058873,
+        "p50": 0.005308644777941813,
+        "p90": 0.005355837698406836,
+        "p99": 0.014590345752427337
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 0.9369382756572071,
+        "p50": 0.9158835929993074,
+        "p90": 0.9229304600012256,
+        "p99": 1.5052119748995754
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.7607197400066,
+      "amplification": 1.003418081467173,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.7293765576349805,
+        "p50": 0.6887177099997643,
+        "p90": 0.8911576298065484,
+        "p99": 1.1879012519156076
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005308951736224922,
+        "p50": 0.0053098901269203495,
+        "p90": 0.005347346425488857,
+        "p99": 0.005420954272433716
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 1.064155324243521,
+        "p50": 1.0238807429996086,
+        "p90": 1.22566919879755,
+        "p99": 1.521622942700049
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 1167,
+      "n_success": 1167,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.736197,
+      "offered_qps": 3.8934236561358655,
+      "wall_clock_s": 300.8507765819959,
+      "amplification": 1.003718535142407,
+      "ttft": {
+        "count": 1167,
+        "mean": 0.7028102769056583,
+        "p50": 0.6950660040020011,
+        "p90": 0.7394948218076024,
+        "p99": 0.8658222487580485
+      },
+      "tpot": {
+        "count": 1167,
+        "mean": 0.005782890183689697,
+        "p50": 0.00558671395230617,
+        "p90": 0.006629064533303286,
+        "p99": 0.007366264003757681
+      },
+      "e2e": {
+        "count": 1167,
+        "mean": 1.0674261210719522,
+        "p50": 1.0612060459970962,
+        "p90": 1.1425091800017981,
+        "p99": 1.3045775910036173
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    },
+    {
+      "ttft_slo_s": 5.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 1167
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o1024_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 1950,
+      "completion_rate": 0.5489864864864865,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 348.4953830049999,
+      "amplification": 1.1626534102882857,
+      "ttft": {
+        "count": 1950,
+        "mean": 0.8245253885559363,
+        "p50": 0.7988759850013594,
+        "p90": 1.5384256363991882,
+        "p99": 2.125257717882705
+      },
+      "tpot": {
+        "count": 1950,
+        "mean": 0.07444567704430907,
+        "p50": 0.08914986800879843,
+        "p90": 0.09127071481447005,
+        "p99": 0.09233926755616054
+      },
+      "e2e": {
+        "count": 1950,
+        "mean": 77.04034660455785,
+        "p50": 92.29079191399796,
+        "p90": 94.47993849659906,
+        "p99": 95.72745552870882
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 2754,
+      "completion_rate": 0.7753378378378378,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 744.3694221920014,
+      "amplification": 2.4833719166185753,
+      "ttft": {
+        "count": 2754,
+        "mean": 4.5006646864741695,
+        "p50": 2.329877773499902,
+        "p90": 10.863291689799375,
+        "p99": 21.572781211729307
+      },
+      "tpot": {
+        "count": 2754,
+        "mean": 0.046089308743682535,
+        "p50": 0.04704797274047173,
+        "p90": 0.04792202048768524,
+        "p99": 0.059544689888886115
+      },
+      "e2e": {
+        "count": 2754,
+        "mean": 51.68415100813728,
+        "p50": 50.91020956099965,
+        "p90": 59.22212425199977,
+        "p99": 68.42946432940865
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 1928,
+      "completion_rate": 0.5427927927927928,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 821.7403777359868,
+      "amplification": 2.7414975897474716,
+      "ttft": {
+        "count": 1928,
+        "mean": 39.07585104131927,
+        "p50": 42.6814165695032,
+        "p90": 63.28579387369681,
+        "p99": 73.84470144698193
+      },
+      "tpot": {
+        "count": 1928,
+        "mean": 0.04064862157373329,
+        "p50": 0.0398688508558163,
+        "p90": 0.04277554483881431,
+        "p99": 0.0695667276081441
+      },
+      "e2e": {
+        "count": 1928,
+        "mean": 80.66499786808757,
+        "p50": 84.05398454950046,
+        "p90": 105.42814423799427,
+        "p99": 113.52591980495636
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.10191441441441441,
+          "pd_advantage": 1.0,
+          "n_slo": 362
+        },
+        "4P+4D": {
+          "attainment": 0.3502252252252252,
+          "pd_advantage": 3.43646408839779,
+          "n_slo": 1244
+        },
+        "6P+2D": {
+          "attainment": 0.06447072072072071,
+          "pd_advantage": 0.6325966850828729,
+          "n_slo": 229
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o128_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 300.61600343801547,
+      "amplification": 1.0029177964157683,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.139721406387908,
+        "p50": 0.13759525847854093,
+        "p90": 0.14188919204752892,
+        "p99": 0.17052793529117438
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.006496044329480822,
+        "p50": 0.006430442302898453,
+        "p90": 0.007889476366154117,
+        "p99": 0.009282101713493095
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 0.9651016481538636,
+        "p50": 0.9548319909954444,
+        "p90": 1.1440699087572284,
+        "p99": 1.3324073514551853
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 300.7410873459885,
+      "amplification": 1.0033351024670651,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.18467717330343286,
+        "p50": 0.1806962049449794,
+        "p90": 0.1929074571467936,
+        "p99": 0.2592483600333798
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.006980159784640828,
+        "p50": 0.006945047342387123,
+        "p90": 0.008164690015837552,
+        "p99": 0.009163911323187739
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 1.0715517728354285,
+        "p50": 1.063076548918616,
+        "p90": 1.2275727374362762,
+        "p99": 1.4021264237130526
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 301.1243950970238,
+      "amplification": 1.004613897210595,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.20078434001432569,
+        "p50": 0.19958186394069344,
+        "p90": 0.213167638995219,
+        "p99": 0.23225504373782313
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.010393430778126047,
+        "p50": 0.010396533917478015,
+        "p90": 0.012065167099743436,
+        "p99": 0.01323438493101344
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 1.5212831554822883,
+        "p50": 1.5218885459471494,
+        "p90": 1.7440477049094625,
+        "p99": 1.9191367691196497
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        },
+        "4P+4D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o2048_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 1050,
+      "completion_rate": 0.2956081081081081,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 414.92026689198974,
+      "amplification": 1.3842607013613617,
+      "ttft": {
+        "count": 1050,
+        "mean": 0.8393797435154972,
+        "p50": 0.8442945880015031,
+        "p90": 1.4299255040969001,
+        "p99": 2.1686189359705894
+      },
+      "tpot": {
+        "count": 1050,
+        "mean": 0.08418485046407832,
+        "p50": 0.08937374470713316,
+        "p90": 0.09085719287396188,
+        "p99": 0.09190563933350096
+      },
+      "e2e": {
+        "count": 1050,
+        "mean": 173.22149556513375,
+        "p50": 183.9983977669981,
+        "p90": 187.03014375649946,
+        "p99": 189.36978534828302
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 1320,
+      "completion_rate": 0.3716216216216216,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 802.1317602040072,
+      "amplification": 2.6760791447511205,
+      "ttft": {
+        "count": 1320,
+        "mean": 19.92388417242484,
+        "p50": 8.059029546493548,
+        "p90": 64.71132232169766,
+        "p99": 87.07121913193099
+      },
+      "tpot": {
+        "count": 1320,
+        "mean": 0.05007250707090057,
+        "p50": 0.0492711045202749,
+        "p90": 0.07142825628519621,
+        "p99": 0.08267229383702501
+      },
+      "e2e": {
+        "count": 1320,
+        "mean": 122.44309290737698,
+        "p50": 118.35192646600626,
+        "p90": 167.89536100080443,
+        "p99": 187.19021692074878
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 1041,
+      "completion_rate": 0.29307432432432434,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 879.8981197069952,
+      "amplification": 2.935523968100636,
+      "ttft": {
+        "count": 1041,
+        "mean": 86.06142147925078,
+        "p50": 92.45557322700915,
+        "p90": 153.47880471601093,
+        "p99": 159.16995789420034
+      },
+      "tpot": {
+        "count": 1041,
+        "mean": 0.04275836784091199,
+        "p50": 0.038143618996089604,
+        "p90": 0.06297858072350163,
+        "p99": 0.07100282797010407
+      },
+      "e2e": {
+        "count": 1041,
+        "mean": 173.5925440399473,
+        "p50": 199.81516771799943,
+        "p90": 230.71809448600106,
+        "p99": 234.9438766648003
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.009009009009009009,
+          "pd_advantage": 1.0,
+          "n_slo": 32
+        },
+        "4P+4D": {
+          "attainment": 0.11486486486486487,
+          "pd_advantage": 12.75,
+          "n_slo": 408
+        },
+        "6P+2D": {
+          "attainment": 0.04954954954954955,
+          "pd_advantage": 5.5,
+          "n_slo": 176
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o256_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 301.78380814404227,
+      "amplification": 1.0068138369093473,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.14506312318632342,
+        "p50": 0.14291562204016373,
+        "p90": 0.1491016250802204,
+        "p99": 0.1832571337954128
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.0089611879488046,
+        "p50": 0.008925844482420122,
+        "p90": 0.010698060123752072,
+        "p99": 0.011754058378410762
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 2.430673889741114,
+        "p50": 2.419925262918696,
+        "p90": 2.874893436266575,
+        "p99": 3.1480205075151746
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 3551,
+      "completion_rate": 0.9997184684684685,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 744.8868488909211,
+      "amplification": 2.4850981601942745,
+      "ttft": {
+        "count": 3551,
+        "mean": 0.20500909556101396,
+        "p50": 0.20017725799698383,
+        "p90": 0.21613375598099083,
+        "p99": 0.29023518750909716
+      },
+      "tpot": {
+        "count": 3551,
+        "mean": 0.010307210080394247,
+        "p50": 0.010299421709450874,
+        "p90": 0.011793581666607482,
+        "p99": 0.012639003840969035
+      },
+      "e2e": {
+        "count": 3551,
+        "mean": 2.8339171693501317,
+        "p50": 2.8360355379991233,
+        "p90": 3.2198793930001557,
+        "p99": 3.438588996999897
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 303.4300506779691,
+      "amplification": 1.0123060459587976,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.2333820717117756,
+        "p50": 0.23283391550648957,
+        "p90": 0.24394672318594532,
+        "p99": 0.2734717815916518
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.016453822599812207,
+        "p50": 0.016576926972415737,
+        "p90": 0.017214638463623238,
+        "p99": 0.01769411424845092
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 4.430368151474328,
+        "p50": 4.463736370031256,
+        "p90": 4.628522685484495,
+        "p99": 4.75812664218829
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        },
+        "4P+4D": {
+          "attainment": 0.9997184684684685,
+          "pd_advantage": 0.9997184684684685,
+          "n_slo": 3551
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o4096_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 545,
+      "completion_rate": 0.15343468468468469,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 505.4139580530027,
+      "amplification": 1.6861665622961597,
+      "ttft": {
+        "count": 545,
+        "mean": 4.137136383750316,
+        "p50": 0.5702517530007754,
+        "p90": 1.6976309113961179,
+        "p99": 56.1814190310361
+      },
+      "tpot": {
+        "count": 545,
+        "mean": 0.08390926873267023,
+        "p50": 0.08933810225128375,
+        "p90": 0.09449654152820693,
+        "p99": 0.10591519025133134
+      },
+      "e2e": {
+        "count": 545,
+        "mean": 347.7765141811284,
+        "p50": 366.3568219640001,
+        "p90": 388.1241692415934,
+        "p99": 435.2848098113155
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 771,
+      "completion_rate": 0.2170608108108108,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 850.1328976760124,
+      "amplification": 2.8362209684341706,
+      "ttft": {
+        "count": 771,
+        "mean": 63.210636654403324,
+        "p50": 1.1097561890055658,
+        "p90": 179.64081536799495,
+        "p99": 186.67013871119852
+      },
+      "tpot": {
+        "count": 771,
+        "mean": 0.05484690890217986,
+        "p50": 0.04795774376874187,
+        "p90": 0.08067800507863548,
+        "p99": 0.09099416300415113
+      },
+      "e2e": {
+        "count": 771,
+        "mean": 287.81779562259663,
+        "p50": 312.2160719559906,
+        "p90": 372.0004520520015,
+        "p99": 379.3279856524052
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 627,
+      "completion_rate": 0.17652027027027026,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 867.8833550199925,
+      "amplification": 2.8954402028102515,
+      "ttft": {
+        "count": 627,
+        "mean": 179.58769048342904,
+        "p50": 238.1998468660022,
+        "p90": 378.29023678940143,
+        "p99": 385.40577973942356
+      },
+      "tpot": {
+        "count": 627,
+        "mean": 0.04188420961205498,
+        "p50": 0.03654626756630041,
+        "p90": 0.06031132874202571,
+        "p99": 0.06738955674930582
+      },
+      "e2e": {
+        "count": 627,
+        "mean": 351.1066520824709,
+        "p50": 387.2650127700035,
+        "p90": 507.9008203571953,
+        "p99": 570.6463984230224
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 0.0,
+          "pd_advantage": NaN,
+          "n_slo": 0
+        },
+        "4P+4D": {
+          "attainment": 0.05855855855855856,
+          "pd_advantage": NaN,
+          "n_slo": 208
+        },
+        "6P+2D": {
+          "attainment": 0.036036036036036036,
+          "pd_advantage": NaN,
+          "n_slo": 128
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o512_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 304.97662343096454,
+      "amplification": 1.0174657357946404,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.16242270423816507,
+        "p50": 0.15836620499612764,
+        "p90": 0.17363918052287775,
+        "p99": 0.24847999344696287
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.013691483182659388,
+        "p50": 0.013865661168213087,
+        "p90": 0.015242900696529327,
+        "p99": 0.01626683903372128
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 7.160272567887971,
+        "p50": 7.247483663493767,
+        "p90": 7.962273431208451,
+        "p99": 8.480892732607899
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 3551,
+      "completion_rate": 0.9997184684684685,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 744.9637431279989,
+      "amplification": 2.4853546954349977,
+      "ttft": {
+        "count": 3551,
+        "mean": 0.2446379999283825,
+        "p50": 0.24067969399038702,
+        "p90": 0.2630795220611617,
+        "p99": 0.3426029055262916
+      },
+      "tpot": {
+        "count": 3551,
+        "mean": 0.016916919575073883,
+        "p50": 0.016998299133030737,
+        "p90": 0.01775875886689886,
+        "p99": 0.018166751548973206
+      },
+      "e2e": {
+        "count": 3551,
+        "mean": 8.89104466149889,
+        "p50": 8.933106195996515,
+        "p90": 9.333998591057025,
+        "p99": 9.536390463996213
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 3551,
+      "completion_rate": 0.9997184684684685,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 312.45468625507783,
+      "amplification": 1.0424141157985172,
+      "ttft": {
+        "count": 3551,
+        "mean": 0.36231219612768273,
+        "p50": 0.3462264990666881,
+        "p90": 0.4059687410481274,
+        "p99": 0.9837204645154998
+      },
+      "tpot": {
+        "count": 3551,
+        "mean": 0.03268022218101953,
+        "p50": 0.03333031399418835,
+        "p90": 0.03557429400772957,
+        "p99": 0.038558618127279086
+      },
+      "e2e": {
+        "count": 3551,
+        "mean": 17.068018403084057,
+        "p50": 17.392655145958997,
+        "p90": 18.56302695896011,
+        "p99": 20.159411454980727
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        },
+        "4P+4D": {
+          "attainment": 0.9997184684684685,
+          "pd_advantage": 0.9997184684684685,
+          "n_slo": 3551
+        },
+        "6P+2D": {
+          "attainment": 0.9997184684684685,
+          "pd_advantage": 0.9997184684684685,
+          "n_slo": 3551
+        }
+      }
+    }
+  ]
+}

analysis/crossover/d2_o64_goodput.json

@@ -0,0 +1,121 @@
+{
+  "baseline": "8C-proxy",
+  "arms": {
+    "8C-proxy": {
+      "name": "8C-proxy",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 300.21845804993063,
+      "amplification": 1.001591501940313,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.1369056966029782,
+        "p50": 0.13516780693316832,
+        "p90": 0.13835511771030723,
+        "p99": 0.15317223175661632
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.005430781936309906,
+        "p50": 0.005039893150780468,
+        "p90": 0.007063257358303028,
+        "p99": 0.0077793481296283135
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 0.4793568905444592,
+        "p50": 0.45412574551301077,
+        "p90": 0.5810393166844734,
+        "p99": 0.6301419002050533
+      }
+    },
+    "4P+4D": {
+      "name": "4P+4D",
+      "n_offered": 3552,
+      "n_success": 3551,
+      "completion_rate": 0.9997184684684685,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 735.1059510430787,
+      "amplification": 2.452467041410379,
+      "ttft": {
+        "count": 3551,
+        "mean": 0.17374673821534295,
+        "p50": 0.17102365801110864,
+        "p90": 0.18273873499128968,
+        "p99": 0.24688138795318082
+      },
+      "tpot": {
+        "count": 3551,
+        "mean": 0.005485491774206834,
+        "p50": 0.0053864502226046865,
+        "p90": 0.006091995366168992,
+        "p99": 0.007109403222178419
+      },
+      "e2e": {
+        "count": 3551,
+        "mean": 0.5196563635758822,
+        "p50": 0.5100997349945828,
+        "p90": 0.5655082209268585,
+        "p99": 0.6639982180204242
+      }
+    },
+    "6P+2D": {
+      "name": "6P+2D",
+      "n_offered": 3552,
+      "n_success": 3552,
+      "completion_rate": 1.0,
+      "offered_window_s": 299.74141899999995,
+      "offered_qps": 11.850214134070008,
+      "wall_clock_s": 300.38101644301787,
+      "amplification": 1.0021338307036503,
+      "ttft": {
+        "count": 3552,
+        "mean": 0.18293367427152893,
+        "p50": 0.1822461549891159,
+        "p90": 0.1938482352765277,
+        "p99": 0.21272844232735222
+      },
+      "tpot": {
+        "count": 3552,
+        "mean": 0.007192309629699456,
+        "p50": 0.007143509595484902,
+        "p90": 0.008732455453672816,
+        "p99": 0.009842920153335268
+      },
+      "e2e": {
+        "count": 3552,
+        "mean": 0.636424056327808,
+        "p50": 0.6324848984950222,
+        "p90": 0.7393875011475757,
+        "p99": 0.8261980937235056
+      }
+    }
+  },
+  "slo_grid": [
+    {
+      "ttft_slo_s": 2.0,
+      "tpot_slo_s": 0.05,
+      "arms": {
+        "8C-proxy": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        },
+        "4P+4D": {
+          "attainment": 0.9997184684684685,
+          "pd_advantage": 0.9997184684684685,
+          "n_slo": 3551
+        },
+        "6P+2D": {
+          "attainment": 1.0,
+          "pd_advantage": 1.0,
+          "n_slo": 3552
+        }
+      }
+    }
+  ]
+}

analysis/lpwl_5policy_600s.md

@@ -0,0 +1,131 @@
+# LPWL vs 4 baselines — parameter-free routing for agentic workloads
+
+Date: 2026-05-29. Hardware: dash1, 8×H20, Qwen3-Coder-30B-A3B, TP=1,
+max_model_len=200000, fresh vLLM per arm (cold APC), `--policy` via
+`scripts/b3_isolated_policy.sh`. Analyzer: `scripts/bench_report.py`.
+
+## Motivation
+
+unified+A+B carries too many knobs (`overload_factor`, `lmetric_decode_weight`,
+the 0.5 `cache_ratio` gate). Goal: a policy derived from the agentic *pattern*
+with no tuned constants, that does not overfit.
+
+## LPWL (Least-Prefill-Work-Left)
+
+`scripts/cache_aware_proxy.py:pick_instance_leastwork`, `--policy leastwork`:
+
+```
+score_i = pending_prefill_tokens_i + max(0, input_len − cache_hit_i)   → argmin
+```
+
+Tie-break: fewest `num_requests`, then round-robin. **Zero hyperparameters.**
+
+Why this shape (straight from the workload):
+- Decode is cheap (I/O ~217×) ⇒ the only load worth modeling is outstanding
+  *prefill* token-work. No decode weight; dropping LMetric's `×num_requests`
+  also makes an idle-but-decoding host score `input` (its true marginal cost),
+  not 0 — fixing the empty-batch degeneracy for free.
+- Cache-awareness *is* the affinity mechanism: a returning session's owner has
+  `new_uncached ≈ 0`, so it sticks unless its prefill backlog exceeds the cache
+  saving (`input`). The stick-vs-spill crossover is computed from real
+  token-work — no `overload_factor`, no `cache_ratio` gate.
+- Session skew degrades gracefully: a heavy session inflates its owner's
+  `pending_prefill`, auto-diverting *other* sessions while the heavy one stays
+  put (no cold re-prefill).
+
+## Results — 600s trace (`w600_r0.0015_st30_first600s.jsonl`, 807 reqs)
+
+This is the colder regime (theoretical APC ceiling ≈ 70% vs 80% for full w600).
+
+| policy | knobs | TTFT mean | TTFT p90 | E2E mean | E2E p90 | E2E p99 | TPOT p90 | APC | req-bal |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| **LPWL** | **0** | **3398** | **7983** | **8116** | **19014** | 87024 | 26 | 0.648 | **1.55×** |
+| unified+A+B | 3 | 3876 | 11562 | 8199 | 22569 | **74266** | **25** | 0.661 | 1.56× |
+| unified default | 2 | 5066 | 16389 | 10481 | 28427 | 96361 | 34 | 0.689 | 2.28× |
+| LMetric | 0 | 4809 | 14037 | 10051 | 26726 | 97442 | 32 | 0.507 | 2.11× |
+| sticky | 0 | 5758 | 20356 | 10815 | 34734 | 82732 | 28 | **0.696** | 3.86× |
+
+(latencies ms; req-bal = max:min per-worker request count; this batch predates
+the GPU-capture harness change so per-worker GPU util reads N/A.)
+
+### Findings
+
+1. **LPWL is overall best with zero knobs:** TTFT mean −12% / p90 −31%, E2E
+   mean ~tie / p90 −16% vs the tuned unified+A+B; best request balance; TPOT
+   tied-best. Only loss is E2E p99 (+17%) from heavy-class decode concentration.
+2. **The baselines bracket the problem and explain why LPWL works:** sticky has
+   the highest APC (0.696) but worst latency (hot-pin, 3.86× imbalance); LMetric
+   has the worst APC (0.507) because `×num_requests` swallows the cache signal.
+   LPWL drops exactly that factor, so locality re-emerges (APC 0.648, beside the
+   explicit-affinity policies) while balance stays tight — the sweet spot, no
+   gate, no tuning.
+3. **Anti-overfit, demonstrated:** unified+A+B was tuned (of=1.3, lmw=0.01) on
+   the *full* w600; on the colder 600s regime the parameter-free policy beats it
+   by 31% TTFT p90. The tuning did not transfer; LPWL did.
+
+### Per-class TTFT (ms, mean / p50 / p90 / p99) — LPWL dominates except the floor
+
+| class | LPWL p90 | A+B p90 | LPWL p99 | A+B p99 |
+|---|---:|---:|---:|---:|
+| WARM<5k | 319 | 324 | 1032 | 2092 |
+| MED5-20k | 1618 | 1952 | 3013 | 33189 |
+| HEAVY20-50k | 4851 | 6198 | 14599 | 29044 |
+| HEAVY+>50k | 28942 | 33777 | 52651 | 50778 |
+
+LPWL's only weak class is the workload-inherent HEAVY+>50k floor (≈tied across
+all policies). Elsewhere it avoids the mid-class tails the unified gate creates
+when it pins a mid request behind a 50k-token turn on a barely-warm owner.
+
+## Full-w600 cross-check (1214 reqs, `outputs/lpwl_vs_ab_live/`)
+
+On the warmer full trace, LPWL vs unified+A+B is a wash: LPWL wins TTFT p90
+(−14%) but loses TPOT (+38%) and per-worker balance. Combined claim across both
+regimes: **LPWL ∈ [tied, clearly-better] vs a tuned baseline, at zero knobs.**
+
+## Ablation: derived-κ decode term (`leastwork_kappa`) — NET-NEGATIVE
+
+Tested the proposed knob-free fix for LPWL's E2E-p99: `--policy leastwork_kappa`,
+`score = (pending_prefill + new_uncached) × (1 + κ·ongoing_decode_tokens)`, with
+κ = 2.5e-6 *derived* from hardware (KV ~100 KB/tok ÷ HBM 4 TB/s ÷ TPOT 10 ms on
+H20+Qwen3-30B-A3B), not trace-tuned. Same 600s trace, fresh vLLM, cold APC.
+
+| metric | leastwork | leastwork_kappa | Δ |
+|---|---:|---:|---:|
+| TTFT p90 | 7983 | 9390 | +18% (worse) |
+| TTFT p99 | 44891 | 42370 | −6% |
+| E2E p90 | 19014 | 21674 | +14% (worse) |
+| E2E p99 | 87024 | 90155 | +4% (did NOT fix) |
+| APC | 0.648 | 0.647 | tie |
+| req-balance | 1.55× | 1.97× | worse |
+
+**Verdict: decode-awareness is the wrong lever for agentic.** The κ term is
+correct physics aimed at a negligible effect (decode is cheap, output p50≈80),
+so it mostly bounces heavy requests off their cache-owner → cold re-prefill
+elsewhere → new hotspots (balance degrades 1.55×→1.97×). It does NOT fix E2E-p99
+because that tail is the **structural HEAVY+>50k floor** (per-class p99 ≈51–52k
+for *all* policies), not decode interference — i.e. not routing-fixable. This is
+a negative result that *justifies* LPWL's omission of any decode term. The policy
+is kept in-tree as a documented ablation; do not revive without a decode-heavy
+regime. (First run on the GPU-capturing harness: per-worker GPU util mean 42–83%,
+1.95× spread — it even shows the κ-induced imbalance.)
+
+## Caveats / open work
+
+- n=1 per arm. The 600s −31% TTFT p90 is corroborated by mean/p50/per-class, but
+  repeat to bound run-to-run noise (no 3× repeats yet, by request — quick single
+  set first).
+- E2E-p99 deep tail is the one consistent LPWL weak spot (heavy-session decode
+  concentration). Proposed knob-free fix: add `+ κ·ongoing_decode` with
+  `κ = measured(TPOT/token) / prefill_throughput` (a derived hardware ratio, not
+  a tuned scalar). Not yet implemented.
+
+## Repro
+
+```bash
+# 5-policy, 600s trace (≈18 min/arm, ~90 min total)
+OUTROOT=.../outputs/policy5_600s \
+    bash microbench/connector_tax/cache_sweep/run_5policy_600s.sh
+# unified report
+.venv/bin/python scripts/bench_report.py --root .../outputs/policy5_600s \
+    leastwork unified_ab unified_def lmetric sticky
+```

analysis/mb1/README.md

@@ -0,0 +1,193 @@
+# MB1 — Prefill–Decode Interference (chunked-prefill on, vLLM 0.18.1 default)
+
+Persistent record of the phase-interference microbench used to put a
+quantitative upper bound on **what PD-disaggregation can buy** under the
+chunked-prefill-on baseline. Re-runs append a dated section at the
+bottom; the **Summary** block is what gets cited.
+
+---
+
+## Summary (latest)
+
+| Headline | Value |
+|---|---|
+| Baseline single-stream TPOT (D=1, idle GPU) | **4.8 ms** |
+| Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** |
+| Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** |
+| Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** |
+
+**What MB1 actually measures**:
+
+> During a prefill burst, every ongoing decode stream is essentially
+> halted (per-stream effective TPOT is 15×–2000× baseline, scaling with
+> prefill size). The **total decode time lost per prefill event is
+> `D × T_prefill`** (D concurrent decodes each lose ~T_prefill of useful
+> work). For the trace mean (P ≈ 33k tokens, T_prefill ≈ 4.5 s) at D=8
+> that's **~36 seconds of decode-equivalent work lost per request**.
+> This is the **upper bound on what PD-disaggregation's phase isolation
+> could recover** on the decode side.
+
+**⚠ Correction (2026-05-27)**: an earlier version of this README framed
+the §3.2 PD-disagg argument as "phase-isolation benefit is capped at
+the decode duration of the new request (~50–200 ms), so MB2 transfer
+cost dominates". That framing was wrong. The correct accounting is
+benefit-per-prefill-event = D × T_prefill (aggregate decode time saved
+across all stalled streams), which is **much larger than per-request
+transfer cost**. The actual reason static PD-disagg fails in agentic
+is **D-side KV pool capacity** (`figs/f4b_pdsep_kv_wall.png`), not a
+cost-vs-benefit imbalance on phase isolation. See `RESULTS_SUMMARY.md`
+section 4 for the corrected framing.
+
+## Setup
+
+| Component | Value |
+|---|---|
+| Host | dash1, H20 96 GiB, driver 570.133.20 |
+| Venv | `/home/admin/cpfs/wjh/agentic-kv-fresh/.venv` |
+| vLLM | 0.18.1 official wheel (chunked-prefill default-on, V1 engine) |
+| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct` |
+| Launch flags | `--tensor-parallel-size 1 --enable-prefix-caching --gpu-memory-utilization 0.9 --max-model-len 200000 --max-num-batched-tokens 8192` |
+| kv_connector | **none** (this measures pure single-GPU phase interference; PD-disagg cost lives in MB2) |
+
+## Method
+
+Adapted from `microbench/interference/driver.py`:
+
+1. Start D streaming decode requests on `/v1/chat/completions` with a
+   long max_tokens cap. Discard the first 32 tokens as warmup.
+2. After 1 s, inject one prefill-only request with `max_tokens=1` and
+   an input of `P` synthetic tokens (uuid-seeded for zero prefix-cache
+   reuse). Measure the prefill's TTFT.
+3. Bin the *during-prefill* tokens from each decode stream by whether
+   their wall-clock falls inside `[prefill_inject_ts, prefill_inject_ts +
+   prefill_ttft]`. Report inter-token p50 / p90.
+4. Bin a baseline run (D streams, no prefill injection) the same way.
+
+We additionally compute the **effective per-stream TPOT during the
+prefill burst** as the single most informative summary:
+
+```
+eff_TPOT_during = prefill_ttft_ms / (num_tokens_during_prefill / D)
+```
+
+This is the average rate at which each decode stream produces tokens
+while a prefill is in flight. Compared to baseline TPOT it gives the
+real per-stream throughput penalty (chunked-prefill p50 looks deceptively
+fine because most decode-token intervals during the burst are at normal
+speed; p90 sees the stall but is itself noisy; the effective TPOT is
+the cleanest "average over the whole burst window" number).
+
+## Results — 2026-05-27, dash1 GPU 0, chunk_tokens=8192
+
+3 D × 5 P × 3 reps. Aggregated by `analyze_mb1.py`.
+
+| D | P (tok) | base TPOT (ms) | prefill_ttft (ms) | per-stream tokens during | effective TPOT during (ms) | penalty | max PD-disagg benefit per stream (ms) |
+|--:|--:|--:|--:|--:|--:|--:|--:|
+| 1 |   2 048 | 4.79 |   163 | 4.0 |     41 |     8× |    144 |
+| 1 |   8 192 | 4.78 |   584 | 5.0 |    117 |    24× |    560 |
+| 1 |  32 768 | 4.78 | 4 515 | 5.0 |    903 |   189× |  4 491 |
+| 1 |  65 536 | 4.78 | 15 568 | 5.3 |  2 919 |   610× | 15 542 |
+| 1 | 131 072 | 4.78 | 56 765 | 5.7 | 10 017 | 2 094× | 56 738 |
+| 4 |   2 048 | 5.62 |   138 | 3.9 |     36 |     6× |    117 |
+| 4 |   8 192 | 6.08 |   574 | 4.5 |    128 |    21× |    547 |
+| 4 |  32 768 | 6.09 | 4 529 | 11.9 |    381 |    63× |  4 457 |
+| 4 |  65 536 | 5.85 | 15 587 | 19.8 |    789 |   135× | 15 471 |
+| 4 | 131 072 | 6.27 | 56 697 | 37.4 |  1 517 |   242× | 56 463 |
+| 8 |   2 048 | 7.71 |   143 | 4.5 |     32 |     4× |    109 |
+| 8 |   8 192 | 7.69 |   583 | 5.1 |    114 |    15× |    544 |
+| 8 |  32 768 | 7.42 | 4 520 | 11.7 |    387 |    52× |  4 434 |
+| 8 |  65 536 | 7.67 | 15 615 | 20.6 |    757 |    99× | 15 457 |
+| 8 | 131 072 | 7.74 | 56 991 | 40.2 |  1 419 |   183× | 56 680 |
+
+**Reading the table**:
+
+- *Baseline TPOT* grows mildly with D (4.8 ms → 7.7 ms as D goes 1 → 8).
+  Multi-stream decoding has small but nonzero contention even without
+  prefill.
+- *Effective TPOT during* grows mostly with P: a single 8k prefill stalls
+  decode for ~580 ms regardless of D, so each stream emits only a handful
+  of tokens during that 580 ms window — effective per-stream TPOT
+  collapses to 100–130 ms. Larger prefill = more chunks = larger stall.
+- *Penalty* is the eff/baseline ratio. Above 50× for P ≥ 32k. Above
+  500× for D=1 at P ≥ 65k.
+- *Max PD-disagg benefit per stream* = `prefill_ttft − per_stream_tokens
+  × baseline_TPOT` ≈ `prefill_ttft` (since interference essentially
+  halts decode). This is the entire prefill duration's worth of decode
+  time that could in principle be recovered.
+
+**Connecting to the §3.2 PD-disagg argument** (corrected):
+
+PD-disagg's promised phase-isolation benefit is **per prefill event**,
+not per request. When a new prefill arrives, it stalls every concurrent
+decode stream on the same GPU. The aggregate decode time lost across
+those D streams is `D × T_prefill`. PD-disagg moving prefill off-decode-GPU
+recovers all of it.
+
+Plugging numbers per prefill event:
+
+| Prefill size | T_prefill | PD-disagg cost (MB2 T_transfer) | PD-disagg benefit (D=8 × T_prefill) | Ratio |
+|---:|---:|---:|---:|---:|
+| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s | 0.7 % |
+| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s | 0.9 % |
+| 125k tok (~p99) | 57 s | 1.9 s | 456 s | 0.4 % |
+
+On the **phase-isolation axis alone**, PD-disagg wins by 100×–250×.
+The reason static PD-disagg nonetheless **fails in agentic** is a
+*different* failure mode: the D-side KV pool cannot fit p90+ requests
+(p99 = 11.5 GiB; D-instance pool ≈ 38 GiB; 4P+4D halves system-wide
+decode capacity → TTFT p50 62×, success rate 99.5% → 52% in colleague's
+4P+4D experiment). The structural problem is **capacity** (see
+`figs/f4b_pdsep_kv_wall.png`), not transfer-cost vs phase-isolation
+trade-off.
+
+## Reproduction
+
+```bash
+# vllm pair-free single-instance launch
+ssh dash1 'GPU=0 PORT=8000 CHUNK_TOKENS=8192 \
+  bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh start'
+
+# sweep
+ssh dash1 'source /home/admin/cpfs/wjh/agentic-kv-fresh/.venv/bin/activate && \
+  python /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_driver.py \
+    --host 127.0.0.1 --port 8000 \
+    --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
+    --decode-batch-sizes 1,4,8 --prefill-tokens 2048,8192,32768,65536,131072 \
+    --reps 3 --output-dir /home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results'
+
+# pull + analyze
+scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/summary.csv \
+    analysis/mb1/summary.csv
+.venv/bin/python microbench/fresh_setup/analyze_mb1.py \
+  --summary analysis/mb1/summary.csv --out analysis/mb1/breakdown.json
+.venv/bin/python microbench/fresh_setup/plot_mb1.py \
+  --mb1 analysis/mb1/breakdown.json \
+  --mb2-intra analysis/mb2/intra_kvboth_breakdown.json \
+  --mb2-inter analysis/mb2/inter_kvboth_breakdown.json
+
+# teardown
+ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop'
+```
+
+## Open questions / next runs
+
+- **Chunk size sensitivity**: this run uses `--max-num-batched-tokens
+  8192`. Sarathi-Serve goes smaller (e.g. 1024) and recovers more
+  decode interleaving inside each prefill burst. Worth running
+  chunk_tokens ∈ {1024, 2048, 4096, 16384} to map the chunk-size axis.
+- **Higher D**: 12, 16 streams to see whether the penalty saturates or
+  keeps shrinking per-stream.
+- **Cross-validate effective_TPOT_during with token-time-series plot**:
+  raw per-token timestamps could reveal whether the stall is a few big
+  spikes or many small ones (currently inferred from p50/p90 spread).
+
+## Run log
+
+### 2026-05-27 — dash1 GPU 0, chunk_tokens=8192
+
+3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on
+dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`.
+Figure: `figs/mb1_interference.png`. The figure
+`figs/pd_cost_vs_benefit.png` from the original commit `029821c` was
+based on the wrong "benefit ≤ decode duration" accounting; **deleted in
+the correction commit**.

analysis/mb1/breakdown.json

@@ -0,0 +1,199 @@
+{
+  "summary": [
+    {
+      "decode_batch_size": 1,
+      "new_prefill_tokens": 2048,
+      "baseline_tpot_ms": 4.79,
+      "during_tpot_p50_ms_raw": 35.43,
+      "during_tpot_p90_ms_raw": 79.91,
+      "prefill_ttft_ms": 163.3,
+      "num_tokens_during_prefill_total": 4.0,
+      "per_stream_tokens_during": 4.0,
+      "effective_tpot_during_ms": 40.8,
+      "interference_penalty_x": 8.5,
+      "max_pd_disagg_benefit_ms_per_stream": 144.2
+    },
+    {
+      "decode_batch_size": 1,
+      "new_prefill_tokens": 8192,
+      "baseline_tpot_ms": 4.78,
+      "during_tpot_p50_ms_raw": 6.56,
+      "during_tpot_p90_ms_raw": 328.57,
+      "prefill_ttft_ms": 583.9,
+      "num_tokens_during_prefill_total": 5.0,
+      "per_stream_tokens_during": 5.0,
+      "effective_tpot_during_ms": 116.8,
+      "interference_penalty_x": 24.4,
+      "max_pd_disagg_benefit_ms_per_stream": 560.0
+    },
+    {
+      "decode_batch_size": 1,
+      "new_prefill_tokens": 32768,
+      "baseline_tpot_ms": 4.78,
+      "during_tpot_p50_ms_raw": 4.75,
+      "during_tpot_p90_ms_raw": 4.9,
+      "prefill_ttft_ms": 4515.3,
+      "num_tokens_during_prefill_total": 5.0,
+      "per_stream_tokens_during": 5.0,
+      "effective_tpot_during_ms": 903.1,
+      "interference_penalty_x": 188.8,
+      "max_pd_disagg_benefit_ms_per_stream": 4491.4
+    },
+    {
+      "decode_batch_size": 1,
+      "new_prefill_tokens": 65536,
+      "baseline_tpot_ms": 4.78,
+      "during_tpot_p50_ms_raw": 4.69,
+      "during_tpot_p90_ms_raw": 4.97,
+      "prefill_ttft_ms": 15567.6,
+      "num_tokens_during_prefill_total": 5.3,
+      "per_stream_tokens_during": 5.33,
+      "effective_tpot_during_ms": 2918.9,
+      "interference_penalty_x": 610.2,
+      "max_pd_disagg_benefit_ms_per_stream": 15542.0
+    },
+    {
+      "decode_batch_size": 1,
+      "new_prefill_tokens": 131072,
+      "baseline_tpot_ms": 4.78,
+      "during_tpot_p50_ms_raw": 4.71,
+      "during_tpot_p90_ms_raw": 4.9,
+      "prefill_ttft_ms": 56765.2,
+      "num_tokens_during_prefill_total": 5.7,
+      "per_stream_tokens_during": 5.67,
+      "effective_tpot_during_ms": 10017.4,
+      "interference_penalty_x": 2094.5,
+      "max_pd_disagg_benefit_ms_per_stream": 56738.1
+    },
+    {
+      "decode_batch_size": 4,
+      "new_prefill_tokens": 2048,
+      "baseline_tpot_ms": 5.62,
+      "during_tpot_p50_ms_raw": 22.18,
+      "during_tpot_p90_ms_raw": 84.85,
+      "prefill_ttft_ms": 138.3,
+      "num_tokens_during_prefill_total": 15.5,
+      "per_stream_tokens_during": 3.88,
+      "effective_tpot_during_ms": 35.7,
+      "interference_penalty_x": 6.3,
+      "max_pd_disagg_benefit_ms_per_stream": 116.6
+    },
+    {
+      "decode_batch_size": 4,
+      "new_prefill_tokens": 8192,
+      "baseline_tpot_ms": 6.08,
+      "during_tpot_p50_ms_raw": 8.45,
+      "during_tpot_p90_ms_raw": 515.39,
+      "prefill_ttft_ms": 574.1,
+      "num_tokens_during_prefill_total": 18.0,
+      "per_stream_tokens_during": 4.5,
+      "effective_tpot_during_ms": 127.6,
+      "interference_penalty_x": 21.0,
+      "max_pd_disagg_benefit_ms_per_stream": 546.8
+    },
+    {
+      "decode_batch_size": 4,
+      "new_prefill_tokens": 32768,
+      "baseline_tpot_ms": 6.09,
+      "during_tpot_p50_ms_raw": 9.83,
+      "during_tpot_p90_ms_raw": 1314.87,
+      "prefill_ttft_ms": 4529.1,
+      "num_tokens_during_prefill_total": 47.5,
+      "per_stream_tokens_during": 11.88,
+      "effective_tpot_during_ms": 381.4,
+      "interference_penalty_x": 62.7,
+      "max_pd_disagg_benefit_ms_per_stream": 4456.9
+    },
+    {
+      "decode_batch_size": 4,
+      "new_prefill_tokens": 65536,
+      "baseline_tpot_ms": 5.85,
+      "during_tpot_p50_ms_raw": 6.41,
+      "during_tpot_p90_ms_raw": 2077.47,
+      "prefill_ttft_ms": 15586.5,
+      "num_tokens_during_prefill_total": 79.0,
+      "per_stream_tokens_during": 19.75,
+      "effective_tpot_during_ms": 789.2,
+      "interference_penalty_x": 135.0,
+      "max_pd_disagg_benefit_ms_per_stream": 15471.0
+    },
+    {
+      "decode_batch_size": 4,
+      "new_prefill_tokens": 131072,
+      "baseline_tpot_ms": 6.27,
+      "during_tpot_p50_ms_raw": 6.3,
+      "during_tpot_p90_ms_raw": 4405.18,
+      "prefill_ttft_ms": 56697.1,
+      "num_tokens_during_prefill_total": 149.5,
+      "per_stream_tokens_during": 37.38,
+      "effective_tpot_during_ms": 1517.0,
+      "interference_penalty_x": 241.8,
+      "max_pd_disagg_benefit_ms_per_stream": 56462.6
+    },
+    {
+      "decode_batch_size": 8,
+      "new_prefill_tokens": 2048,
+      "baseline_tpot_ms": 7.71,
+      "during_tpot_p50_ms_raw": 8.38,
+      "during_tpot_p90_ms_raw": 98.98,
+      "prefill_ttft_ms": 143.1,
+      "num_tokens_during_prefill_total": 35.7,
+      "per_stream_tokens_during": 4.46,
+      "effective_tpot_during_ms": 32.1,
+      "interference_penalty_x": 4.2,
+      "max_pd_disagg_benefit_ms_per_stream": 108.8
+    },
+    {
+      "decode_batch_size": 8,
+      "new_prefill_tokens": 8192,
+      "baseline_tpot_ms": 7.69,
+      "during_tpot_p50_ms_raw": 9.34,
+      "during_tpot_p90_ms_raw": 519.29,
+      "prefill_ttft_ms": 583.3,
+      "num_tokens_during_prefill_total": 41.0,
+      "per_stream_tokens_during": 5.12,
+      "effective_tpot_during_ms": 113.8,
+      "interference_penalty_x": 14.8,
+      "max_pd_disagg_benefit_ms_per_stream": 543.9
+    },
+    {
+      "decode_batch_size": 8,
+      "new_prefill_tokens": 32768,
+      "baseline_tpot_ms": 7.42,
+      "during_tpot_p50_ms_raw": 11.61,
+      "during_tpot_p90_ms_raw": 1315.48,
+      "prefill_ttft_ms": 4520.3,
+      "num_tokens_during_prefill_total": 93.3,
+      "per_stream_tokens_during": 11.67,
+      "effective_tpot_during_ms": 387.5,
+      "interference_penalty_x": 52.2,
+      "max_pd_disagg_benefit_ms_per_stream": 4433.7
+    },
+    {
+      "decode_batch_size": 8,
+      "new_prefill_tokens": 65536,
+      "baseline_tpot_ms": 7.67,
+      "during_tpot_p50_ms_raw": 19.09,
+      "during_tpot_p90_ms_raw": 2471.4,
+      "prefill_ttft_ms": 15615.5,
+      "num_tokens_during_prefill_total": 165.0,
+      "per_stream_tokens_during": 20.62,
+      "effective_tpot_during_ms": 757.1,
+      "interference_penalty_x": 98.8,
+      "max_pd_disagg_benefit_ms_per_stream": 15457.4
+    },
+    {
+      "decode_batch_size": 8,
+      "new_prefill_tokens": 131072,
+      "baseline_tpot_ms": 7.74,
+      "during_tpot_p50_ms_raw": 11.51,
+      "during_tpot_p90_ms_raw": 4895.27,
+      "prefill_ttft_ms": 56991.4,
+      "num_tokens_during_prefill_total": 321.3,
+      "per_stream_tokens_during": 40.17,
+      "effective_tpot_during_ms": 1418.9,
+      "interference_penalty_x": 183.3,
+      "max_pd_disagg_benefit_ms_per_stream": 56680.4
+    }
+  ]
+}

analysis/mb1/summary.csv

@@ -0,0 +1,46 @@
+chunk_size,decode_batch_size,new_prefill_tokens,repetition,tpot_baseline_p50_ms,tpot_baseline_p90_ms,tpot_during_prefill_p50_ms,tpot_during_prefill_p90_ms,tpot_after_prefill_p50_ms,prefill_ttft_ms,num_tokens_during_prefill,tpot_penalty_p50_ms,tpot_penalty_ratio
+8192,1,131072,0,4.777565016411245,4.900234832894057,4.701301048044115,4.948397364933044,0.0,56719.25117995124,7,-0.07626396836712956,0.9840370632099913
+8192,1,131072,1,4.779465030878782,4.883405601140112,4.707481013610959,4.85471700085327,0.0,56696.089847013354,5,-0.07198401726782322,0.9849388965495606
+8192,1,131072,2,4.790953011251986,4.880544205661863,4.728371975943446,4.907831805758178,0.0,56880.19039196661,5,-0.06258103530853987,0.9869376645603573
+8192,1,2048,0,4.77885699365288,4.894876398611814,41.434570477576926,88.97331730695441,0.0,183.2046649651602,4,36.655713483924046,8.670393471202205
+8192,1,2048,1,4.788161953911185,4.949774022679776,41.68213551747613,83.5143867880106,0.0,175.55483896285295,4,36.89397356356494,8.705247633369687
+8192,1,2048,2,4.7893429873511195,4.874200583435595,23.186982492916286,67.25202781381086,0.0,131.23180496040732,4,18.397639505565166,4.841370215946989
+8192,1,32768,0,4.789774015080184,4.870833398308605,4.738486022688448,4.886626999359578,0.0,4500.839321000967,5,-0.051287992391735315,0.9892921895207875
+8192,1,32768,1,4.776834975928068,4.891659819986671,4.729953012429178,4.9245511763729155,0.0,4496.073378017172,5,-0.0468819634988904,0.9901855593221991
+8192,1,32768,2,4.784431017469615,4.866032593417913,4.782894975505769,4.8977664206177,0.0,4549.013931944501,5,-0.0015360419638454914,0.9996789499193871
+8192,1,65536,0,4.778854956384748,4.9255444086156785,4.633405013009906,4.895579582080245,0.0,15530.37424501963,5,-0.1454499433748424,0.9695638506080803
+8192,1,65536,1,4.784283053595573,4.8808404128067195,4.754905996378511,4.985795798711479,0.0,15584.887631004676,5,-0.02937705721706152,0.99385967408534
+8192,1,65536,2,4.787993966601789,4.9004736240021884,4.6836750116199255,5.0271204963792115,0.0,15587.390075030271,6,-0.1043189549818635,0.9782123879625725
+8192,1,8192,0,4.785028984770179,4.878618801012635,7.490115996915847,324.06569679733366,0.0,573.2795029762201,5,2.7050870121456683,1.565323014919123
+8192,1,8192,1,4.778591974172741,4.899543372448534,5.9131429879926145,336.8099076091312,0.0,606.6823820001446,5,1.1345510138198733,1.237423705550061
+8192,1,8192,2,4.78826800826937,4.90188361145556,6.276679981965572,324.8370993998833,0.0,571.7499859747477,5,1.488411973696202,1.310845585736994
+8192,4,131072,0,6.113810988608748,6.309205386787653,0.0,0.0,0.0,56702.702289039735,0,-6.113810988608748,0.0
+8192,4,131072,1,6.630807969486341,7.086459483252838,6.2820459716022015,4400.500871409893,0.0,56807.70832300186,150,-0.3487619978841394,0.9474027902045915
+8192,4,131072,2,6.073819473385811,6.344516028184444,6.326125003397465,4409.856556192978,0.0,56580.784838995896,149,0.2523055300116539,1.0415398467335428
+8192,4,2048,0,5.402160517405719,5.543816485442221,6.210724503034726,84.62208869168535,6.125201500253752,140.3041940066032,18,0.8085639856290072,1.1496741873966574
+8192,4,2048,1,6.067108013667166,6.381415005307645,0.0,0.0,0.0,140.06177097326145,0,-6.067108013667166,0.0
+8192,4,2048,2,5.400336522143334,5.536347016459331,38.15686801681295,85.07051098858938,5.25214200024493,134.67552902875468,13,32.756531494669616,7.065646346363043
+8192,4,32768,0,6.115561525803059,6.369604001520202,7.216634490760043,1314.6978712815326,5.17624247004278,4522.433568025008,50,1.101072964956984,1.1800444587649532
+8192,4,32768,1,6.070095987524837,6.3612310332246125,0.0,0.0,0.0,4508.074064040557,0,-6.070095987524837,0.0
+8192,4,32768,2,6.0734800063073635,6.312666402664036,12.442811043001711,1315.0411327951588,4.754714027512819,4556.892123946454,45,6.369331036694348,2.0487119460473635
+8192,4,65536,0,5.406292999396101,5.540905491216108,0.0,0.0,0.0,15581.590663990937,0,-5.406292999396101,0.0
+8192,4,65536,1,6.076910009142011,6.315114628523588,0.0,0.0,0.0,15574.196094006766,0,-6.076910009142011,0.0
+8192,4,65536,2,6.060379033442587,6.384042033459991,6.411670008674264,2077.4700703914277,4.8022730043157935,15603.720718005206,79,0.3512909752316773,1.0579651822589267
+8192,4,8192,0,6.110575021011755,6.416070973500609,8.451583969872445,515.3855616226792,5.358011490898207,574.6672929963097,18,2.34100894886069,1.3831077993169092
+8192,4,8192,1,6.051429023500532,6.398122606333345,0.0,0.0,0.0,573.6081749782898,0,-6.051429023500532,0.0
+8192,4,8192,2,6.064729997888207,6.366449000779539,0.0,0.0,0.0,574.1707819979638,0,-6.064729997888207,0.0
+8192,8,131072,0,7.737616979284212,7.99839201499708,10.740376019384712,4742.438135773409,7.792441989295185,57010.66731195897,335,3.0027590401005,1.388072845701685
+8192,8,131072,1,7.744895527139306,8.013638522243127,8.647068490972742,5123.228083999129,7.672236970392987,56970.40947602363,310,0.9021729638334364,1.116486137310966
+8192,8,131072,2,7.740180502878502,8.016240986762568,15.140031988266855,4820.136589207682,7.68946303287521,56993.02393599646,319,7.3998514853883535,1.9560308680962177
+8192,8,2048,0,7.741285488009453,8.022559515666217,8.103576023131609,124.87094267853536,7.6825070136692375,141.97922096354887,30,0.36229053512215614,1.046799789993963
+8192,8,2048,1,7.728310010861605,8.021069981623441,8.17067950265482,84.82906777062453,7.745136506855488,144.1582590341568,38,0.4423694917932153,1.0572401328584768
+8192,8,2048,2,7.662211020942777,8.034424972720444,8.87883099494502,87.23540699575096,7.592331967316568,143.27958395006135,39,1.216619974002242,1.1587818412566437
+8192,8,32768,0,7.295333489309996,7.422819995554164,11.429400008637458,1315.43214758276,7.8034960315562785,4523.641717038117,94,4.134066519327462,1.5666727265292526
+8192,8,32768,1,7.278127042809501,7.490781514206901,12.640403030673042,1315.491412486881,7.821676495950669,4519.993302994408,90,5.362275987863541,1.736765922925357
+8192,8,32768,2,7.684049021918327,8.047712198458612,10.752685484476388,1315.5166705255397,7.80402502277866,4517.200137954205,96,3.068636462558061,1.3993514947399404
+8192,8,65536,0,7.708174001891166,8.017168991500512,26.662671996746212,2496.8427699001018,7.768569514155388,15603.601168957539,160,18.954497994855046,3.459012729889679
+8192,8,65536,1,7.594842027174309,7.9874323040712625,13.054963492322713,2459.1690181812737,7.54699349636212,15620.474929979537,174,5.460121465148404,1.7189249553331216
+8192,8,65536,2,7.693717983784154,7.933055714238435,17.5579380011186,2458.176895044744,7.808708498487249,15622.32490995666,161,9.864220017334446,2.2821135422594123
+8192,8,8192,0,7.636573514901102,7.904737605713308,10.151655005756766,514.8188057704829,7.7977380133233964,575.7745200535282,37,2.515081490855664,1.3293468577167538
+8192,8,8192,1,7.687711506150663,7.965393498307094,9.002390026580542,524.0793236298487,7.753994490485638,592.1044679707848,45,1.3146785204298794,1.1710103870804793
+8192,8,8192,2,7.756220467854291,8.035426988499239,8.864110975991935,518.9726910321042,7.770269992761314,581.98908099439,41,1.1078905081376433,1.1428389655411813

analysis/mb2/A_inter_kvboth.jsonl

@@ -0,0 +1,51 @@
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 50331648, "duration_s": 0.06580113701056689, "t_start_unix": 1779885615.6732209, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885615.7390358}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 50331648, "duration_s": 0.0052392969955690205, "t_start_unix": 1779885616.0322638, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885616.0375087}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 201326592, "duration_s": 0.02050818904535845, "t_start_unix": 1779885616.2556505, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885616.2761638}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 201326592, "duration_s": 0.02001398801803589, "t_start_unix": 1779885616.4400308, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885616.46005}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 805306368, "duration_s": 0.08249958901433274, "t_start_unix": 1779885617.072654, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885617.1551628}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 805306368, "duration_s": 0.08082435996038839, "t_start_unix": 1779885617.7853239, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885617.866155}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.00040365802124142647, "t_start_unix": 1779885642.3123364, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.3127441}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.000374739000108093, "t_start_unix": 1779885642.3945863, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.3949661}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 50331648, "duration_s": 0.005158354004379362, "t_start_unix": 1779885642.4766958, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.481858}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 50331648, "duration_s": 0.00513053999748081, "t_start_unix": 1779885642.5614145, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.5665495}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 50331648, "duration_s": 0.0051341859507374465, "t_start_unix": 1779885642.6461189, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.6512585}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 100663296, "duration_s": 0.010125375993084162, "t_start_unix": 1779885642.7582293, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.76836}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 100663296, "duration_s": 0.010367848037276417, "t_start_unix": 1779885642.878179, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.8885527}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 100663296, "duration_s": 0.010208865976892412, "t_start_unix": 1779885642.997267, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.0074801}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 100663296, "duration_s": 0.01015280099818483, "t_start_unix": 1779885643.1138487, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.124007}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 100663296, "duration_s": 0.010063701018225402, "t_start_unix": 1779885643.2299926, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.2400591}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.00040220399387180805, "t_start_unix": 1779885643.4041483, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.4045541}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.0003751559997908771, "t_start_unix": 1779885643.569847, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.5702271}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 201326592, "duration_s": 0.020166583999525756, "t_start_unix": 1779885643.7347023, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.7548745}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 201326592, "duration_s": 0.019989027990959585, "t_start_unix": 1779885643.9190295, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.939023}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 201326592, "duration_s": 0.02000429102918133, "t_start_unix": 1779885644.1027336, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885644.122742}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 402653184, "duration_s": 0.040499016002286226, "t_start_unix": 1779885644.419112, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885644.4596438}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 402653184, "duration_s": 0.04015034798067063, "t_start_unix": 1779885644.761118, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885644.8012745}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 402653184, "duration_s": 0.04006708099041134, "t_start_unix": 1779885645.1008255, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885645.1408994}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 402653184, "duration_s": 0.04012463602703065, "t_start_unix": 1779885645.440819, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885645.4809544}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 402653184, "duration_s": 0.04009692999534309, "t_start_unix": 1779885645.7793777, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885645.8194828}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.00046314700739458203, "t_start_unix": 1779885646.445756, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885646.4462252}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.0004179630195721984, "t_start_unix": 1779885647.0757725, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885647.0761962}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 805306368, "duration_s": 0.08170936699025333, "t_start_unix": 1779885647.7075222, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885647.7892444}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 805306368, "duration_s": 0.08086105203256011, "t_start_unix": 1779885648.422354, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885648.503224}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 805306368, "duration_s": 0.08071460900828242, "t_start_unix": 1779885649.1382625, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885649.2189863}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1610612736, "duration_s": 0.1638482889975421, "t_start_unix": 1779885650.8273866, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885650.991251}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1610612736, "duration_s": 0.16476035403320566, "t_start_unix": 1779885652.5951493, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885652.7599196}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1610612736, "duration_s": 0.1617715489701368, "t_start_unix": 1779885654.3595936, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885654.5214472}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1610612736, "duration_s": 0.1616577100357972, "t_start_unix": 1779885656.125777, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885656.2874432}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1610612736, "duration_s": 0.1617818950326182, "t_start_unix": 1779885657.8867118, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885658.0485108}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 3221225472, "duration_s": 0.30771408596774563, "t_start_unix": 1779885662.69878, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885663.0065253}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 3221225472, "duration_s": 0.3096057590446435, "t_start_unix": 1779885667.6915898, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885668.0012283}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 3221225472, "duration_s": 0.3080696280230768, "t_start_unix": 1779885672.684307, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885672.9923992}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 3221225472, "duration_s": 0.5257709489669651, "t_start_unix": 1779885677.7157974, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885678.2415998}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 3221225472, "duration_s": 0.5268570999614894, "t_start_unix": 1779885682.9683614, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885683.4952588}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 6442450944, "duration_s": 1.901308581000194, "t_start_unix": 1779885698.9342654, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885700.835605}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 6442450944, "duration_s": 1.9211939970264211, "t_start_unix": 1779885716.3570645, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885718.2782962}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 6442450944, "duration_s": 1.7335722800344229, "t_start_unix": 1779885733.838071, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885735.571683}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 6442450944, "duration_s": 0.9299940629862249, "t_start_unix": 1779885751.0288215, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885751.9588547}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 6442450944, "duration_s": 0.6535220990190282, "t_start_unix": 1779885767.3456392, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885767.9991918}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 12884901888, "duration_s": 1.2830275790183805, "t_start_unix": 1779885822.9884846, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885824.2715507}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 12884901888, "duration_s": 5.053741328010801, "t_start_unix": 1779885879.5277712, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885884.5815506}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 12884901888, "duration_s": 9.158571614010725, "t_start_unix": 1779885939.9274101, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885949.086019}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 12884901888, "duration_s": 2.8184301540022716, "t_start_unix": 1779886004.3031003, "ret": 0, "tp_rank": 0, "t_log_unix": 1779886007.121565}
+{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 12884901888, "duration_s": 1.4530277770245448, "t_start_unix": 1779886062.159227, "ret": 0, "tp_rank": 0, "t_log_unix": 1779886063.6122832}

analysis/mb2/A_intra_kvboth.jsonl

@@ -0,0 +1,51 @@
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 50331648, "duration_s": 0.023202952987048775, "t_start_unix": 1779879143.174031, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879143.1972432}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 50331648, "duration_s": 0.005375694017857313, "t_start_unix": 1779879143.2982283, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879143.3036084}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 201326592, "duration_s": 0.021170366962905973, "t_start_unix": 1779879143.5159554, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879143.5371296}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 201326592, "duration_s": 0.020726953051052988, "t_start_unix": 1779879143.6974514, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879143.7181835}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 805306368, "duration_s": 0.08536655298667029, "t_start_unix": 1779879144.3294952, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879144.4148676}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 805306368, "duration_s": 0.08367906499188393, "t_start_unix": 1779879145.0419943, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879145.125678}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.0004059679922647774, "t_start_unix": 1779879221.7078288, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879221.7082384}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.000346789020113647, "t_start_unix": 1779879221.7838593, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879221.7842083}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 50331648, "duration_s": 0.005353622022084892, "t_start_unix": 1779879221.8607252, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879221.8660822}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 50331648, "duration_s": 0.005279594974126667, "t_start_unix": 1779879221.9432015, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879221.9484842}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 50331648, "duration_s": 0.0053006180096417665, "t_start_unix": 1779879222.0243337, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.0296378}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 100663296, "duration_s": 0.010396577999927104, "t_start_unix": 1779879222.130936, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.141335}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 100663296, "duration_s": 0.010438029014039785, "t_start_unix": 1779879222.2442062, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.2546473}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 100663296, "duration_s": 0.010436972021125257, "t_start_unix": 1779879222.3581295, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.3685696}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 100663296, "duration_s": 0.010396371013484895, "t_start_unix": 1779879222.4725878, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.4829867}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 100663296, "duration_s": 0.010352785000577569, "t_start_unix": 1779879222.5837166, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.5940716}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.00034007197245955467, "t_start_unix": 1779879222.7521152, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.752458}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.00041691696969792247, "t_start_unix": 1779879222.9143836, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.914805}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 201326592, "duration_s": 0.020633380976505578, "t_start_unix": 1779879223.0778644, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879223.098502}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 201326592, "duration_s": 0.020639199996367097, "t_start_unix": 1779879223.2603853, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879223.2810278}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 201326592, "duration_s": 0.020575353992171586, "t_start_unix": 1779879223.4418828, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879223.4624615}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 402653184, "duration_s": 0.041439525957684964, "t_start_unix": 1779879223.7544343, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879223.7958782}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 402653184, "duration_s": 0.04152030003024265, "t_start_unix": 1779879224.0914912, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879224.133016}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 402653184, "duration_s": 0.04148670402355492, "t_start_unix": 1779879224.4262393, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879224.4677298}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 402653184, "duration_s": 0.04146742797456682, "t_start_unix": 1779879224.7617002, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879224.8031723}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 402653184, "duration_s": 0.04143296502297744, "t_start_unix": 1779879225.0978234, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879225.1392617}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.0003991159610450268, "t_start_unix": 1779879225.760789, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879225.7611918}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.00041423802031204104, "t_start_unix": 1779879226.3864496, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879226.3868673}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 805306368, "duration_s": 0.08309489500243217, "t_start_unix": 1779879227.0107942, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879227.0938945}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 805306368, "duration_s": 0.08372796402545646, "t_start_unix": 1779879227.7207224, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879227.8044555}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 805306368, "duration_s": 0.08398396399570629, "t_start_unix": 1779879228.4314566, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879228.5154452}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1610612736, "duration_s": 0.16950496198842302, "t_start_unix": 1779879230.1334376, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879230.3029544}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1610612736, "duration_s": 0.16713789198547602, "t_start_unix": 1779879231.8981037, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879232.0652575}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1610612736, "duration_s": 0.16713115200400352, "t_start_unix": 1779879233.6608078, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879233.827945}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1610612736, "duration_s": 0.16709016199456528, "t_start_unix": 1779879235.419875, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879235.5869706}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1610612736, "duration_s": 0.166486973001156, "t_start_unix": 1779879237.1821773, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879237.34867}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 3221225472, "duration_s": 0.31926770601421595, "t_start_unix": 1779879241.9880297, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879242.3073065}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 3221225472, "duration_s": 0.3197040680097416, "t_start_unix": 1779879246.9779432, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879247.2976692}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 3221225472, "duration_s": 0.32088329299585894, "t_start_unix": 1779879251.9643052, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879252.285209}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 3221225472, "duration_s": 0.5439103110111319, "t_start_unix": 1779879256.9989722, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879257.5428913}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 3221225472, "duration_s": 0.5193864739849232, "t_start_unix": 1779879262.2562187, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879262.7756212}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 6442450944, "duration_s": 1.9844180009677075, "t_start_unix": 1779879278.199048, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879280.1834915}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 6442450944, "duration_s": 2.1099297259934247, "t_start_unix": 1779879295.6967168, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879297.8066647}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 6442450944, "duration_s": 1.8950715209939517, "t_start_unix": 1779879313.3236735, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879315.2187643}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 6442450944, "duration_s": 0.9277855920372531, "t_start_unix": 1779879330.6715357, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879331.599329}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 6442450944, "duration_s": 0.6652462020283565, "t_start_unix": 1779879346.9950044, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879347.6602724}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 12884901888, "duration_s": 1.3330365709844045, "t_start_unix": 1779879402.7169023, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879404.04997}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 12884901888, "duration_s": 5.839069904992357, "t_start_unix": 1779879459.0566247, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879464.8957155}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 12884901888, "duration_s": 9.862486142024864, "t_start_unix": 1779879519.9567635, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879529.8192694}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 12884901888, "duration_s": 2.8350498770014383, "t_start_unix": 1779879584.9780834, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879587.813154}
+{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 12884901888, "duration_s": 1.485496642999351, "t_start_unix": 1779879642.639775, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879644.1252885}

analysis/mb2/B_intra_kvboth.jsonl

@@ -0,0 +1,102 @@
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ad00672f263a6643-0-9479211a"], "t_start_unix": 1779879143.1678784, "tp_rank": 0, "t_log_unix": 1779879143.167884}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ad00672f263a6643-0-9479211a"], "duration_s": 0.03333390498301014, "t_start_unix": 1779879143.1678784, "tp_rank": 0, "t_log_unix": 1779879143.201217}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ace77e2b02f9f141-0-b3c061bc"], "t_start_unix": 1779879143.2968972, "tp_rank": 0, "t_log_unix": 1779879143.2969005}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ace77e2b02f9f141-0-b3c061bc"], "duration_s": 0.007019245007541031, "t_start_unix": 1779879143.2968972, "tp_rank": 0, "t_log_unix": 1779879143.3039184}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4a2366879c68ded-0-8ac4098e"], "t_start_unix": 1779879143.5146625, "tp_rank": 0, "t_log_unix": 1779879143.5146651}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4a2366879c68ded-0-8ac4098e"], "duration_s": 0.02278437599306926, "t_start_unix": 1779879143.5146625, "tp_rank": 0, "t_log_unix": 1779879143.537448}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8690cafcace0d5e2-0-b89f33d2"], "t_start_unix": 1779879143.6958342, "tp_rank": 0, "t_log_unix": 1779879143.6958375}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8690cafcace0d5e2-0-b89f33d2"], "duration_s": 0.022794076008722186, "t_start_unix": 1779879143.6958342, "tp_rank": 0, "t_log_unix": 1779879143.71863}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b087e2ec4cfa8eb7-0-b908f425"], "t_start_unix": 1779879144.3279662, "tp_rank": 0, "t_log_unix": 1779879144.3279696}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b087e2ec4cfa8eb7-0-b908f425"], "duration_s": 0.08753501297906041, "t_start_unix": 1779879144.3279662, "tp_rank": 0, "t_log_unix": 1779879144.415505}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a115d16ff5575e08-0-9fa81984"], "t_start_unix": 1779879145.040141, "tp_rank": 0, "t_log_unix": 1779879145.0401456}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a115d16ff5575e08-0-9fa81984"], "duration_s": 0.0860149699728936, "t_start_unix": 1779879145.040141, "tp_rank": 0, "t_log_unix": 1779879145.1261594}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9e585ed083951df5-0-b03f812b"], "t_start_unix": 1779879221.7062025, "tp_rank": 0, "t_log_unix": 1779879221.7062056}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9e585ed083951df5-0-b03f812b"], "duration_s": 0.002459956973325461, "t_start_unix": 1779879221.7062025, "tp_rank": 0, "t_log_unix": 1779879221.7086644}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9271d403c044eadd-0-9c3c4639"], "t_start_unix": 1779879221.7826598, "tp_rank": 0, "t_log_unix": 1779879221.782662}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9271d403c044eadd-0-9c3c4639"], "duration_s": 0.0020201010047458112, "t_start_unix": 1779879221.7826598, "tp_rank": 0, "t_log_unix": 1779879221.7846813}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82a580cefd3e2440-0-a383c3c4"], "t_start_unix": 1779879221.859549, "tp_rank": 0, "t_log_unix": 1779879221.8595514}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82a580cefd3e2440-0-a383c3c4"], "duration_s": 0.006836243963334709, "t_start_unix": 1779879221.859549, "tp_rank": 0, "t_log_unix": 1779879221.8663864}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a31cb4bc9e7f63d2-0-8f48aacd"], "t_start_unix": 1779879221.9419758, "tp_rank": 0, "t_log_unix": 1779879221.9419782}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a31cb4bc9e7f63d2-0-8f48aacd"], "duration_s": 0.00694335694424808, "t_start_unix": 1779879221.9419758, "tp_rank": 0, "t_log_unix": 1779879221.9489205}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a9dfc1a5b425d994-0-a0930098"], "t_start_unix": 1779879222.0232244, "tp_rank": 0, "t_log_unix": 1779879222.0232272}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a9dfc1a5b425d994-0-a0930098"], "duration_s": 0.006697195000015199, "t_start_unix": 1779879222.0232244, "tp_rank": 0, "t_log_unix": 1779879222.0299227}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9712857755af2efc-0-90b2dc9b"], "t_start_unix": 1779879222.1297998, "tp_rank": 0, "t_log_unix": 1779879222.1298025}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9712857755af2efc-0-90b2dc9b"], "duration_s": 0.01183948403922841, "t_start_unix": 1779879222.1297998, "tp_rank": 0, "t_log_unix": 1779879222.1416407}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4f0a10dee65acbe-0-a3c132fc"], "t_start_unix": 1779879222.243023, "tp_rank": 0, "t_log_unix": 1779879222.243025}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4f0a10dee65acbe-0-a3c132fc"], "duration_s": 0.01214482297655195, "t_start_unix": 1779879222.243023, "tp_rank": 0, "t_log_unix": 1779879222.2551687}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4c514b80b52a3f2-0-bcd24f8e"], "t_start_unix": 1779879222.3569698, "tp_rank": 0, "t_log_unix": 1779879222.356972}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4c514b80b52a3f2-0-bcd24f8e"], "duration_s": 0.011961110983975232, "t_start_unix": 1779879222.3569698, "tp_rank": 0, "t_log_unix": 1779879222.368932}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ac7118d8090d181c-0-8af4adf0"], "t_start_unix": 1779879222.4715128, "tp_rank": 0, "t_log_unix": 1779879222.4715152}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ac7118d8090d181c-0-8af4adf0"], "duration_s": 0.011788576026447117, "t_start_unix": 1779879222.4715128, "tp_rank": 0, "t_log_unix": 1779879222.4833028}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-85291bcb93aaf638-0-868db1a8"], "t_start_unix": 1779879222.5826046, "tp_rank": 0, "t_log_unix": 1779879222.5826073}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-85291bcb93aaf638-0-868db1a8"], "duration_s": 0.0118055299972184, "t_start_unix": 1779879222.5826046, "tp_rank": 0, "t_log_unix": 1779879222.5944116}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a448cf2e059ba0c9-0-a1360796"], "t_start_unix": 1779879222.750828, "tp_rank": 0, "t_log_unix": 1779879222.7508304}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a448cf2e059ba0c9-0-a1360796"], "duration_s": 0.0021119200391694903, "t_start_unix": 1779879222.750828, "tp_rank": 0, "t_log_unix": 1779879222.7529414}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b486fd9e945a4658-0-8bb561cd"], "t_start_unix": 1779879222.913044, "tp_rank": 0, "t_log_unix": 1779879222.9130466}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b486fd9e945a4658-0-8bb561cd"], "duration_s": 0.0022232600022107363, "t_start_unix": 1779879222.913044, "tp_rank": 0, "t_log_unix": 1779879222.9152684}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82da2bfe65f276c6-0-88d9a9a2"], "t_start_unix": 1779879223.0765986, "tp_rank": 0, "t_log_unix": 1779879223.0766027}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82da2bfe65f276c6-0-88d9a9a2"], "duration_s": 0.022250515001360327, "t_start_unix": 1779879223.0765986, "tp_rank": 0, "t_log_unix": 1779879223.0988505}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-93bd777652eba5f3-0-9ec3d058"], "t_start_unix": 1779879223.2591784, "tp_rank": 0, "t_log_unix": 1779879223.2591808}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-93bd777652eba5f3-0-9ec3d058"], "duration_s": 0.022157608007546514, "t_start_unix": 1779879223.2591784, "tp_rank": 0, "t_log_unix": 1779879223.2813375}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81f950480a3cabf9-0-bbf8584f"], "t_start_unix": 1779879223.4402068, "tp_rank": 0, "t_log_unix": 1779879223.440209}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81f950480a3cabf9-0-bbf8584f"], "duration_s": 0.022589912987314165, "t_start_unix": 1779879223.4402068, "tp_rank": 0, "t_log_unix": 1779879223.4627984}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b109ed06b5882659-0-8d14993c"], "t_start_unix": 1779879223.7529812, "tp_rank": 0, "t_log_unix": 1779879223.752984}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b109ed06b5882659-0-8d14993c"], "duration_s": 0.043345845013391227, "t_start_unix": 1779879223.7529812, "tp_rank": 0, "t_log_unix": 1779879223.796329}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8a57776c81d64b2c-0-ace8fb2b"], "t_start_unix": 1779879224.0899644, "tp_rank": 0, "t_log_unix": 1779879224.0899673}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8a57776c81d64b2c-0-ace8fb2b"], "duration_s": 0.04341953102266416, "t_start_unix": 1779879224.0899644, "tp_rank": 0, "t_log_unix": 1779879224.1333857}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9b1a5dce18758450-0-b17b3649"], "t_start_unix": 1779879224.424807, "tp_rank": 0, "t_log_unix": 1779879224.42481}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9b1a5dce18758450-0-b17b3649"], "duration_s": 0.04336977802449837, "t_start_unix": 1779879224.424807, "tp_rank": 0, "t_log_unix": 1779879224.4681823}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8c7d412b85f43ed7-0-9dea4add"], "t_start_unix": 1779879224.7599711, "tp_rank": 0, "t_log_unix": 1779879224.7599735}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8c7d412b85f43ed7-0-9dea4add"], "duration_s": 0.043769759009592235, "t_start_unix": 1779879224.7599711, "tp_rank": 0, "t_log_unix": 1779879224.8037443}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8860308db3f010a5-0-ad51eb46"], "t_start_unix": 1779879225.0962389, "tp_rank": 0, "t_log_unix": 1779879225.0962446}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8860308db3f010a5-0-ad51eb46"], "duration_s": 0.043612666020635515, "t_start_unix": 1779879225.0962389, "tp_rank": 0, "t_log_unix": 1779879225.1398532}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-86cca1a2b9427801-0-ba41ade7"], "t_start_unix": 1779879225.7592747, "tp_rank": 0, "t_log_unix": 1779879225.759278}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-86cca1a2b9427801-0-ba41ade7"], "duration_s": 0.002386144013144076, "t_start_unix": 1779879225.7592747, "tp_rank": 0, "t_log_unix": 1779879225.7616625}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a208c6d804293be7-0-94d265ab"], "t_start_unix": 1779879226.384918, "tp_rank": 0, "t_log_unix": 1779879226.384921}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a208c6d804293be7-0-94d265ab"], "duration_s": 0.0023903060355223715, "t_start_unix": 1779879226.384918, "tp_rank": 0, "t_log_unix": 1779879226.3873098}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b53bea2317cc1211-0-8fcad8a8"], "t_start_unix": 1779879227.0092332, "tp_rank": 0, "t_log_unix": 1779879227.0092363}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b53bea2317cc1211-0-8fcad8a8"], "duration_s": 0.08524628396844491, "t_start_unix": 1779879227.0092332, "tp_rank": 0, "t_log_unix": 1779879227.094482}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9daf909593bbdf03-0-8fd7d50e"], "t_start_unix": 1779879227.7190688, "tp_rank": 0, "t_log_unix": 1779879227.719072}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9daf909593bbdf03-0-8fd7d50e"], "duration_s": 0.08596085698809475, "t_start_unix": 1779879227.7190688, "tp_rank": 0, "t_log_unix": 1779879227.8050315}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9ef40f3b6d736128-0-8e8e1c30"], "t_start_unix": 1779879228.4297745, "tp_rank": 0, "t_log_unix": 1779879228.4297774}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9ef40f3b6d736128-0-8e8e1c30"], "duration_s": 0.0860762019874528, "t_start_unix": 1779879228.4297745, "tp_rank": 0, "t_log_unix": 1779879228.5158527}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-851e5d7e3e83d7ea-0-a66a5e0b"], "t_start_unix": 1779879230.131392, "tp_rank": 0, "t_log_unix": 1779879230.1313956}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-851e5d7e3e83d7ea-0-a66a5e0b"], "duration_s": 0.1721468890318647, "t_start_unix": 1779879230.131392, "tp_rank": 0, "t_log_unix": 1779879230.3035412}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9be12af6a9ccccf5-0-af1230c7"], "t_start_unix": 1779879231.896075, "tp_rank": 0, "t_log_unix": 1779879231.896078}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9be12af6a9ccccf5-0-af1230c7"], "duration_s": 0.16974544001277536, "t_start_unix": 1779879231.896075, "tp_rank": 0, "t_log_unix": 1779879232.0658224}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b61b9b237366297b-0-9832f0e3"], "t_start_unix": 1779879233.6589305, "tp_rank": 0, "t_log_unix": 1779879233.6589334}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b61b9b237366297b-0-9832f0e3"], "duration_s": 0.16975757898762822, "t_start_unix": 1779879233.6589305, "tp_rank": 0, "t_log_unix": 1779879233.8286898}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bae0d0efe47ece8f-0-affbc685"], "t_start_unix": 1779879235.4181106, "tp_rank": 0, "t_log_unix": 1779879235.418114}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bae0d0efe47ece8f-0-affbc685"], "duration_s": 0.1695251659839414, "t_start_unix": 1779879235.4181106, "tp_rank": 0, "t_log_unix": 1779879235.587638}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a34bc73c9cd2efc1-0-90d647fc"], "t_start_unix": 1779879237.1803744, "tp_rank": 0, "t_log_unix": 1779879237.1803775}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a34bc73c9cd2efc1-0-90d647fc"], "duration_s": 0.16962904302636161, "t_start_unix": 1779879237.1803744, "tp_rank": 0, "t_log_unix": 1779879237.3500054}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-89a36c12ee6b0ff3-0-9fddbc0f"], "t_start_unix": 1779879241.9859307, "tp_rank": 0, "t_log_unix": 1779879241.9859338}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-89a36c12ee6b0ff3-0-9fddbc0f"], "duration_s": 0.32203804596792907, "t_start_unix": 1779879241.9859307, "tp_rank": 0, "t_log_unix": 1779879242.3079708}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8d65512eb7e3c36c-0-8b23597c"], "t_start_unix": 1779879246.9755645, "tp_rank": 0, "t_log_unix": 1779879246.9755676}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8d65512eb7e3c36c-0-8b23597c"], "duration_s": 0.3227974839974195, "t_start_unix": 1779879246.9755645, "tp_rank": 0, "t_log_unix": 1779879247.2983644}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a13c271ecbbca78b-0-b76a0370"], "t_start_unix": 1779879251.9618897, "tp_rank": 0, "t_log_unix": 1779879251.961893}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a13c271ecbbca78b-0-b76a0370"], "duration_s": 0.3240378479822539, "t_start_unix": 1779879251.9618897, "tp_rank": 0, "t_log_unix": 1779879252.2859304}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bada04ec8c556aca-0-a263d637"], "t_start_unix": 1779879256.9512377, "tp_rank": 0, "t_log_unix": 1779879256.9512408}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bada04ec8c556aca-0-a263d637"], "duration_s": 0.5924434679909609, "t_start_unix": 1779879256.9512377, "tp_rank": 0, "t_log_unix": 1779879257.5436878}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9641a077022e6123-0-8c3c0975"], "t_start_unix": 1779879262.2127163, "tp_rank": 0, "t_log_unix": 1779879262.2127194}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9641a077022e6123-0-8c3c0975"], "duration_s": 0.5644763479940593, "t_start_unix": 1779879262.2127163, "tp_rank": 0, "t_log_unix": 1779879262.777195}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bb3a4e5084af8c3a-0-bdfa0931"], "t_start_unix": 1779879278.1063075, "tp_rank": 0, "t_log_unix": 1779879278.106311}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bb3a4e5084af8c3a-0-bdfa0931"], "duration_s": 2.0784930550144054, "t_start_unix": 1779879278.1063075, "tp_rank": 0, "t_log_unix": 1779879280.1848085}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-91b951f85c93a71b-0-8396bee5"], "t_start_unix": 1779879295.600993, "tp_rank": 0, "t_log_unix": 1779879295.6009963}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-91b951f85c93a71b-0-8396bee5"], "duration_s": 2.2067435560165904, "t_start_unix": 1779879295.600993, "tp_rank": 0, "t_log_unix": 1779879297.8077443}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81d236ecb6aadadf-0-ac184d51"], "t_start_unix": 1779879313.2315958, "tp_rank": 0, "t_log_unix": 1779879313.2315989}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81d236ecb6aadadf-0-ac184d51"], "duration_s": 1.9879729640088044, "t_start_unix": 1779879313.2315958, "tp_rank": 0, "t_log_unix": 1779879315.219571}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4c76c62b44c4295-0-b007a6ed"], "t_start_unix": 1779879330.6154163, "tp_rank": 0, "t_log_unix": 1779879330.6154196}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4c76c62b44c4295-0-b007a6ed"], "duration_s": 0.9849357060156763, "t_start_unix": 1779879330.6154163, "tp_rank": 0, "t_log_unix": 1779879331.6003594}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a06d4b774a8af9a5-0-980e9d23"], "t_start_unix": 1779879346.990221, "tp_rank": 0, "t_log_unix": 1779879346.9902246}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a06d4b774a8af9a5-0-980e9d23"], "duration_s": 0.6725030990201049, "t_start_unix": 1779879346.990221, "tp_rank": 0, "t_log_unix": 1779879347.6627269}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bf0d435e06e3349f-0-8507c933"], "t_start_unix": 1779879402.7123013, "tp_rank": 0, "t_log_unix": 1779879402.7123044}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bf0d435e06e3349f-0-8507c933"], "duration_s": 1.3384539679973386, "t_start_unix": 1779879402.7123013, "tp_rank": 0, "t_log_unix": 1779879404.0507588}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f87ae0fb0c7eec8-0-a8a1daea"], "t_start_unix": 1779879458.9232886, "tp_rank": 0, "t_log_unix": 1779879458.9232917}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f87ae0fb0c7eec8-0-a8a1daea"], "duration_s": 5.973284716019407, "t_start_unix": 1779879458.9232886, "tp_rank": 0, "t_log_unix": 1779879464.896582}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a62e48e40e6c6ad7-0-acca9741"], "t_start_unix": 1779879519.7647448, "tp_rank": 0, "t_log_unix": 1779879519.7647479}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a62e48e40e6c6ad7-0-acca9741"], "duration_s": 10.056511385017075, "t_start_unix": 1779879519.7647448, "tp_rank": 0, "t_log_unix": 1779879529.8212643}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-824479d53bab40e4-0-af951a11"], "t_start_unix": 1779879584.888362, "tp_rank": 0, "t_log_unix": 1779879584.8883653}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-824479d53bab40e4-0-af951a11"], "duration_s": 2.925714804965537, "t_start_unix": 1779879584.888362, "tp_rank": 0, "t_log_unix": 1779879587.814085}
+{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f06f19c981c0b3f-0-b3afb370"], "t_start_unix": 1779879642.6076336, "tp_rank": 0, "t_log_unix": 1779879642.6076367}
+{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f06f19c981c0b3f-0-b3afb370"], "duration_s": 1.5183607729850337, "t_start_unix": 1779879642.6076336, "tp_rank": 0, "t_log_unix": 1779879644.1259985}

analysis/mb2/README.md

@@ -0,0 +1,314 @@
+# MB2 — Mooncake KV Transfer Cost (vanilla vLLM 0.18.1)
+
+Persistent record of the per-stage KV transfer microbench used in §3.2 of
+the EAR paper. Re-runs append a dated section at the bottom; the
+**Summary** block at the top is what gets cited in the paper.
+
+---
+
+## Summary (latest)
+
+| Path | Steady-state BW | Agentic-tail p99 transfer (11.5 GiB KV) |
+|---|---|---|
+| **intra-node** (dash1 GPU 0↔1) | **~9.7 GB/s** (96 MiB – 3 GiB) | p50 **1.9 s** · min **1.5 s** · max **10 s** |
+| **inter-node** (dash1 GPU0 → dash2 GPU0, 200 Gbps RoCE) | **~10.0 GB/s** (essentially identical) | p50 **1.7 s** · min **1.3 s** · max **9.2 s** |
+
+**Cross-cutting finding** (2026-05-27): **Mooncake transfer cost is
+topology-independent** on this hardware. Intra-node and inter-node curves
+are statistically indistinguishable (see `figs/mb2_transfer_time_compare.png`,
+`figs/mb2_transfer_bw_compare.png`). Mechanism: Mooncake's
+`batch_transfer_sync_write` always goes through the RDMA NIC, including
+the intra-node case (RDMA loopback). The 200 Gbps NIC, not NVLink, is
+the bottleneck. **Implication for §3.2**: PD-disaggregation does not
+get cheaper by co-locating P and D on the same node — the ~9.7 GB/s
+ceiling applies regardless. Halving the transfer cost cannot be bought
+back by topology.
+
+**What MB2 actually measures**: the **per-request charge** that
+PD-disagg pays for every routed request — `T_transfer ≈ KV_size / 9.7
+GB/s`. For agentic this is **8 ms (192 MiB / trace lower) – 1.9 s
+(11.5 GiB / p99)**.
+
+**⚠ Correction (2026-05-27)**: an earlier version of this README
+framed §3.2 as "transfer cost (1.5–10 s) >> decode duration (50–200 ms),
+so PD-disagg loses on cost-vs-benefit." That accounting was wrong:
+PD-disagg's phase-isolation benefit is **per-prefill-event** and equals
+`D × T_prefill` (aggregate across stalled decode streams), not the
+single-request decode duration. With trace-mean `T_prefill = 4.5 s` and
+D = 8, the benefit is ~36 s — far larger than the ~0.32 s transfer
+cost. PD-disagg's phase-isolation axis is a *win*, not a loss.
+
+The actual reason static PD-disagg fails in agentic is **D-side KV
+capacity** (`figs/f4b_pdsep_kv_wall.png`), not a cost-vs-benefit
+imbalance. See `RESULTS_SUMMARY.md` section 4 for the corrected
+framing. MB2 still serves as the source of the per-request transfer
+cost number used in that analysis.
+
+---
+
+## Setup
+
+| Component | Value |
+|---|---|
+| Host | `dash1` (`ds-6348bee4-1-...-rwkv2`), 8× NVIDIA H20 96 GiB, driver 570.133.20 |
+| Venv  | `/home/admin/cpfs/wjh/agentic-kv-fresh/.venv` (shared via cpfs from any dash host) |
+| vLLM | 0.18.1 official wheel |
+| mooncake-transfer-engine | 0.3.11.post1 (`pip install mooncake-transfer-engine`) |
+| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct` |
+| Per-token KV | 98304 B |
+| kv_role | `kv_both` on both instances (see *Known limitations* re kv_producer/kv_consumer) |
+| Per-instance config | `--tensor-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 200000 --enable-prefix-caching` |
+
+## Method
+
+3-step black-box bench:
+
+1. `do_remote_decode` to A (producer) with a client-generated `transfer_id`.
+   `max_tokens=1`; A computes prefill and parks the KV for later pull.
+2. `do_remote_prefill` to B (consumer) with the same `transfer_id` plus
+   `remote_engine_id` (from A's `/query` on bootstrap port) and
+   `remote_bootstrap_addr` (`http://127.0.0.1:8998`). **This step
+   triggers the actual KV transfer; it is the measured step.**
+3. Plain `completion` on B (`--skip-verify` off): expect `cached_tokens
+   ≈ prompt_len`, confirming the KV landed on B.
+
+Per-stage breakdown is obtained by instrumenting the vLLM-shipped
+`MooncakeConnector` (NOT the mooncake-package's `mooncake_connector_v1`,
+which vLLM 0.18.1 does not load) at two sites:
+
+- **`_send_blocks`** (P-side, line 980): emits `send_blocks` event with
+  `total_bytes`, `duration_s`, `t_start_unix`. The `duration_s` is the
+  wall-time of a single `batch_transfer_sync_write` call — **this is
+  what we call `pure_transfer`**.
+- **`receive_kv_from_single_worker`** (D-side, line 1139, async):
+  emits `receive_kv_enter` at function start and `receive_kv_finish`
+  on FINISH-status response. The wall-time between them is
+  **`rx_total`** (= ZMQ round-trip + setup + pure_transfer + ack).
+
+Pairing across A's and B's logs is by **time window**: each B
+(enter, finish) pair is matched to the A send_blocks whose
+`t_start_unix` falls in `[rx_t_start, rx_t_end]`. With single-request
+benchmarks this is unambiguous.
+
+Scripts:
+- `microbench/fresh_setup/start_vllm_pair.sh` — bring up pair + apply/revert patch
+- `microbench/fresh_setup/instrument_mooncake.py` — apply/revert MB2 patches
+- `microbench/fresh_setup/mb2_kv_transfer.py` — client (3-step bench loop)
+- `microbench/fresh_setup/analyze_mb2.py` — pair A/B events into per-size table
+- `microbench/fresh_setup/plot_mb2.py` — log-log time + bandwidth curves
+
+## Results — intra-node (2026-05-27, dash1 GPU 0+1, kv_both)
+
+Raw events: `A_intra_kvboth.jsonl`, `B_intra_kvboth.jsonl`.
+Joined + aggregated: `intra_kvboth_breakdown.json`.
+Figures: `figs/mb2_transfer_time_intra.png`, `figs/mb2_transfer_bw_intra.png`.
+
+| input_tokens | KV (MiB) | n | pure_ms p50 | pure_ms max | rx_total_ms | overhead_ms | BW p50 (GB/s) | BW max (GB/s) |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+|    512 |    48 | 5 |   5.3 |   5.6 |  12.2 |  3.3 | 9.40 |  9.53 |
+|   1024 |    96 | 5 |  10.4 |  10.5 |  11.9 |  1.5 | 9.68 |  9.72 |
+|   2048 |   192 | 5 |  20.6 |  21.0 |  22.5 |  1.8 | 9.75 |  9.78 |
+|   4096 |   384 | 5 |  41.5 |  41.7 |  43.5 |  2.0 | 9.71 |  9.72 |
+|   8192 |   768 | 5 |  83.7 |  84.4 |  86.2 |  2.2 | 9.62 |  9.69 |
+|  16384 |  1536 | 5 | 167.1 | 167.7 | 170.2 |  2.7 | 9.64 |  9.67 |
+|  32768 |  3072 | 5 | 320.9 | 322.1 | 425.2 | 20.5 | 10.04 | 10.09 |
+|  65536 |  6144 | 5 | **1895.1** | 2375.2 | 1586.1 | 69.6 | 3.40 | 9.68 |
+| 131072 | 12288 | 5 | **2835.1** | 8923.6 | 4362.5 | 91.4 | 4.54 | 9.67 |
+
+**Three regimes** in the data:
+
+1. **<= 3 GiB** — linear in size, bandwidth ≈ **9.7 GB/s steady**.
+2. **6 GiB ± a bit** — onset of variance: max bandwidth still 9.7 GB/s,
+   but p50 collapses to ~3.4 GB/s. Some runs achieve full speed; others
+   take 2–3 × longer.
+3. **12 GiB** — wide spread (min 1.5 s, max 10 s for the same 11.5 GiB
+   transfer). This is the agentic-p99 size region.
+
+The bandwidth ceiling of ~10 GB/s is well below H20's NVLink p2p
+(claimed ~900 GB/s in IB) — likely the transfer is **PCIe-staged
+through host memory** rather than NVLink direct. To confirm we would
+need `nvidia-smi topo -m` and `mooncake_transfer_engine_topology_dump`
+analysis; not done yet.
+
+## Known limitations of this measurement
+
+- **kv_both, not strict PD-disagg.** vLLM 0.18.1 with
+  `kv_role=kv_consumer` raises `AttributeError: 'MooncakeConnectorWorker'
+  object has no attribute 'bootstrap_server'` (the attribute is only
+  assigned inside `if not self.is_kv_consumer`). The transfer mechanics
+  are identical — same `batch_transfer_sync_write` — so the cost
+  measurement is comparable. The role gate only affects which request
+  types each instance *accepts*. §5.2 strict PD-disagg baseline will
+  need either to fix that bug or front the pair with a role-aware proxy.
+- **Single in-flight request.** All measurements here are serial.
+  Real PD-disagg will have many concurrent transfers; bandwidth
+  contention is not characterized.
+- **Intra-node only.** Inter-node RDMA path will be slower; not yet
+  measured.
+- **Sanity preamble events.** The raw logs include 6 events from
+  earlier sanity runs in addition to the 45-event sweep. `analyze_mb2.py`
+  treats them as additional samples (same sizes); the per-size
+  aggregates use all of them.
+
+## Implications for §3.2 PD-disagg argument
+
+For each PD-disagg-routed request, transfer wall-time is:
+
+```
+T_transfer(KV_size) ≈ KV_size / 9.7 GB/s   for KV_size ≤ 3 GiB
+                    ≈ 0.3 – 10 s            for KV_size in [3, 12] GiB
+```
+
+This is the **per-request transfer charge** of PD-disagg. It's a
+real cost, but in the context of phase-isolation accounting it is
+*small* compared to the benefit:
+
+| Prefill | T_prefill (MB1) | T_transfer (MB2) | Phase-isolation benefit at D=8 = D × T_prefill |
+|---:|---:|---:|---:|
+| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s |
+| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s |
+| 125k tok (~p99) | 57 s | 1.9 s | 456 s |
+
+On the phase-isolation axis alone, PD-disagg recovers two orders of
+magnitude more decode time than it pays in transfer. **It is NOT this
+axis that defeats static PD-disagg in agentic** — see colleague's
+4P+4D experiment (TTFT p50 62×, success rate 99.5% → 52%) which is
+driven by **D-side KV-pool overflow** on long-context requests
+(`figs/f4b_pdsep_kv_wall.png`), not by transfer latency.
+
+What MB2 contributes to the paper is therefore:
+- The **per-request transfer cost number** (used as the cost input
+  to the cost-benefit accounting above).
+- The empirical observation that **Mooncake's transfer cost is
+  topology-independent** — intra-node and inter-node both go through
+  the RDMA NIC and hit the same 9.7 GB/s ceiling. PD-disagg's
+  transfer cost does not get cheaper by co-locating P and D.
+
+The dominant §3.2 failure mode of static PD-disagg in agentic is
+**capacity**, not transfer cost. MB3 / MB4 / MB5 will quantify the
+remaining axes (D-pool occupancy, cache reuse degradation under PD
+routing, static-partition mismatch).
+
+## Open questions / next runs
+
+- **Inter-node RDMA**: dash1 ↔ dash2. Expected lower bandwidth (~5–15
+  GB/s); want to see if the 6 GiB-onset variance moves.
+- **Bandwidth ceiling investigation**: is the 9.7 GB/s ceiling PCIe (so
+  the connector is not using NVLink direct) or some internal limit? If
+  PCIe, can it be lifted with NVLink-direct mooncake config?
+- **Variance at 6+ GiB**: investigate. Maybe related to chunking
+  inside `batch_transfer_sync_write`, or GPU memory pressure when KV
+  approaches HBM ceiling.
+- **Concurrent transfers**: measure aggregate bandwidth when N
+  simultaneous transfers happen. PD-disagg in practice does this.
+- **Strict kv_producer/kv_consumer**: patch the bootstrap_server bug
+  or use a proxy; verify transfer time is unchanged.
+
+## Reproduction
+
+```bash
+# On dash machine with cpfs mount + ssh access:
+bash microbench/fresh_setup/install.sh        # once (idempotent)
+bash microbench/fresh_setup/deploy.sh dash1    # push scripts to cpfs
+
+# bring up pair (intra-node)
+ssh dash1 'GPU_A=0 GPU_B=1 bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/start_vllm_pair.sh start'
+
+# sweep
+ssh dash1 'source /home/admin/cpfs/wjh/agentic-kv-fresh/.venv/bin/activate && \
+  python /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb2_kv_transfer.py \
+    --sizes 512,1024,2048,4096,8192,16384,32768,65536,131072 \
+    --repeats 5 --label intra-kvboth \
+    --out /home/admin/cpfs/wjh/agentic-kv-fresh/mb2_results/intra_kvboth.json'
+
+# pull logs
+scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb2_transfer_logs/A/.efc_*_mb2_transfer_pid*.jsonl \
+    analysis/mb2/A_intra_kvboth.jsonl
+scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb2_transfer_logs/B/.efc_*_mb2_transfer_pid*.jsonl \
+    analysis/mb2/B_intra_kvboth.jsonl
+
+# analyze
+.venv/bin/python microbench/fresh_setup/analyze_mb2.py \
+  --a-log analysis/mb2/A_intra_kvboth.jsonl \
+  --b-log analysis/mb2/B_intra_kvboth.jsonl \
+  --out analysis/mb2/intra_kvboth_breakdown.json
+
+.venv/bin/python microbench/fresh_setup/plot_mb2.py \
+  --breakdown analysis/mb2/intra_kvboth_breakdown.json \
+  --out-time figs/mb2_transfer_time_intra.png \
+  --out-bw figs/mb2_transfer_bw_intra.png
+
+# tear down
+ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/start_vllm_pair.sh stop'
+```
+
+## Run log
+
+### 2026-05-27 — intra-node, kv_both, dash1 GPU 0+1
+
+Sweep: `512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072` tokens
+× 5 repeats. Sanity preamble of `512, 2048, 8192` × 2 included in the
+raw logs (counted as additional samples for those sizes).
+
+Result table above. **9.7 GB/s steady-state up to 3 GiB**, variance
+opens at 6 GiB, p99 agentic-tail transfer 1.5 – 10 s.
+
+Committed as `de164e5`.
+
+### 2026-05-27 — inter-node, kv_both, dash1 GPU 0 → dash2 GPU 0
+
+Same sweep config. 200 Gbps RoCE between hosts (RTT ~0.2 ms ping).
+Producer A on dash1 GPU 0, consumer B on dash2 GPU 0.
+remote_bootstrap_addr=`http://172.27.123.142:8998` (dash1's internal IP).
+
+Raw events: `A_inter_kvboth.jsonl` (45 send_blocks + 6 sanity).
+B's receive_kv events are **missing** for this run — the
+`MB2_LOG_DIR` env var did not propagate from the start-script through
+vLLM's EngineCore subprocess on dash2 (visible via
+`cat /proc/$ENGINE_PID/environ` shows empty for dash2 but contains
+MB2_LOG_DIR for dash1 — bookmark for future investigation, likely
+spawn-vs-fork difference in vLLM's multiproc executor across hosts).
+Pure-transfer numbers below come from A's send_blocks alone; full
+rx_total breakdown not available for this run.
+
+Per-size pure-transfer (analyzed by `analyze_mb2_send_only.py`):
+
+| input_tokens | KV (MiB) | n | pure_ms p50 | min | max | BW p50 (GB/s) | BW max |
+|---:|---:|---:|---:|---:|---:|---:|---:|
+|    512 |    48 | 5 |   5.2 |   5.1 |  65.8 |  9.76 |  9.81 |
+|   1024 |    96 | 5 |  10.2 |  10.1 |  10.4 |  9.91 | 10.00 |
+|   2048 |   192 | 5 |  20.0 |  20.0 |  20.5 | 10.06 | 10.07 |
+|   4096 |   384 | 5 |  40.1 |  40.1 |  40.5 | 10.04 | 10.05 |
+|   8192 |   768 | 5 |  80.9 |  80.7 |  82.5 |  9.96 |  9.98 |
+|  16384 |  1536 | 5 | 161.8 | 161.7 | 164.8 |  9.96 |  9.96 |
+|  32768 |  3072 | 5 | 309.6 | 307.7 | 526.9 | 10.40 | 10.47 |
+|  65536 |  6144 | 5 | 1733.6 | 653.5 | 1921.2 | 3.72 | 9.86 |
+| 131072 | 12288 | 5 | 2818.4 | 1283.0 | 9158.6 | 4.57 | 10.04 |
+
+Side-by-side comparison with the 2026-05-27 intra-node run:
+
+| Size | intra p50 ms | inter p50 ms | gap | intra GB/s | inter GB/s |
+|---|---:|---:|---:|---:|---:|
+|   512 |    5.3 |    5.2 | −2% | 9.40 |  9.76 |
+|  1024 |   10.4 |   10.2 | −2% | 9.68 |  9.91 |
+|  2048 |   20.6 |   20.0 | −3% | 9.75 | 10.06 |
+|  4096 |   41.5 |   40.1 | −3% | 9.71 | 10.04 |
+|  8192 |   83.7 |   80.9 | −3% | 9.62 |  9.96 |
+| 16384 |  167.1 |  161.8 | −3% | 9.64 |  9.96 |
+| 32768 |  320.9 |  309.6 | −3% | 10.04 | 10.40 |
+| 65536 | 1895.1 | 1733.6 | −9% | 3.40 |  3.72 |
+|131072 | 2835.1 | 2818.4 | −1% | 4.54 |  4.57 |
+
+The two paths produce essentially the same numbers — **mooncake intra-
+node is not using NVLink**, it's going through RDMA-loopback on the
+local NIC and gets the same ~10 GB/s ceiling as cross-node RDMA. The
+6+ GiB variance regime is also identical between paths.
+
+Figures: `figs/mb2_transfer_time_inter.png`, `figs/mb2_transfer_bw_inter.png`,
+`figs/mb2_transfer_time_compare.png` (overlay), `figs/mb2_transfer_bw_compare.png`.
+
+This collapses the §3.2 narrative to a single number: **PD-disagg
+across this cluster costs ~9.7–10 GB/s of transfer bandwidth no matter
+how you place P and D** (within-node or across-node). For p99 agentic
+KV (11.5 GiB), that's 1.3–10 s of transfer; for 6 GiB it's 0.7–2 s.
+Decode is 50–200 ms. So PD-disagg's cost dominates regardless of layout.

analysis/mb2/inter_kvboth_breakdown.json

@@ -0,0 +1,112 @@
+{
+  "summary": [
+    {
+      "input_tokens": 512,
+      "kv_mib": 48.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 17.29,
+      "pure_transfer_ms_p50": 5.16,
+      "pure_transfer_ms_min": 5.13,
+      "pure_transfer_ms_max": 65.8,
+      "throughput_gbps_mean": 7.95,
+      "throughput_gbps_p50": 9.76,
+      "throughput_gbps_max": 9.81
+    },
+    {
+      "input_tokens": 1024,
+      "kv_mib": 96.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 10.18,
+      "pure_transfer_ms_p50": 10.15,
+      "pure_transfer_ms_min": 10.06,
+      "pure_transfer_ms_max": 10.37,
+      "throughput_gbps_mean": 9.89,
+      "throughput_gbps_p50": 9.91,
+      "throughput_gbps_max": 10.0
+    },
+    {
+      "input_tokens": 2048,
+      "kv_mib": 192.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 20.14,
+      "pure_transfer_ms_p50": 20.01,
+      "pure_transfer_ms_min": 19.99,
+      "pure_transfer_ms_max": 20.51,
+      "throughput_gbps_mean": 10.0,
+      "throughput_gbps_p50": 10.06,
+      "throughput_gbps_max": 10.07
+    },
+    {
+      "input_tokens": 4096,
+      "kv_mib": 384.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 40.19,
+      "pure_transfer_ms_p50": 40.12,
+      "pure_transfer_ms_min": 40.07,
+      "pure_transfer_ms_max": 40.5,
+      "throughput_gbps_mean": 10.02,
+      "throughput_gbps_p50": 10.04,
+      "throughput_gbps_max": 10.05
+    },
+    {
+      "input_tokens": 8192,
+      "kv_mib": 768.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 81.32,
+      "pure_transfer_ms_p50": 80.86,
+      "pure_transfer_ms_min": 80.71,
+      "pure_transfer_ms_max": 82.5,
+      "throughput_gbps_mean": 9.9,
+      "throughput_gbps_p50": 9.96,
+      "throughput_gbps_max": 9.98
+    },
+    {
+      "input_tokens": 16384,
+      "kv_mib": 1536.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 162.76,
+      "pure_transfer_ms_p50": 161.78,
+      "pure_transfer_ms_min": 161.66,
+      "pure_transfer_ms_max": 164.76,
+      "throughput_gbps_mean": 9.9,
+      "throughput_gbps_p50": 9.96,
+      "throughput_gbps_max": 9.96
+    },
+    {
+      "input_tokens": 32768,
+      "kv_mib": 3072.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 395.6,
+      "pure_transfer_ms_p50": 309.61,
+      "pure_transfer_ms_min": 307.71,
+      "pure_transfer_ms_max": 526.86,
+      "throughput_gbps_mean": 8.71,
+      "throughput_gbps_p50": 10.4,
+      "throughput_gbps_max": 10.47
+    },
+    {
+      "input_tokens": 65536,
+      "kv_mib": 6144.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 1427.92,
+      "pure_transfer_ms_p50": 1733.57,
+      "pure_transfer_ms_min": 653.52,
+      "pure_transfer_ms_max": 1921.19,
+      "throughput_gbps_mean": 5.45,
+      "throughput_gbps_p50": 3.72,
+      "throughput_gbps_max": 9.86
+    },
+    {
+      "input_tokens": 131072,
+      "kv_mib": 12288.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 3953.36,
+      "pure_transfer_ms_p50": 2818.43,
+      "pure_transfer_ms_min": 1283.03,
+      "pure_transfer_ms_max": 9158.57,
+      "throughput_gbps_mean": 5.49,
+      "throughput_gbps_p50": 4.57,
+      "throughput_gbps_max": 10.04
+    }
+  ]
+}

analysis/mb2/inter_kvboth_client.json

@@ -0,0 +1,679 @@
+{
+  "model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct",
+  "kv_bytes_per_token": 98304,
+  "src_host": "172.27.123.142",
+  "src_port": 8000,
+  "dst_host": "172.27.123.133",
+  "dst_port": 8001,
+  "config_label": "inter-kvboth",
+  "raw": [
+    {
+      "input_tokens": 512,
+      "session": "677ca02530774bae990d549d0591d336",
+      "t_step1_client_unix": 1779885642.2549865,
+      "t_step2_client_unix": 1779885642.3058398,
+      "t_step2_end_unix": 1779885642.3249059,
+      "t_prefill_s": 0.05082751903682947,
+      "t_transfer_s": 0.019030610972549766,
+      "t_followup_s": 0.01642513304250315,
+      "cached_followup": 496,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 512,
+      "session": "6758b8079675495a9a13a73db1199a71",
+      "t_step1_client_unix": 1779885642.3416288,
+      "t_step2_client_unix": 1779885642.3896646,
+      "t_step2_end_unix": 1779885642.40659,
+      "t_prefill_s": 0.04800783301470801,
+      "t_transfer_s": 0.016899197013117373,
+      "t_followup_s": 0.017115428985562176,
+      "cached_followup": 496,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 512,
+      "session": "e9da485a88cc40eb927e18c37298d815",
+      "t_step1_client_unix": 1779885642.4239926,
+      "t_step2_client_unix": 1779885642.4724562,
+      "t_step2_end_unix": 1779885642.492528,
+      "t_prefill_s": 0.04843967000488192,
+      "t_transfer_s": 0.02004897501319647,
+      "t_followup_s": 0.016289777995552868,
+      "cached_followup": 496,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 512,
+      "session": "a70db16a01b04409b04ef30e94082720",
+      "t_step1_client_unix": 1779885642.509085,
+      "t_step2_client_unix": 1779885642.5566719,
+      "t_step2_end_unix": 1779885642.577205,
+      "t_prefill_s": 0.047563806001562625,
+      "t_transfer_s": 0.020512140006758273,
+      "t_followup_s": 0.016482150997035205,
+      "cached_followup": 496,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 512,
+      "session": "d13cddd850904d4591b15c112ab32a5f",
+      "t_step1_client_unix": 1779885642.5939884,
+      "t_step2_client_unix": 1779885642.6417143,
+      "t_step2_end_unix": 1779885642.6617038,
+      "t_prefill_s": 0.04769411502638832,
+      "t_transfer_s": 0.019964047998655587,
+      "t_followup_s": 0.01518680399749428,
+      "cached_followup": 496,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 1024,
+      "session": "64d9b1335bd04698ab4f681fe4fe1e40",
+      "t_step1_client_unix": 1779885642.6773622,
+      "t_step2_client_unix": 1779885642.7512894,
+      "t_step2_end_unix": 1779885642.7811818,
+      "t_prefill_s": 0.07390057999873534,
+      "t_transfer_s": 0.029867444012779742,
+      "t_followup_s": 0.017206061049364507,
+      "cached_followup": 1008,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 1024,
+      "session": "0fc46cc27a844482b9eab2fef48a56d2",
+      "t_step1_client_unix": 1779885642.7988534,
+      "t_step2_client_unix": 1779885642.8733544,
+      "t_step2_end_unix": 1779885642.9005985,
+      "t_prefill_s": 0.074467666039709,
+      "t_transfer_s": 0.027220223972108215,
+      "t_followup_s": 0.0172900699544698,
+      "cached_followup": 1008,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 1024,
+      "session": "dac7cceb94c0465abce6bff73119de06",
+      "t_step1_client_unix": 1779885642.9183617,
+      "t_step2_client_unix": 1779885642.9924595,
+      "t_step2_end_unix": 1779885643.0189831,
+      "t_prefill_s": 0.07407327799592167,
+      "t_transfer_s": 0.026499781000893563,
+      "t_followup_s": 0.01622009096900001,
+      "cached_followup": 1008,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 1024,
+      "session": "4137a7ebecf148db8aa61249ef0d0100",
+      "t_step1_client_unix": 1779885643.0356703,
+      "t_step2_client_unix": 1779885643.109194,
+      "t_step2_end_unix": 1779885643.135519,
+      "t_prefill_s": 0.07349911500932649,
+      "t_transfer_s": 0.02630024799145758,
+      "t_followup_s": 0.01637468097032979,
+      "cached_followup": 1008,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 1024,
+      "session": "e28432915bf64c4891dc5157d0bebe41",
+      "t_step1_client_unix": 1779885643.1523507,
+      "t_step2_client_unix": 1779885643.2252986,
+      "t_step2_end_unix": 1779885643.2520146,
+      "t_prefill_s": 0.07292484800564125,
+      "t_transfer_s": 0.026693789986893535,
+      "t_followup_s": 0.016969401971437037,
+      "cached_followup": 1008,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 2048,
+      "session": "717a42513a2c48358ebfbb73ca07c3fd",
+      "t_step1_client_unix": 1779885643.269825,
+      "t_step2_client_unix": 1779885643.3983366,
+      "t_step2_end_unix": 1779885643.417481,
+      "t_prefill_s": 0.12848620803561062,
+      "t_transfer_s": 0.019112227018922567,
+      "t_followup_s": 0.017094306997023523,
+      "cached_followup": 2032,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 2048,
+      "session": "f2d1287f0e564a60a0d8476bb917ec9a",
+      "t_step1_client_unix": 1779885643.4353812,
+      "t_step2_client_unix": 1779885643.5640473,
+      "t_step2_end_unix": 1779885643.5819182,
+      "t_prefill_s": 0.12862860300811008,
+      "t_transfer_s": 0.01783871802035719,
+      "t_followup_s": 0.016443474043626338,
+      "cached_followup": 2032,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 2048,
+      "session": "cff5a22ae9c341d8acc455188ecc63b5",
+      "t_step1_client_unix": 1779885643.5992126,
+      "t_step2_client_unix": 1779885643.7292078,
+      "t_step2_end_unix": 1779885643.766638,
+      "t_prefill_s": 0.12995763396611437,
+      "t_transfer_s": 0.037401642999611795,
+      "t_followup_s": 0.018064863979816437,
+      "cached_followup": 2032,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 2048,
+      "session": "cc86ca16270b493fab68af1e889007af",
+      "t_step1_client_unix": 1779885643.7855458,
+      "t_step2_client_unix": 1779885643.9140093,
+      "t_step2_end_unix": 1779885643.9510815,
+      "t_prefill_s": 0.12843843002337962,
+      "t_transfer_s": 0.03704743104754016,
+      "t_followup_s": 0.017415895010344684,
+      "cached_followup": 2032,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 2048,
+      "session": "85f348460077474ab5c7f6db23005f48",
+      "t_step1_client_unix": 1779885643.9692879,
+      "t_step2_client_unix": 1779885644.0971973,
+      "t_step2_end_unix": 1779885644.1345108,
+      "t_prefill_s": 0.12788102397462353,
+      "t_transfer_s": 0.037287415005266666,
+      "t_followup_s": 0.01738263398874551,
+      "cached_followup": 2032,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 4096,
+      "session": "f715284b17af4bc29ccb689f21ac61b7",
+      "t_step1_client_unix": 1779885644.1534681,
+      "t_step2_client_unix": 1779885644.4121828,
+      "t_step2_end_unix": 1779885644.4760094,
+      "t_prefill_s": 0.25868210894986987,
+      "t_transfer_s": 0.06378706597024575,
+      "t_followup_s": 0.020318155991844833,
+      "cached_followup": 4080,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 4096,
+      "session": "b99bf61a90d54373a57af6a38d67581d",
+      "t_step1_client_unix": 1779885644.497922,
+      "t_step2_client_unix": 1779885644.7539463,
+      "t_step2_end_unix": 1779885644.8172574,
+      "t_prefill_s": 0.255985789000988,
+      "t_transfer_s": 0.06327042501652613,
+      "t_followup_s": 0.021031456999480724,
+      "cached_followup": 4080,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 4096,
+      "session": "69925f5b50f74c48800999908730d9ec",
+      "t_step1_client_unix": 1779885644.839903,
+      "t_step2_client_unix": 1779885645.0941288,
+      "t_step2_end_unix": 1779885645.1562943,
+      "t_prefill_s": 0.2541897820192389,
+      "t_transfer_s": 0.06213490100344643,
+      "t_followup_s": 0.021715508948545903,
+      "cached_followup": 4080,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 4096,
+      "session": "312743112bc84c0699a91e3110a63f03",
+      "t_step1_client_unix": 1779885645.1795704,
+      "t_step2_client_unix": 1779885645.4336376,
+      "t_step2_end_unix": 1779885645.4955726,
+      "t_prefill_s": 0.2540306959999725,
+      "t_transfer_s": 0.06190510099986568,
+      "t_followup_s": 0.020819986995775253,
+      "cached_followup": 4080,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 4096,
+      "session": "e5975bd25e1b4aa586cbfe9b68370de2",
+      "t_step1_client_unix": 1779885645.5179617,
+      "t_step2_client_unix": 1779885645.7724028,
+      "t_step2_end_unix": 1779885645.8355522,
+      "t_prefill_s": 0.2544059020001441,
+      "t_transfer_s": 0.06311210000421852,
+      "t_followup_s": 0.02235932502662763,
+      "cached_followup": 4080,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 8192,
+      "session": "f8989efa9261470e81c2891945b460a5",
+      "t_step1_client_unix": 1779885645.8609917,
+      "t_step2_client_unix": 1779885646.4364054,
+      "t_step2_end_unix": 1779885646.4636223,
+      "t_prefill_s": 0.575372110994067,
+      "t_transfer_s": 0.02718952501891181,
+      "t_followup_s": 0.024976509041152894,
+      "cached_followup": 8176,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 8192,
+      "session": "3a8632e8562c4cbcaa32b57ba23bbf68",
+      "t_step1_client_unix": 1779885646.4916308,
+      "t_step2_client_unix": 1779885647.0670328,
+      "t_step2_end_unix": 1779885647.0934644,
+      "t_prefill_s": 0.5753633370040916,
+      "t_transfer_s": 0.026399013004265726,
+      "t_followup_s": 0.027285586984362453,
+      "cached_followup": 8176,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 8192,
+      "session": "4d2cfc02defd492d89e90f652032f630",
+      "t_step1_client_unix": 1779885647.1237462,
+      "t_step2_client_unix": 1779885647.6985745,
+      "t_step2_end_unix": 1779885647.8096595,
+      "t_prefill_s": 0.574782071984373,
+      "t_transfer_s": 0.11104170099133626,
+      "t_followup_s": 0.02618999598780647,
+      "cached_followup": 8176,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 8192,
+      "session": "3485a775304947ada42d61cfd39b7520",
+      "t_step1_client_unix": 1779885647.8389955,
+      "t_step2_client_unix": 1779885648.4133987,
+      "t_step2_end_unix": 1779885648.5235367,
+      "t_prefill_s": 0.574362020008266,
+      "t_transfer_s": 0.11010575300315395,
+      "t_followup_s": 0.028998883964959532,
+      "cached_followup": 8176,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 8192,
+      "session": "6f87692fb5884094b7b141116ba739f8",
+      "t_step1_client_unix": 1779885648.5556417,
+      "t_step2_client_unix": 1779885649.1294918,
+      "t_step2_end_unix": 1779885649.2386518,
+      "t_prefill_s": 0.5738097460125573,
+      "t_transfer_s": 0.1091307660099119,
+      "t_followup_s": 0.025612210971303284,
+      "cached_followup": 8176,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 16384,
+      "session": "c4e007ebdcf244cf95a8ccb3ab59ea0f",
+      "t_step1_client_unix": 1779885649.270192,
+      "t_step2_client_unix": 1779885650.7972357,
+      "t_step2_end_unix": 1779885651.0219116,
+      "t_prefill_s": 1.5270042870542966,
+      "t_transfer_s": 0.22463428904302418,
+      "t_followup_s": 0.03729638300137594,
+      "cached_followup": 16368,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 16384,
+      "session": "6f34ab6672e14f41b7780323b30ce21e",
+      "t_step1_client_unix": 1779885651.0651574,
+      "t_step2_client_unix": 1779885652.5826244,
+      "t_step2_end_unix": 1779885652.7871268,
+      "t_prefill_s": 1.5174251759890467,
+      "t_transfer_s": 0.2044643560075201,
+      "t_followup_s": 0.03547715302556753,
+      "cached_followup": 16368,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 16384,
+      "session": "ca02bd5aa32544f2b1b09096e5f63c8b",
+      "t_step1_client_unix": 1779885652.8284712,
+      "t_step2_client_unix": 1779885654.3465412,
+      "t_step2_end_unix": 1779885654.5522587,
+      "t_prefill_s": 1.5180304029490799,
+      "t_transfer_s": 0.20567574101733044,
+      "t_followup_s": 0.03611186001216993,
+      "cached_followup": 16368,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 16384,
+      "session": "59047ab966574f72ab12693ad29b090f",
+      "t_step1_client_unix": 1779885654.5942106,
+      "t_step2_client_unix": 1779885656.1123874,
+      "t_step2_end_unix": 1779885656.3158703,
+      "t_prefill_s": 1.5181243289844133,
+      "t_transfer_s": 0.20344976399792358,
+      "t_followup_s": 0.035486461012624204,
+      "cached_followup": 16368,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 16384,
+      "session": "9a6e2228fd40414c8fd2b78bf399552c",
+      "t_step1_client_unix": 1779885656.357184,
+      "t_step2_client_unix": 1779885657.8739264,
+      "t_step2_end_unix": 1779885658.079835,
+      "t_prefill_s": 1.5166968539706431,
+      "t_transfer_s": 0.20585927896900102,
+      "t_followup_s": 0.03941280301660299,
+      "cached_followup": 16368,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 32768,
+      "session": "5cf9fb856b674c0a9237c06980287cb3",
+      "t_step1_client_unix": 1779885658.1312225,
+      "t_step2_client_unix": 1779885662.6780548,
+      "t_step2_end_unix": 1779885663.0500648,
+      "t_prefill_s": 4.54679255298106,
+      "t_transfer_s": 0.3719712099991739,
+      "t_followup_s": 0.05820785299874842,
+      "cached_followup": 32752,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 32768,
+      "session": "524f192c92834453bc48052bc6516126",
+      "t_step1_client_unix": 1779885663.1199641,
+      "t_step2_client_unix": 1779885667.6679316,
+      "t_step2_end_unix": 1779885668.047607,
+      "t_prefill_s": 4.547927976993378,
+      "t_transfer_s": 0.3796397229889408,
+      "t_followup_s": 0.058314190013334155,
+      "cached_followup": 32752,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 32768,
+      "session": "fb7aa2c99bf24febb83536c54e30a36d",
+      "t_step1_client_unix": 1779885668.117419,
+      "t_step2_client_unix": 1779885672.6644971,
+      "t_step2_end_unix": 1779885673.0345452,
+      "t_prefill_s": 4.547034470015205,
+      "t_transfer_s": 0.3700092720100656,
+      "t_followup_s": 0.05658108199713752,
+      "cached_followup": 32752,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 32768,
+      "session": "85612444c3224b65afb09d65a829b5cc",
+      "t_step1_client_unix": 1779885673.1025856,
+      "t_step2_client_unix": 1779885677.6487892,
+      "t_step2_end_unix": 1779885678.2856307,
+      "t_prefill_s": 4.546165039995685,
+      "t_transfer_s": 0.6368027949938551,
+      "t_followup_s": 0.05732736398931593,
+      "cached_followup": 32752,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 32768,
+      "session": "fe51fbacc4d544b3a500b099f852b19b",
+      "t_step1_client_unix": 1779885678.3544915,
+      "t_step2_client_unix": 1779885682.9011996,
+      "t_step2_end_unix": 1779885683.542081,
+      "t_prefill_s": 4.546669405011926,
+      "t_transfer_s": 0.6408427829737775,
+      "t_followup_s": 0.057976288022473454,
+      "cached_followup": 32752,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 65536,
+      "session": "9628b04d4d9e4a9fb26decccef84d32f",
+      "t_step1_client_unix": 1779885683.6233714,
+      "t_step2_client_unix": 1779885698.8039112,
+      "t_step2_end_unix": 1779885700.9075115,
+      "t_prefill_s": 15.18050062697148,
+      "t_transfer_s": 2.103562032978516,
+      "t_followup_s": 0.0921451430185698,
+      "cached_followup": 65520,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 65536,
+      "session": "b8bd7328d2274613b473e932db11081c",
+      "t_step1_client_unix": 1779885701.0228417,
+      "t_step2_client_unix": 1779885716.2263126,
+      "t_step2_end_unix": 1779885718.3809934,
+      "t_prefill_s": 15.200471161981113,
+      "t_transfer_s": 2.1520405350020155,
+      "t_followup_s": 0.11590708000585437,
+      "cached_followup": 65520,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 65536,
+      "session": "8ca1dfe708c945e59c194ee59b7de87d",
+      "t_step1_client_unix": 1779885718.5226529,
+      "t_step2_client_unix": 1779885733.706992,
+      "t_step2_end_unix": 1779885735.6409419,
+      "t_prefill_s": 15.184306180977728,
+      "t_transfer_s": 1.9339170460007153,
+      "t_followup_s": 0.09697375196265057,
+      "cached_followup": 65520,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 65536,
+      "session": "aa7555475c87427ab1a1d95277545655",
+      "t_step1_client_unix": 1779885735.7609437,
+      "t_step2_client_unix": 1779885750.9334345,
+      "t_step2_end_unix": 1779885752.0260077,
+      "t_prefill_s": 15.172441756993067,
+      "t_transfer_s": 1.09253996796906,
+      "t_followup_s": 0.08802600798662752,
+      "cached_followup": 65520,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 65536,
+      "session": "dece39eeadd645c7a8e563763f93c35f",
+      "t_step1_client_unix": 1779885752.1370633,
+      "t_step2_client_unix": 1779885767.3073487,
+      "t_step2_end_unix": 1779885768.0632908,
+      "t_prefill_s": 15.17024433100596,
+      "t_transfer_s": 0.7559080219944008,
+      "t_followup_s": 0.08695380098652095,
+      "cached_followup": 65520,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 131072,
+      "session": "98a339dad05a463e8ac1425413221d27",
+      "t_step1_client_unix": 1779885768.196416,
+      "t_step2_client_unix": 1779885822.9193504,
+      "t_step2_end_unix": 1779885824.3877244,
+      "t_prefill_s": 54.72287795698503,
+      "t_transfer_s": 1.4683381259674206,
+      "t_followup_s": 0.16380483901593834,
+      "cached_followup": 131056,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 131072,
+      "session": "4ce2151ad3d4493397bb77c7a62871e7",
+      "t_step1_client_unix": 1779885824.5981278,
+      "t_step2_client_unix": 1779885879.3268123,
+      "t_step2_end_unix": 1779885884.7123976,
+      "t_prefill_s": 54.72864995297277,
+      "t_transfer_s": 5.38555136800278,
+      "t_followup_s": 0.17249851900851354,
+      "cached_followup": 131056,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 131072,
+      "session": "6a214438dcf74f36bbefaaed3583c04c",
+      "t_step1_client_unix": 1779885884.9482107,
+      "t_step2_client_unix": 1779885939.6747935,
+      "t_step2_end_unix": 1779885949.2077906,
+      "t_prefill_s": 54.72654917498585,
+      "t_transfer_s": 9.532963149016723,
+      "t_followup_s": 0.1658564520184882,
+      "cached_followup": 131056,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 131072,
+      "session": "4cd33910256e43768a2f475650d7714c",
+      "t_step1_client_unix": 1779885949.4215908,
+      "t_step2_client_unix": 1779886004.145101,
+      "t_step2_end_unix": 1779886007.2428164,
+      "t_prefill_s": 54.72347703698324,
+      "t_transfer_s": 3.0976800689822994,
+      "t_followup_s": 0.16629640298197046,
+      "cached_followup": 131056,
+      "pull_completion_tokens": 1,
+      "ok": true
+    },
+    {
+      "input_tokens": 131072,
+      "session": "7e3626253b5d4fe29b3cec1e3598bb39",
+      "t_step1_client_unix": 1779886007.4549844,
+      "t_step2_client_unix": 1779886062.0564816,
+      "t_step2_end_unix": 1779886063.7279255,
+      "t_prefill_s": 54.60145856201416,
+      "t_transfer_s": 1.6714086790452711,
+      "t_followup_s": 0.1628041280200705,
+      "cached_followup": 131056,
+      "pull_completion_tokens": 1,
+      "ok": true
+    }
+  ],
+  "summary": [
+    {
+      "input_tokens": 512,
+      "n_ok": 5,
+      "transfer_s_mean": 0.019290994200855494,
+      "transfer_s_p50": 0.019964047998655587,
+      "transfer_s_p90": 0.020512140006758273,
+      "transfer_s_min": 0.016899197013117373,
+      "transfer_s_max": 0.020512140006758273
+    },
+    {
+      "input_tokens": 1024,
+      "n_ok": 5,
+      "transfer_s_mean": 0.027316297392826528,
+      "transfer_s_p50": 0.026693789986893535,
+      "transfer_s_p90": 0.029867444012779742,
+      "transfer_s_min": 0.02630024799145758,
+      "transfer_s_max": 0.029867444012779742
+    },
+    {
+      "input_tokens": 2048,
+      "n_ok": 5,
+      "transfer_s_mean": 0.029737486818339674,
+      "transfer_s_p50": 0.03704743104754016,
+      "transfer_s_p90": 0.037401642999611795,
+      "transfer_s_min": 0.01783871802035719,
+      "transfer_s_max": 0.037401642999611795
+    },
+    {
+      "input_tokens": 4096,
+      "n_ok": 5,
+      "transfer_s_mean": 0.0628419185988605,
+      "transfer_s_p50": 0.06311210000421852,
+      "transfer_s_p90": 0.06378706597024575,
+      "transfer_s_min": 0.06190510099986568,
+      "transfer_s_max": 0.06378706597024575
+    },
+    {
+      "input_tokens": 8192,
+      "n_ok": 5,
+      "transfer_s_mean": 0.07677335160551593,
+      "transfer_s_p50": 0.1091307660099119,
+      "transfer_s_p90": 0.11104170099133626,
+      "transfer_s_min": 0.026399013004265726,
+      "transfer_s_max": 0.11104170099133626
+    },
+    {
+      "input_tokens": 16384,
+      "n_ok": 5,
+      "transfer_s_mean": 0.20881668580695986,
+      "transfer_s_p50": 0.20567574101733044,
+      "transfer_s_p90": 0.22463428904302418,
+      "transfer_s_min": 0.20344976399792358,
+      "transfer_s_max": 0.22463428904302418
+    },
+    {
+      "input_tokens": 32768,
+      "n_ok": 5,
+      "transfer_s_mean": 0.4798531565931626,
+      "transfer_s_p50": 0.3796397229889408,
+      "transfer_s_p90": 0.6408427829737775,
+      "transfer_s_min": 0.3700092720100656,
+      "transfer_s_max": 0.6408427829737775
+    },
+    {
+      "input_tokens": 65536,
+      "n_ok": 5,
+      "transfer_s_mean": 1.6075935207889416,
+      "transfer_s_p50": 1.9339170460007153,
+      "transfer_s_p90": 2.1520405350020155,
+      "transfer_s_min": 0.7559080219944008,
+      "transfer_s_max": 2.1520405350020155
+    },
+    {
+      "input_tokens": 131072,
+      "n_ok": 5,
+      "transfer_s_mean": 4.2311882782028984,
+      "transfer_s_p50": 3.0976800689822994,
+      "transfer_s_p90": 9.532963149016723,
+      "transfer_s_min": 1.4683381259674206,
+      "transfer_s_max": 9.532963149016723
+    }
+  ]
+}

analysis/mb2/intra_kvboth_breakdown.json

@@ -0,0 +1,758 @@
+{
+  "rows": [
+    {
+      "input_tokens_est": 512,
+      "total_bytes": 50331648,
+      "pure_transfer_s": 0.023202952987048775,
+      "rx_total_s": 0.03333390498301014,
+      "rx_overhead_s": 0.010130951995961368,
+      "rx_t_start_unix": 1779879143.1678784,
+      "send_t_start_unix": 1779879143.174031,
+      "req_ids": [
+        "cmpl-ad00672f263a6643-0-9479211a"
+      ]
+    },
+    {
+      "input_tokens_est": 512,
+      "total_bytes": 50331648,
+      "pure_transfer_s": 0.005375694017857313,
+      "rx_total_s": 0.007019245007541031,
+      "rx_overhead_s": 0.0016435509896837175,
+      "rx_t_start_unix": 1779879143.2968972,
+      "send_t_start_unix": 1779879143.2982283,
+      "req_ids": [
+        "cmpl-ace77e2b02f9f141-0-b3c061bc"
+      ]
+    },
+    {
+      "input_tokens_est": 2048,
+      "total_bytes": 201326592,
+      "pure_transfer_s": 0.021170366962905973,
+      "rx_total_s": 0.02278437599306926,
+      "rx_overhead_s": 0.0016140090301632881,
+      "rx_t_start_unix": 1779879143.5146625,
+      "send_t_start_unix": 1779879143.5159554,
+      "req_ids": [
+        "cmpl-a4a2366879c68ded-0-8ac4098e"
+      ]
+    },
+    {
+      "input_tokens_est": 2048,
+      "total_bytes": 201326592,
+      "pure_transfer_s": 0.020726953051052988,
+      "rx_total_s": 0.022794076008722186,
+      "rx_overhead_s": 0.0020671229576691985,
+      "rx_t_start_unix": 1779879143.6958342,
+      "send_t_start_unix": 1779879143.6974514,
+      "req_ids": [
+        "cmpl-8690cafcace0d5e2-0-b89f33d2"
+      ]
+    },
+    {
+      "input_tokens_est": 8192,
+      "total_bytes": 805306368,
+      "pure_transfer_s": 0.08536655298667029,
+      "rx_total_s": 0.08753501297906041,
+      "rx_overhead_s": 0.002168459992390126,
+      "rx_t_start_unix": 1779879144.3279662,
+      "send_t_start_unix": 1779879144.3294952,
+      "req_ids": [
+        "cmpl-b087e2ec4cfa8eb7-0-b908f425"
+      ]
+    },
+    {
+      "input_tokens_est": 8192,
+      "total_bytes": 805306368,
+      "pure_transfer_s": 0.08367906499188393,
+      "rx_total_s": 0.0860149699728936,
+      "rx_overhead_s": 0.002335904981009662,
+      "rx_t_start_unix": 1779879145.040141,
+      "send_t_start_unix": 1779879145.0419943,
+      "req_ids": [
+        "cmpl-a115d16ff5575e08-0-9fa81984"
+      ]
+    },
+    {
+      "input_tokens_est": 16,
+      "total_bytes": 1572864,
+      "pure_transfer_s": 0.0004059679922647774,
+      "rx_total_s": 0.002459956973325461,
+      "rx_overhead_s": 0.0020539889810606837,
+      "rx_t_start_unix": 1779879221.7062025,
+      "send_t_start_unix": 1779879221.7078288,
+      "req_ids": [
+        "cmpl-9e585ed083951df5-0-b03f812b"
+      ]
+    },
+    {
+      "input_tokens_est": 16,
+      "total_bytes": 1572864,
+      "pure_transfer_s": 0.000346789020113647,
+      "rx_total_s": 0.0020201010047458112,
+      "rx_overhead_s": 0.0016733119846321642,
+      "rx_t_start_unix": 1779879221.7826598,
+      "send_t_start_unix": 1779879221.7838593,
+      "req_ids": [
+        "cmpl-9271d403c044eadd-0-9c3c4639"
+      ]
+    },
+    {
+      "input_tokens_est": 512,
+      "total_bytes": 50331648,
+      "pure_transfer_s": 0.005353622022084892,
+      "rx_total_s": 0.006836243963334709,
+      "rx_overhead_s": 0.0014826219412498176,
+      "rx_t_start_unix": 1779879221.859549,
+      "send_t_start_unix": 1779879221.8607252,
+      "req_ids": [
+        "cmpl-82a580cefd3e2440-0-a383c3c4"
+      ]
+    },
+    {
+      "input_tokens_est": 512,
+      "total_bytes": 50331648,
+      "pure_transfer_s": 0.005279594974126667,
+      "rx_total_s": 0.00694335694424808,
+      "rx_overhead_s": 0.0016637619701214135,
+      "rx_t_start_unix": 1779879221.9419758,
+      "send_t_start_unix": 1779879221.9432015,
+      "req_ids": [
+        "cmpl-a31cb4bc9e7f63d2-0-8f48aacd"
+      ]
+    },
+    {
+      "input_tokens_est": 512,
+      "total_bytes": 50331648,
+      "pure_transfer_s": 0.0053006180096417665,
+      "rx_total_s": 0.006697195000015199,
+      "rx_overhead_s": 0.0013965769903734326,
+      "rx_t_start_unix": 1779879222.0232244,
+      "send_t_start_unix": 1779879222.0243337,
+      "req_ids": [
+        "cmpl-a9dfc1a5b425d994-0-a0930098"
+      ]
+    },
+    {
+      "input_tokens_est": 1024,
+      "total_bytes": 100663296,
+      "pure_transfer_s": 0.010396577999927104,
+      "rx_total_s": 0.01183948403922841,
+      "rx_overhead_s": 0.001442906039301306,
+      "rx_t_start_unix": 1779879222.1297998,
+      "send_t_start_unix": 1779879222.130936,
+      "req_ids": [
+        "cmpl-9712857755af2efc-0-90b2dc9b"
+      ]
+    },
+    {
+      "input_tokens_est": 1024,
+      "total_bytes": 100663296,
+      "pure_transfer_s": 0.010438029014039785,
+      "rx_total_s": 0.01214482297655195,
+      "rx_overhead_s": 0.0017067939625121653,
+      "rx_t_start_unix": 1779879222.243023,
+      "send_t_start_unix": 1779879222.2442062,
+      "req_ids": [
+        "cmpl-b4f0a10dee65acbe-0-a3c132fc"
+      ]
+    },
+    {
+      "input_tokens_est": 1024,
+      "total_bytes": 100663296,
+      "pure_transfer_s": 0.010436972021125257,
+      "rx_total_s": 0.011961110983975232,
+      "rx_overhead_s": 0.0015241389628499746,
+      "rx_t_start_unix": 1779879222.3569698,
+      "send_t_start_unix": 1779879222.3581295,
+      "req_ids": [
+        "cmpl-b4c514b80b52a3f2-0-bcd24f8e"
+      ]
+    },
+    {
+      "input_tokens_est": 1024,
+      "total_bytes": 100663296,
+      "pure_transfer_s": 0.010396371013484895,
+      "rx_total_s": 0.011788576026447117,
+      "rx_overhead_s": 0.001392205012962222,
+      "rx_t_start_unix": 1779879222.4715128,
+      "send_t_start_unix": 1779879222.4725878,
+      "req_ids": [
+        "cmpl-ac7118d8090d181c-0-8af4adf0"
+      ]
+    },
+    {
+      "input_tokens_est": 1024,
+      "total_bytes": 100663296,
+      "pure_transfer_s": 0.010352785000577569,
+      "rx_total_s": 0.0118055299972184,
+      "rx_overhead_s": 0.0014527449966408312,
+      "rx_t_start_unix": 1779879222.5826046,
+      "send_t_start_unix": 1779879222.5837166,
+      "req_ids": [
+        "cmpl-85291bcb93aaf638-0-868db1a8"
+      ]
+    },
+    {
+      "input_tokens_est": 16,
+      "total_bytes": 1572864,
+      "pure_transfer_s": 0.00034007197245955467,
+      "rx_total_s": 0.0021119200391694903,
+      "rx_overhead_s": 0.0017718480667099357,
+      "rx_t_start_unix": 1779879222.750828,
+      "send_t_start_unix": 1779879222.7521152,
+      "req_ids": [
+        "cmpl-a448cf2e059ba0c9-0-a1360796"
+      ]
+    },
+    {
+      "input_tokens_est": 16,
+      "total_bytes": 1572864,
+      "pure_transfer_s": 0.00041691696969792247,
+      "rx_total_s": 0.0022232600022107363,
+      "rx_overhead_s": 0.0018063430325128138,
+      "rx_t_start_unix": 1779879222.913044,
+      "send_t_start_unix": 1779879222.9143836,
+      "req_ids": [
+        "cmpl-b486fd9e945a4658-0-8bb561cd"
+      ]
+    },
+    {
+      "input_tokens_est": 2048,
+      "total_bytes": 201326592,
+      "pure_transfer_s": 0.020633380976505578,
+      "rx_total_s": 0.022250515001360327,
+      "rx_overhead_s": 0.0016171340248547494,
+      "rx_t_start_unix": 1779879223.0765986,
+      "send_t_start_unix": 1779879223.0778644,
+      "req_ids": [
+        "cmpl-82da2bfe65f276c6-0-88d9a9a2"
+      ]
+    },
+    {
+      "input_tokens_est": 2048,
+      "total_bytes": 201326592,
+      "pure_transfer_s": 0.020639199996367097,
+      "rx_total_s": 0.022157608007546514,
+      "rx_overhead_s": 0.0015184080111794174,
+      "rx_t_start_unix": 1779879223.2591784,
+      "send_t_start_unix": 1779879223.2603853,
+      "req_ids": [
+        "cmpl-93bd777652eba5f3-0-9ec3d058"
+      ]
+    },
+    {
+      "input_tokens_est": 2048,
+      "total_bytes": 201326592,
+      "pure_transfer_s": 0.020575353992171586,
+      "rx_total_s": 0.022589912987314165,
+      "rx_overhead_s": 0.002014558995142579,
+      "rx_t_start_unix": 1779879223.4402068,
+      "send_t_start_unix": 1779879223.4418828,
+      "req_ids": [
+        "cmpl-81f950480a3cabf9-0-bbf8584f"
+      ]
+    },
+    {
+      "input_tokens_est": 4096,
+      "total_bytes": 402653184,
+      "pure_transfer_s": 0.041439525957684964,
+      "rx_total_s": 0.043345845013391227,
+      "rx_overhead_s": 0.0019063190557062626,
+      "rx_t_start_unix": 1779879223.7529812,
+      "send_t_start_unix": 1779879223.7544343,
+      "req_ids": [
+        "cmpl-b109ed06b5882659-0-8d14993c"
+      ]
+    },
+    {
+      "input_tokens_est": 4096,
+      "total_bytes": 402653184,
+      "pure_transfer_s": 0.04152030003024265,
+      "rx_total_s": 0.04341953102266416,
+      "rx_overhead_s": 0.0018992309924215078,
+      "rx_t_start_unix": 1779879224.0899644,
+      "send_t_start_unix": 1779879224.0914912,
+      "req_ids": [
+        "cmpl-8a57776c81d64b2c-0-ace8fb2b"
+      ]
+    },
+    {
+      "input_tokens_est": 4096,
+      "total_bytes": 402653184,
+      "pure_transfer_s": 0.04148670402355492,
+      "rx_total_s": 0.04336977802449837,
+      "rx_overhead_s": 0.0018830740009434521,
+      "rx_t_start_unix": 1779879224.424807,
+      "send_t_start_unix": 1779879224.4262393,
+      "req_ids": [
+        "cmpl-9b1a5dce18758450-0-b17b3649"
+      ]
+    },
+    {
+      "input_tokens_est": 4096,
+      "total_bytes": 402653184,
+      "pure_transfer_s": 0.04146742797456682,
+      "rx_total_s": 0.043769759009592235,
+      "rx_overhead_s": 0.002302331035025418,
+      "rx_t_start_unix": 1779879224.7599711,
+      "send_t_start_unix": 1779879224.7617002,
+      "req_ids": [
+        "cmpl-8c7d412b85f43ed7-0-9dea4add"
+      ]
+    },
+    {
+      "input_tokens_est": 4096,
+      "total_bytes": 402653184,
+      "pure_transfer_s": 0.04143296502297744,
+      "rx_total_s": 0.043612666020635515,
+      "rx_overhead_s": 0.002179700997658074,
+      "rx_t_start_unix": 1779879225.0962389,
+      "send_t_start_unix": 1779879225.0978234,
+      "req_ids": [
+        "cmpl-8860308db3f010a5-0-ad51eb46"
+      ]
+    },
+    {
+      "input_tokens_est": 16,
+      "total_bytes": 1572864,
+      "pure_transfer_s": 0.0003991159610450268,
+      "rx_total_s": 0.002386144013144076,
+      "rx_overhead_s": 0.001987028052099049,
+      "rx_t_start_unix": 1779879225.7592747,
+      "send_t_start_unix": 1779879225.760789,
+      "req_ids": [
+        "cmpl-86cca1a2b9427801-0-ba41ade7"
+      ]
+    },
+    {
+      "input_tokens_est": 16,
+      "total_bytes": 1572864,
+      "pure_transfer_s": 0.00041423802031204104,
+      "rx_total_s": 0.0023903060355223715,
+      "rx_overhead_s": 0.0019760680152103305,
+      "rx_t_start_unix": 1779879226.384918,
+      "send_t_start_unix": 1779879226.3864496,
+      "req_ids": [
+        "cmpl-a208c6d804293be7-0-94d265ab"
+      ]
+    },
+    {
+      "input_tokens_est": 8192,
+      "total_bytes": 805306368,
+      "pure_transfer_s": 0.08309489500243217,
+      "rx_total_s": 0.08524628396844491,
+      "rx_overhead_s": 0.002151388966012746,
+      "rx_t_start_unix": 1779879227.0092332,
+      "send_t_start_unix": 1779879227.0107942,
+      "req_ids": [
+        "cmpl-b53bea2317cc1211-0-8fcad8a8"
+      ]
+    },
+    {
+      "input_tokens_est": 8192,
+      "total_bytes": 805306368,
+      "pure_transfer_s": 0.08372796402545646,
+      "rx_total_s": 0.08596085698809475,
+      "rx_overhead_s": 0.0022328929626382887,
+      "rx_t_start_unix": 1779879227.7190688,
+      "send_t_start_unix": 1779879227.7207224,
+      "req_ids": [
+        "cmpl-9daf909593bbdf03-0-8fd7d50e"
+      ]
+    },
+    {
+      "input_tokens_est": 8192,
+      "total_bytes": 805306368,
+      "pure_transfer_s": 0.08398396399570629,
+      "rx_total_s": 0.0860762019874528,
+      "rx_overhead_s": 0.002092237991746515,
+      "rx_t_start_unix": 1779879228.4297745,
+      "send_t_start_unix": 1779879228.4314566,
+      "req_ids": [
+        "cmpl-9ef40f3b6d736128-0-8e8e1c30"
+      ]
+    },
+    {
+      "input_tokens_est": 16384,
+      "total_bytes": 1610612736,
+      "pure_transfer_s": 0.16950496198842302,
+      "rx_total_s": 0.1721468890318647,
+      "rx_overhead_s": 0.002641927043441683,
+      "rx_t_start_unix": 1779879230.131392,
+      "send_t_start_unix": 1779879230.1334376,
+      "req_ids": [
+        "cmpl-851e5d7e3e83d7ea-0-a66a5e0b"
+      ]
+    },
+    {
+      "input_tokens_est": 16384,
+      "total_bytes": 1610612736,
+      "pure_transfer_s": 0.16713789198547602,
+      "rx_total_s": 0.16974544001277536,
+      "rx_overhead_s": 0.0026075480272993445,
+      "rx_t_start_unix": 1779879231.896075,
+      "send_t_start_unix": 1779879231.8981037,
+      "req_ids": [
+        "cmpl-9be12af6a9ccccf5-0-af1230c7"
+      ]
+    },
+    {
+      "input_tokens_est": 16384,
+      "total_bytes": 1610612736,
+      "pure_transfer_s": 0.16713115200400352,
+      "rx_total_s": 0.16975757898762822,
+      "rx_overhead_s": 0.0026264269836246967,
+      "rx_t_start_unix": 1779879233.6589305,
+      "send_t_start_unix": 1779879233.6608078,
+      "req_ids": [
+        "cmpl-b61b9b237366297b-0-9832f0e3"
+      ]
+    },
+    {
+      "input_tokens_est": 16384,
+      "total_bytes": 1610612736,
+      "pure_transfer_s": 0.16709016199456528,
+      "rx_total_s": 0.1695251659839414,
+      "rx_overhead_s": 0.0024350039893761277,
+      "rx_t_start_unix": 1779879235.4181106,
+      "send_t_start_unix": 1779879235.419875,
+      "req_ids": [
+        "cmpl-bae0d0efe47ece8f-0-affbc685"
+      ]
+    },
+    {
+      "input_tokens_est": 16384,
+      "total_bytes": 1610612736,
+      "pure_transfer_s": 0.166486973001156,
+      "rx_total_s": 0.16962904302636161,
+      "rx_overhead_s": 0.003142070025205612,
+      "rx_t_start_unix": 1779879237.1803744,
+      "send_t_start_unix": 1779879237.1821773,
+      "req_ids": [
+        "cmpl-a34bc73c9cd2efc1-0-90d647fc"
+      ]
+    },
+    {
+      "input_tokens_est": 32768,
+      "total_bytes": 3221225472,
+      "pure_transfer_s": 0.31926770601421595,
+      "rx_total_s": 0.32203804596792907,
+      "rx_overhead_s": 0.002770339953713119,
+      "rx_t_start_unix": 1779879241.9859307,
+      "send_t_start_unix": 1779879241.9880297,
+      "req_ids": [
+        "cmpl-89a36c12ee6b0ff3-0-9fddbc0f"
+      ]
+    },
+    {
+      "input_tokens_est": 32768,
+      "total_bytes": 3221225472,
+      "pure_transfer_s": 0.3197040680097416,
+      "rx_total_s": 0.3227974839974195,
+      "rx_overhead_s": 0.003093415987677872,
+      "rx_t_start_unix": 1779879246.9755645,
+      "send_t_start_unix": 1779879246.9779432,
+      "req_ids": [
+        "cmpl-8d65512eb7e3c36c-0-8b23597c"
+      ]
+    },
+    {
+      "input_tokens_est": 32768,
+      "total_bytes": 3221225472,
+      "pure_transfer_s": 0.32088329299585894,
+      "rx_total_s": 0.3240378479822539,
+      "rx_overhead_s": 0.003154554986394942,
+      "rx_t_start_unix": 1779879251.9618897,
+      "send_t_start_unix": 1779879251.9643052,
+      "req_ids": [
+        "cmpl-a13c271ecbbca78b-0-b76a0370"
+      ]
+    },
+    {
+      "input_tokens_est": 32768,
+      "total_bytes": 3221225472,
+      "pure_transfer_s": 0.5439103110111319,
+      "rx_total_s": 0.5924434679909609,
+      "rx_overhead_s": 0.04853315697982907,
+      "rx_t_start_unix": 1779879256.9512377,
+      "send_t_start_unix": 1779879256.9989722,
+      "req_ids": [
+        "cmpl-bada04ec8c556aca-0-a263d637"
+      ]
+    },
+    {
+      "input_tokens_est": 32768,
+      "total_bytes": 3221225472,
+      "pure_transfer_s": 0.5193864739849232,
+      "rx_total_s": 0.5644763479940593,
+      "rx_overhead_s": 0.04508987400913611,
+      "rx_t_start_unix": 1779879262.2127163,
+      "send_t_start_unix": 1779879262.2562187,
+      "req_ids": [
+        "cmpl-9641a077022e6123-0-8c3c0975"
+      ]
+    },
+    {
+      "input_tokens_est": 65536,
+      "total_bytes": 6442450944,
+      "pure_transfer_s": 1.9844180009677075,
+      "rx_total_s": 2.0784930550144054,
+      "rx_overhead_s": 0.09407505404669791,
+      "rx_t_start_unix": 1779879278.1063075,
+      "send_t_start_unix": 1779879278.199048,
+      "req_ids": [
+        "cmpl-bb3a4e5084af8c3a-0-bdfa0931"
+      ]
+    },
+    {
+      "input_tokens_est": 65536,
+      "total_bytes": 6442450944,
+      "pure_transfer_s": 2.1099297259934247,
+      "rx_total_s": 2.2067435560165904,
+      "rx_overhead_s": 0.09681383002316579,
+      "rx_t_start_unix": 1779879295.600993,
+      "send_t_start_unix": 1779879295.6967168,
+      "req_ids": [
+        "cmpl-91b951f85c93a71b-0-8396bee5"
+      ]
+    },
+    {
+      "input_tokens_est": 65536,
+      "total_bytes": 6442450944,
+      "pure_transfer_s": 1.8950715209939517,
+      "rx_total_s": 1.9879729640088044,
+      "rx_overhead_s": 0.0929014430148527,
+      "rx_t_start_unix": 1779879313.2315958,
+      "send_t_start_unix": 1779879313.3236735,
+      "req_ids": [
+        "cmpl-81d236ecb6aadadf-0-ac184d51"
+      ]
+    },
+    {
+      "input_tokens_est": 65536,
+      "total_bytes": 6442450944,
+      "pure_transfer_s": 0.9277855920372531,
+      "rx_total_s": 0.9849357060156763,
+      "rx_overhead_s": 0.05715011397842318,
+      "rx_t_start_unix": 1779879330.6154163,
+      "send_t_start_unix": 1779879330.6715357,
+      "req_ids": [
+        "cmpl-a4c76c62b44c4295-0-b007a6ed"
+      ]
+    },
+    {
+      "input_tokens_est": 65536,
+      "total_bytes": 6442450944,
+      "pure_transfer_s": 0.6652462020283565,
+      "rx_total_s": 0.6725030990201049,
+      "rx_overhead_s": 0.007256896991748363,
+      "rx_t_start_unix": 1779879346.990221,
+      "send_t_start_unix": 1779879346.9950044,
+      "req_ids": [
+        "cmpl-a06d4b774a8af9a5-0-980e9d23"
+      ]
+    },
+    {
+      "input_tokens_est": 131072,
+      "total_bytes": 12884901888,
+      "pure_transfer_s": 1.3330365709844045,
+      "rx_total_s": 1.3384539679973386,
+      "rx_overhead_s": 0.005417397012934089,
+      "rx_t_start_unix": 1779879402.7123013,
+      "send_t_start_unix": 1779879402.7169023,
+      "req_ids": [
+        "cmpl-bf0d435e06e3349f-0-8507c933"
+      ]
+    },
+    {
+      "input_tokens_est": 131072,
+      "total_bytes": 12884901888,
+      "pure_transfer_s": 5.839069904992357,
+      "rx_total_s": 5.973284716019407,
+      "rx_overhead_s": 0.13421481102705002,
+      "rx_t_start_unix": 1779879458.9232886,
+      "send_t_start_unix": 1779879459.0566247,
+      "req_ids": [
+        "cmpl-9f87ae0fb0c7eec8-0-a8a1daea"
+      ]
+    },
+    {
+      "input_tokens_est": 131072,
+      "total_bytes": 12884901888,
+      "pure_transfer_s": 9.862486142024864,
+      "rx_total_s": 10.056511385017075,
+      "rx_overhead_s": 0.19402524299221113,
+      "rx_t_start_unix": 1779879519.7647448,
+      "send_t_start_unix": 1779879519.9567635,
+      "req_ids": [
+        "cmpl-a62e48e40e6c6ad7-0-acca9741"
+      ]
+    },
+    {
+      "input_tokens_est": 131072,
+      "total_bytes": 12884901888,
+      "pure_transfer_s": 2.8350498770014383,
+      "rx_total_s": 2.925714804965537,
+      "rx_overhead_s": 0.09066492796409875,
+      "rx_t_start_unix": 1779879584.888362,
+      "send_t_start_unix": 1779879584.9780834,
+      "req_ids": [
+        "cmpl-824479d53bab40e4-0-af951a11"
+      ]
+    },
+    {
+      "input_tokens_est": 131072,
+      "total_bytes": 12884901888,
+      "pure_transfer_s": 1.485496642999351,
+      "rx_total_s": 1.5183607729850337,
+      "rx_overhead_s": 0.032864129985682666,
+      "rx_t_start_unix": 1779879642.6076336,
+      "send_t_start_unix": 1779879642.639775,
+      "req_ids": [
+        "cmpl-9f06f19c981c0b3f-0-b3afb370"
+      ]
+    }
+  ],
+  "summary": [
+    {
+      "input_tokens": 16,
+      "kv_mib": 1.5,
+      "n": 6,
+      "pure_transfer_ms_mean": 0.39,
+      "pure_transfer_ms_p50": 0.4,
+      "pure_transfer_ms_max": 0.42,
+      "pure_transfer_ms_min": 0.34,
+      "rx_total_ms_mean": 2.27,
+      "rx_overhead_ms_mean": 1.88,
+      "throughput_gbps_mean": 4.09,
+      "throughput_gbps_p50": 3.91,
+      "throughput_gbps_max": 4.63
+    },
+    {
+      "input_tokens": 512,
+      "kv_mib": 48.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 8.9,
+      "pure_transfer_ms_p50": 5.35,
+      "pure_transfer_ms_max": 23.2,
+      "pure_transfer_ms_min": 5.28,
+      "rx_total_ms_mean": 12.17,
+      "rx_overhead_ms_mean": 3.26,
+      "throughput_gbps_mean": 7.99,
+      "throughput_gbps_p50": 9.4,
+      "throughput_gbps_max": 9.53
+    },
+    {
+      "input_tokens": 1024,
+      "kv_mib": 96.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 10.4,
+      "pure_transfer_ms_p50": 10.4,
+      "pure_transfer_ms_max": 10.44,
+      "pure_transfer_ms_min": 10.35,
+      "rx_total_ms_mean": 11.91,
+      "rx_overhead_ms_mean": 1.5,
+      "throughput_gbps_mean": 9.68,
+      "throughput_gbps_p50": 9.68,
+      "throughput_gbps_max": 9.72
+    },
+    {
+      "input_tokens": 2048,
+      "kv_mib": 192.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 20.75,
+      "pure_transfer_ms_p50": 20.64,
+      "pure_transfer_ms_max": 21.17,
+      "pure_transfer_ms_min": 20.58,
+      "rx_total_ms_mean": 22.52,
+      "rx_overhead_ms_mean": 1.77,
+      "throughput_gbps_mean": 9.7,
+      "throughput_gbps_p50": 9.75,
+      "throughput_gbps_max": 9.78
+    },
+    {
+      "input_tokens": 4096,
+      "kv_mib": 384.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 41.47,
+      "pure_transfer_ms_p50": 41.47,
+      "pure_transfer_ms_max": 41.52,
+      "pure_transfer_ms_min": 41.43,
+      "rx_total_ms_mean": 43.5,
+      "rx_overhead_ms_mean": 2.03,
+      "throughput_gbps_mean": 9.71,
+      "throughput_gbps_p50": 9.71,
+      "throughput_gbps_max": 9.72
+    },
+    {
+      "input_tokens": 8192,
+      "kv_mib": 768.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 83.97,
+      "pure_transfer_ms_p50": 83.73,
+      "pure_transfer_ms_max": 85.37,
+      "pure_transfer_ms_min": 83.09,
+      "rx_total_ms_mean": 86.17,
+      "rx_overhead_ms_mean": 2.2,
+      "throughput_gbps_mean": 9.59,
+      "throughput_gbps_p50": 9.62,
+      "throughput_gbps_max": 9.69
+    },
+    {
+      "input_tokens": 16384,
+      "kv_mib": 1536.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 167.47,
+      "pure_transfer_ms_p50": 167.13,
+      "pure_transfer_ms_max": 169.5,
+      "pure_transfer_ms_min": 166.49,
+      "rx_total_ms_mean": 170.16,
+      "rx_overhead_ms_mean": 2.69,
+      "throughput_gbps_mean": 9.62,
+      "throughput_gbps_p50": 9.64,
+      "throughput_gbps_max": 9.67
+    },
+    {
+      "input_tokens": 32768,
+      "kv_mib": 3072.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 404.63,
+      "pure_transfer_ms_p50": 320.88,
+      "pure_transfer_ms_max": 543.91,
+      "pure_transfer_ms_min": 319.27,
+      "rx_total_ms_mean": 425.16,
+      "rx_overhead_ms_mean": 20.53,
+      "throughput_gbps_mean": 8.47,
+      "throughput_gbps_p50": 10.04,
+      "throughput_gbps_max": 10.09
+    },
+    {
+      "input_tokens": 65536,
+      "kv_mib": 6144.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 1516.49,
+      "pure_transfer_ms_p50": 1895.07,
+      "pure_transfer_ms_max": 2109.93,
+      "pure_transfer_ms_min": 665.25,
+      "rx_total_ms_mean": 1586.13,
+      "rx_overhead_ms_mean": 69.64,
+      "throughput_gbps_mean": 5.27,
+      "throughput_gbps_p50": 3.4,
+      "throughput_gbps_max": 9.68
+    },
+    {
+      "input_tokens": 131072,
+      "kv_mib": 12288.0,
+      "n": 5,
+      "pure_transfer_ms_mean": 4271.03,
+      "pure_transfer_ms_p50": 2835.05,
+      "pure_transfer_ms_max": 9862.49,
+      "pure_transfer_ms_min": 1333.04,
+      "rx_total_ms_mean": 4362.47,
+      "rx_overhead_ms_mean": 91.44,
+      "throughput_gbps_mean": 5.28,
+      "throughput_gbps_p50": 4.54,
+      "throughput_gbps_max": 9.67
+    }
+  ]
+}

analysis/mb5_pd_ablation/README.md

@@ -0,0 +1,107 @@
+# PD-disagg vs colocation — controlled reuse & concurrency axes (v2)
+
+Self-contained results for the **controlled-variable** redo of the MB5 PD-vs-colo
+ablation. Supersedes the confounded first cut (held input fixed and sliced the
+prefix, so "more reuse" was entangled with "less prefill"). All arms route through
+the proxy at fair **APC parity** (session-routed producers reach the same prefix-cache
+hit rate as colo), so PD loses on *structure*, not on broken cache.
+
+- **Config arms:** `colo` = 8×kv_both (8C-proxy, session-affinity); PD = `6P+2D / 4P+4D / 2P+6D`.
+- **Driver:** closed-loop N (`REPLAY_MAX_INFLIGHT`) + fixed think-time; `gen_synthetic_trace.py --mode regular`.
+- **PD-arm wall-cap:** collapsed PD arms drain pathologically slowly, so PD arms run with a
+  wall-deadline (`REPLAY_MAX_DURATION`; un-run turns counted as failures → honest completion%);
+  **colo is uncapped** so the reference is always fully measured.
+- **Hardware:** run on **dash2** (8×H20). dash0's RDMA NICs were faulted for Mooncake during
+  this work (could not init the transfer engine; needs an admin reset — no sudo); dash2's NICs
+  are healthy. cpfs/venv/data are shared across the boxes.
+
+---
+
+## 1. Reuse / APC axis — fixed real prefill, vary cached prefix
+
+N=8. Hold the **real new-prefill work per turn constant** (`--delta-len`) and grow the cached
+prefix → reuse = prefix/(prefix+delta). Three shapes isolate output vs delta:
+
+| | delta (real prefill/turn) | output | role |
+|---|---|---|---|
+| **A** | 2048 | 256 | original |
+| **C** | 2048 | 128 | A vs C = pure **output** 256→128 |
+| **B** | 1024 | 128 | C vs B = pure **delta** 2048→1024 |
+
+**PD-best advantage** = colo E2E-p90 / best-PD E2E-p90 (>1 ⇒ PD wins):
+
+| reuse% | A d2048/o256 | C d2048/o128 | B d1024/o128 |
+|---|---|---|---|
+| 20 | 1.34 | 1.41 | — |
+| 50 | 1.36 | 1.37 | — |
+| 67 | **1.47** | **1.49** | **1.27** |
+| 80 | 1.31 | 1.23 | 1.25 |
+| 90 | 1.15 | 1.01 | — |
+| 95 | 0.87 | 0.89 | 0.89 |
+
+![reuse 3-way](../../figs/mb5_pd_ablation/reuse_compare_ABC.png)
+
+**Findings:**
+1. **Output length is ~negligible.** A and C (same delta) track each other across the whole
+   range — halving output barely moves PD's advantage.
+2. **Delta (real prefill/turn) is the dominant shape factor.** B (delta=1024) sits clearly
+   below A/C at mid reuse (67%: 1.27 vs ~1.48). More real prefill per turn → bigger PD win,
+   because PD-disagg's benefit is isolating prefill from decode — more prefill to isolate.
+3. **Crossover to colo at reuse ~90–95% is robust** across all three shapes: PD always loses
+   the high-reuse / large-resident-context corner (it must KV-transfer the whole resident
+   context every turn for a few hundred new tokens; colo keeps it local).
+
+*Caveat:* the clean, uncapped, 100%-completion comparison region is reuse **20–80%** (carries
+findings 1–2). At reuse 90/95% the PD arms collapse and C's points are capped-completion, while
+A/B are full-drain — comparable in direction, not in exact PD completion%.
+
+Data: `fig1_reuse_fixed.json` (A), `fig1_reuse_d2048_o128.json` (C), `fig1_reuse_d1024_o128.json` (B).
+
+---
+
+## 2. Concurrency axis — agentic corner, sweep N
+
+in=32768 (prefix 32256 + delta 512, **reuse 0.984**), out=128; closed-loop N ∈ {8,16,32,48,64,96,128};
+PD arms capped 600s, colo uncapped.
+
+| N | **colo** completion · E2E-mean · TPS | best PD-arm completion |
+|---|---|---|
+| 8 | **256/256** · 2.4s · 326 | 6P+2D 256/256 |
+| 16 | **512/512** · 3.5s · 462 | 6P+2D 439/512 (86%) |
+| 32 | **1024/1024** · 13.3s · 190 | all PD **<27%** |
+| 48 | **1536/1536** · 24.9s · 168 | all PD <32% |
+| 64 | **2048/2048** · 38.4s · 166 | all PD <31% |
+| 96 | **3072/3072** · 60.0s · 171 | PD **2–7%** |
+| 128 | **4096/4096** · 80.8s · 181 | 4P+4D 6%, 2P+6D <1% |
+
+![concurrency](../../figs/mb5_pd_ablation/fig3_concurrency_axis.png)
+
+**Finding:** **colo completes 100% of requests at every concurrency level** — it degrades
+*gracefully* (latency rises 2.4s→81s, nothing dropped). **Every static PD split collapses, and
+progressively earlier as N rises**: PD is viable only at N≤8–16; by N≥32 it drops 70–99% of
+requests while its prefix-cache hit-rate craters to ~0%. colo's elastic pool absorbs the
+time-varying P/D demand; the static partition + per-turn 32k KV-transfer cannot. (Latency
+percentiles count successes only, so they *understate* PD — read them with the completion column.)
+
+Data: `fig3_conc32k.json`.
+
+*Caveat:* N=128 6P+2D is missing (one transient vLLM/Mooncake startup flake at the end); does
+not change the picture (all PD arms are already collapsed by N=128). The SLO auto-stop in the
+driver is a no-op (a stdout-capture bug), so the full grid ran — more points, not fewer.
+
+---
+
+## 3. Reproduce
+
+```bash
+# on a box with healthy Mooncake/RDMA NICs (dash2), cpfs mounted:
+R=/home/admin/cpfs/wjh/agentic-kv-fresh
+# reuse axis (three shapes): DELTA/OL pick the shape; tag carries _o${OL}
+ssh dash2 "cd $R && DELTA=2048 OL=256 bash microbench/fresh_setup/run_reuse_fixed.sh"
+ssh dash2 "cd $R && DELTA=2048 OL=128 bash microbench/fresh_setup/run_reuse_fixed.sh"
+ssh dash2 "cd $R && DELTA=1024 OL=128 bash microbench/fresh_setup/run_reuse_fixed.sh"
+# concurrency axis (capped):
+ssh dash2 "cd $R && NLIST='8 16 32 48 64 96 128' CONC_PD_MAXDUR=600 bash microbench/fresh_setup/run_conc.sh"
+# render (reads the *.json in this dir):
+python microbench/fresh_setup/plot_pd_crossover.py
+```

analysis/migration_trigger_validation/README.md

@@ -0,0 +1,147 @@
+# Migration Trigger Validation (unified_v4) — 2026-05-30
+
+Hardware: dash2, 8×H20, Qwen3-Coder-30B-A3B, TP=1, kv_both + DR-fix substrate.
+Trace: `w600_r0.0015_st30_first600s.jsonl` (807 reqs, 600s span).
+Policy: `unified_v4` = unified hybrid routing + pending-prefill-queue-triggered
+session migration (commit `3a6bf5d` on `kzlin-dev` branch).
+
+## Research Question
+
+Does Pillar 2 (hot-triggered session migration) provide measurable benefit on
+top of Pillar 1 (affinity-default routing)?
+
+## Experiment Design
+
+| Arm | Policy | Substrate | Trace QPS |
+|---|---|---|---|
+| unified_1x | unified (affinity-only) | kv_both + DR-fix | ~1.3 (original) |
+| unified_v4_1x | unified_v4 (affinity + migration) | kv_both + DR-fix | ~1.3 |
+| unified_v4_2x | unified_v4 (affinity + migration) | kv_both + DR-fix | ~2.7 (2× compressed) |
+
+The 2× trace was generated by halving inter-request intervals:
+`ts_new = ts_min + (ts_orig - ts_min) / 2`.
+
+## Results
+
+### 1x QPS: unified vs unified_v4
+
+| Metric | unified | unified_v4 | Delta |
+|---|---:|---:|---:|
+| OK/total | 807/807 | 807/807 | — |
+| TTFT mean | 3.990s | 4.142s | +3.8% |
+| TTFT p50 | 0.719s | 0.711s | −1.0% |
+| TTFT p90 | 11.499s | 12.293s | +6.9% |
+| TPOT p90 | 0.024s | 0.022s | −9.3% |
+| E2E p50 | 2.265s | 2.293s | +1.2% |
+| E2E p90 | 24.507s | 23.955s | −2.3% |
+| **Migrations** | **0** | **0** | — |
+
+**Conclusion**: At 1x QPS (~1.3 req/s, ~0.16 req/instance/s), the migration
+trigger NEVER fires. The two arms produce statistically identical results.
+
+### 2x QPS: unified_v4 under higher load
+
+| Metric | unified_v4 @ 2x |
+|---|---:|
+| OK/total | 807/807 |
+| TTFT mean | 5.227s |
+| TTFT p90 | 15.000s |
+| E2E p90 | 39.401s |
+| **Migrations** | **4/807 (0.5%)** |
+
+4 migrated requests (all verified via `v3_decode_target_url` in breakdown):
+
+| Session | Input | new_local | src_pp | fleet_median | proj_prefill | Target |
+|---|---:|---:|---:|---:|---:|---|
+| 1313181 | 22,686 | 22,686 | 13,360 | 6,680 | 5.1s | inst_5 |
+| 1310590 | 32,440 | 14,520 | 57,051 | 12,630 | 10.2s | inst_4 |
+| 1373431 | 126,340 | 126,340 | 73,385 | 33,294 | 28.5s | inst_4 |
+| 1313181 | 60,004 | 17,508 | 19,503 | 3,806 | 5.3s | inst_5 |
+
+## Root Cause Analysis: Why Zero Migrations at 1x
+
+The unified_v4 trigger requires BOTH arms to fire simultaneously:
+- **Absolute SLO arm**: `proj_prefill_s(src) > 2.5s` — fires for 41% of eligible requests
+- **Relative arm**: `src_pending_prefill > fleet_median × 1.5` — NEVER fires at 1x
+
+The relative arm fails because `pending_prefill_tokens` (the proxy's shadow
+counter) is 0 for **95% of routing decisions** at 1x QPS:
+
+| QPS | src_pp > 0 (% of eligible) | Migrations |
+|---:|---:|---:|
+| 1.3 (1×) | 5% (8/241) | 0 |
+| 2.7 (2×) | 24% (62/257) | 4 |
+
+**Mechanism**: `pending_prefill_tokens` reflects previously-dispatched requests
+that haven't finished their prefill yet. At 0.16 req/instance/s, each instance
+completes its prefill before the next request arrives — the counter is almost
+always 0 at decision time. Only under genuine queueing pressure (2× and above)
+does the counter become non-zero and the relative arm can fire.
+
+The high TTFT at 1x (~11.5s p90) comes from **compute-bound large prefills**
+(single 60k+ token requests inherently need ~9s), NOT from queue depth.
+
+## Interpretation for Paper
+
+1. **The migration mechanism is functionally correct.** At 2x it fires on the
+   right signal (src genuinely overloaded relative to fleet) and selects valid
+   targets (cooler instances with load gap).
+
+2. **At benchmark scale (8 instances, ~1 QPS), migration is not needed.** The
+   affinity-default routing (Pillar 1) already achieves APC ~79% and the
+   remaining hot-pin issue is mild (max-worker/median-worker ≈ 3.7×). The
+   "dispatch coupling" feedback loop is present but not yet at the catastrophic
+   amplification regime.
+
+3. **Migration becomes relevant under scale-out + higher load.** With more
+   instances (16–32), session skew concentrates more load per hot instance
+   while cold instances sit idle — exactly the condition where `src_pp >
+   fleet_median × 1.5` naturally fires. The 1x→2x progression (0%→0.5%
+   migration rate) shows the correct scaling direction.
+
+4. **Paper §3.3 framing**: Migration is a **scale-out insurance mechanism**
+   that gracefully degrades to no-op under low load. Its value is NOT
+   demonstrable at 8-instance single-node benchmark; the argument must rely on
+   (a) the mechanism's correctness (this experiment), (b) the substrate's
+   net-positive property (commit `ef9e010`), and (c) scale-out projection
+   (future: 16+ GPU, multi-node).
+
+## Next Steps
+
+- [ ] **Scale-out validation** (16 GPU, 2 nodes): With more instances and the
+  same trace, more sessions compete per-instance → higher pending_prefill →
+  migration triggers naturally. This is the strongest evidence path.
+- [ ] **3–4× QPS on 8 instances**: Push to saturation to measure migration's
+  effect in the catastrophic regime. Risk: may exceed serving capacity (errors).
+- [ ] **Threshold sensitivity**: Ablate `v4_rel_hi` (1.5→1.2→1.0) and
+  `v4_ttft_slo_s` (2.5→1.5→1.0) to characterize the trigger landscape.
+
+## Reproduction
+
+```bash
+# On dash2 (local /tmp, does NOT modify shared NAS):
+# 1x QPS
+bash /tmp/migration_exp/run_migration_ab.sh   # interleaved unified vs unified_v4
+
+# 2x QPS
+python3 -c "
+import json
+trace_in = '/home/admin/cpfs/wjh/agentic-kv/traces/w600_r0.0015_st30_first600s.jsonl'
+rows = [json.loads(l) for l in open(trace_in)]
+ts_min = min(r['timestamp'] for r in rows)
+for r in rows: r['timestamp'] = ts_min + (r['timestamp'] - ts_min) / 2.0
+with open('/tmp/migration_exp/trace_2x.jsonl', 'w') as f:
+    for r in rows: f.write(json.dumps(r) + '\n')
+"
+bash /tmp/migration_exp/run_2x.sh
+```
+
+## Data Locations (dash2 /tmp, ephemeral)
+
+| Path | Content |
+|---|---|
+| `/tmp/migration_exp/outputs/unified_run1/` | Baseline arm (1x) |
+| `/tmp/migration_exp/outputs/unified_v4_run1/` | Migration arm (1x) |
+| `/tmp/migration_exp/outputs/unified_v4_2x/` | Migration arm (2x) |
+| `/tmp/migration_exp/outputs/*/breakdown.json` | Per-request routing decisions with v4_* fields |
+| `/tmp/migration_exp/outputs/*/metrics.jsonl` | Per-request latency metrics |

analysis/migration_trigger_validation/results.json

@@ -0,0 +1,110 @@
+{
+  "experiment": "migration_trigger_validation",
+  "date": "2026-05-30",
+  "hardware": "dash2, 8xH20, Qwen3-Coder-30B-A3B, TP=1",
+  "substrate": "kv_both + DR-fix (delay_free_blocks + VLLM_EVICT_SENT_BLOCKS gate)",
+  "arms": {
+    "unified_1x": {
+      "metrics": {
+        "ok": 807,
+        "total": 807,
+        "ttft_mean": 3.99,
+        "ttft_p50": 0.719,
+        "ttft_p90": 11.499,
+        "ttft_p99": 45.982,
+        "tpot_p90": 0.0239,
+        "e2e_p50": 2.265,
+        "e2e_p90": 24.507,
+        "e2e_p99": 71.233
+      },
+      "migrations": 0
+    },
+    "unified_v4_1x": {
+      "metrics": {
+        "ok": 807,
+        "total": 807,
+        "ttft_mean": 4.142,
+        "ttft_p50": 0.711,
+        "ttft_p90": 12.293,
+        "ttft_p99": 46.148,
+        "tpot_p90": 0.0217,
+        "e2e_p50": 2.293,
+        "e2e_p90": 23.955,
+        "e2e_p99": 75.915
+      },
+      "trigger_summary": {
+        "trace": "w600_r0.0015_st30_first600s.jsonl",
+        "qps_factor": 1,
+        "total_requests": 807,
+        "migrations_triggered": 13,
+        "size_floor_filtered": 552,
+        "eligible_requests": 255,
+        "slo_arm_true": 100,
+        "src_pp_nonzero": 22
+      }
+    },
+    "unified_v4_2x": {
+      "metrics": {
+        "ok": 807,
+        "total": 807,
+        "ttft_mean": 5.227,
+        "ttft_p50": 0.942,
+        "ttft_p90": 15.0,
+        "ttft_p99": 59.227,
+        "tpot_p90": 0.1087,
+        "e2e_p50": 5.035,
+        "e2e_p90": 39.401,
+        "e2e_p99": 163.032
+      },
+      "trigger_summary": {
+        "trace": "trace_2x.jsonl (timestamps / 2)",
+        "qps_factor": 2,
+        "total_requests": 807,
+        "migrations_triggered": 4,
+        "size_floor_filtered": 550,
+        "eligible_requests": 257,
+        "slo_arm_true": 133,
+        "src_pp_nonzero": 62,
+        "pending_prefill_p90_when_nonzero": 68131,
+        "migrated_details": [
+          {
+            "session_id": "1313181",
+            "input_length": 22686,
+            "new_local": 22686,
+            "src_pending_prefill": 13360,
+            "fleet_median_pp": 6680.0,
+            "proj_prefill_s": 5.15,
+            "target_idx": 5
+          },
+          {
+            "session_id": "1310590",
+            "input_length": 32440,
+            "new_local": 14520,
+            "src_pending_prefill": 57051,
+            "fleet_median_pp": 12630.5,
+            "proj_prefill_s": 10.22,
+            "target_idx": 4
+          },
+          {
+            "session_id": "1373431",
+            "input_length": 126340,
+            "new_local": 126340,
+            "src_pending_prefill": 73385,
+            "fleet_median_pp": 33294.5,
+            "proj_prefill_s": 28.53,
+            "target_idx": 4
+          },
+          {
+            "session_id": "1313181",
+            "input_length": 60004,
+            "new_local": 17508,
+            "src_pending_prefill": 19503,
+            "fleet_median_pp": 3806.5,
+            "proj_prefill_s": 5.29,
+            "target_idx": 5
+          }
+        ]
+      }
+    }
+  }
+}

analysis/pd_separation_analysis.md

@@ -23,6 +23,22 @@ Per-request breakdown shows **87.7% of TTFT** is spent waiting for KV cache memo
 > Earlier cross-machine comparison (commit `1e86285`) was invalidated — baseline used warm instances. See REPORT.md §3.5.
 | **Delta** | **-45%** | **-36%** | **-44%** | **+30pp** |
 
+> ✅⚠️ **2026-05-30 — confirmed + refined by the clean MB5 ablation; one caveat.**
+> A producer-side contamination (`e13391e`: evicts a producer's prefix-cache on every
+> KV transfer) was found in the *agentic-kv-fresh* MB5 stack and gated off; everything
+> was re-run clean.
+> - **Confirmed:** this doc's central thesis — PD's failure is a **decode-side KV memory
+>   wall**, not transfer/prefill cost — is reproduced on the clean stack (concurrency
+>   axis: at N=64 the split collapses, APC 71%→1.4%, TPS −30%; colo scales). Fig 3.
+> - **Refined:** "PD separation is net negative" is **regime-dependent**, not universal.
+>   Clean ablation shows PD *wins* at low load / decode-heavy / low-reuse and loses the
+>   **agentic corner** (high reuse + short output + large context + high concurrency).
+> - **Caveat (cross-check):** if this study's runs used the fork vLLM that contains
+>   `e13391e`, any **producer prefix-cache / APC reuse** figures here (e.g. §5.3.1) may be
+>   understated. The decode-memory-wall result is *not* reuse-dependent and is unaffected.
+>   On the clean stack, session-routed producers reach APC parity with colo (71–82%).
+> Figures: [`figs/mb5_pd_ablation/`](../figs/mb5_pd_ablation/).
+
 ---
 
 ## 1. Workload Characterization

analysis/v2/PD_DISAGG_LMETRIC.md

@@ -0,0 +1,165 @@
+# PD-colo vs PD-disagg on the real agentic trace — LMetric (cache-aware) clean-stack anchor
+
+**Figure:** [`figs/v2/fig4_lmetric_pd_vs_colo.png`](../../figs/v2/fig4_lmetric_pd_vs_colo.png)
+**Data:**   [`analysis/v2/fig4l_lmetric.json`](fig4l_lmetric.json)
+**Date:**   2026-05-31 · Hardware: dash1, 8×H20 · Model: Qwen3-Coder-30B-A3B-Instruct
+· vLLM 0.18.1 (V1, chunked-prefill on, `e13391e` eviction gated **off**)
+· Mooncake 0.3.11 · Routing: cache-aware proxy with **`--policy lmetric`**
+· Replayer per-request timeout 600 s.
+
+## TL;DR
+
+On the production agentic trace with the *right* routing baseline (LMetric, cache-aware),
+**PD-colo (8× kv_both) keeps 100 % completion on both traces** and matches the daily-bench
+expectation (~17 min for the high-load first600s, ~50 min for the full trace, with E2E p50
+~3 s and TTFT p50 ~1 s — **3.5–7× better than the original §3 round-robin baseline at the
+same wall-clock**). Every static **PD-disagg ratio fails** (14–65 % completion), and the
+failure mode rotates predictably with the split — **no static partition has a working
+operating point on this workload**. LMetric improves colo dramatically; it does *not*
+rescue PD-disagg, confirming the bottleneck is structural (D-pool admission capacity +
+multi-turn KV accumulation), not routing. A follow-up linear-policy run with PD-disagg
+**wall-capped at 2× the colo wall** (see end of doc) hits the **identical** success-rate
+ceiling — confirming the cap is structural, not policy-driven.
+
+## Setup
+
+- Trace: `w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions, agentic multi-turn,
+  contexts up to ~112 k tokens; "first600s" variant = same heavy sessions compressed
+  into 600 s → 807 reqs at 3.2× higher arrival rate).
+- 8 instances on 8 GPUs.
+- `--mode baseline` for colo (plain vLLM); `--mode pdsep --pd-ratio P:D` for the three PD
+  splits, all with Mooncake KV transfer.
+- Cache-aware proxy with LMetric scoring (`P_tokens × num_requests`) + session affinity
+  for multi-turn (the colleague's canonical baseline).
+
+## Results
+
+### first600s (1.35 req/s, high-load stress)
+
+| arm | success | E2E mean / p50 / p90 / p99 | TTFT p90 | TPOT p99 | TPS | wall |
+|---|---|---|---|---|---|---|
+| **colo (8C)** | **807/807 = 100 %** | 11.1 / 3.27 / 28.6 / 95.9 s | 14.5 s | 388 ms | 226 | 17.0 min |
+| pd6 (6:2) | 474/807 = **58.7 %** | 83.2 / 6.75 / 382 / 542 s | 380 s | 19 ms | 40 | 55 min |
+| pd4 (4:4) | 348/807 = **43.1 %** | 203 / 215 / 477 / 575 s | 475 s | 25 ms | 15 | 114 min |
+| pd2 (2:6) | 180/807 = **22.3 %** | 380 / 536 / 579 / 602 s | 577 s | 18 ms | 34 | 321 min* |
+
+### Full trace (0.42 req/s, original §3 anchor load)
+
+| arm | success | E2E mean / p50 / p90 / p99 | TTFT p90 | TPOT p99 | TPS | wall |
+|---|---|---|---|---|---|---|
+| **colo (8C)** | **1214/1214 = 100 %** | 10.9 / 3.13 / 29.6 / 93.7 s | 16.9 s | 254 ms | 125 | 49.9 min |
+| pd6 (6:2) | 793/1214 = **65.3 %** | 61.9 / 3.70 / 307 / 477 s | 300 s | 18 ms | 46 | 94 min |
+| pd4 (4:4) | 533/1214 = **43.9 %** | 131 / 8.22 / 468 / 531 s | 467 s | 21 ms | 13 | 231 min |
+| pd2 (2:6) | 169/1214 = **13.9 %** | 195 / 6.82 / 552 / 593 s | 549 s | 13 ms | 1 | 563 min |
+
+\* The pd2 wall-clock is dominated by per-request timeouts (`request_timeout=600 s`)
+draining concurrently behind the multi-turn session causality.
+
+## Five clean findings
+
+1. **LMetric+colo is the right baseline.** Full-trace colo wall **49.9 min ≈ the original
+   §3 RR's 49.9 min**, but E2E p50 **3.13 s vs §3's 10.8 s (3.5×)** and TTFT p50
+   **1.02 s vs §3's 7.0 s (7×)**. Same throughput envelope, far better latency — by virtue
+   of cache-aware routing concentrating each session's turns onto one instance for
+   prefix-cache reuse. The original §3 RR was an *unfairly weak* colo baseline.
+
+2. **Every static PD-disagg ratio fails on the agentic workload.** Completion drops to
+   14–65 %, on *both* traces. The drop is not a high-load artifact (it holds at the
+   original §3 arrival rate of 0.42 req/s); it is structural.
+
+3. **Failure mode rotates predictably with the P:D split:**
+   - **pd2 (2 producers)** → prefill-bound → 78–86 % TTFT timeouts.
+   - **pd6 (2 decode)** → decode-admission-bound → 35–41 % TTFT timeouts.
+   - **pd4 (4P+4D)** → both bottlenecks hit → 57 % TTFT timeouts.
+   - **No static ratio works.** Colo's elastic 8-GPU pool absorbs whichever phase is
+     hot at the moment.
+
+4. **Decode isolation works, but doesn't matter under failure.** TPOT p99 on every PD
+   arm is **13–25 ms** — an order of magnitude better than colo's 254–388 ms — but the
+   win applies only to the 14–65 % of requests that get admitted. The other 35–86 %
+   time out before ever seeing a first token, so the TPOT win is invisible to the end user.
+
+5. **The §3 RR "100 % PD completion" was a measurement artifact.** Original §3
+   (contaminated stack, RR routing) reported 100 % completion for pd6/pd4. LMetric on
+   the clean stack shows 44–65 %. Most plausible mechanism: `e13391e`'s eviction of
+   producer KV on every transfer acted as a **relief valve**, reducing producer-pool
+   pressure and letting more requests squeeze under the 600 s timeout — at the (uncosted)
+   price of cross-turn re-prefill. With the eviction off, producers retain prefix
+   correctly → cache works on PD too → but the cache itself contends for producer
+   pool capacity, and the decode-pool admission ceiling tips earlier. **PD-disagg is
+   worse on agentic than §3 advertised, not better.**
+
+## Linear-policy + wall-cap follow-up (2026-06-01) — the success ceiling is policy-invariant
+
+To check whether the LMetric routing was secretly handicapping PD-disagg, we re-ran
+first600s with the **default `--policy linear`** (cache-aware load score + sticky
+session affinity — the original baseline of the cache_aware_proxy stack) and
+**wall-capped each PD-disagg arm at 2 × the colo wall** (kill bench.sh + cleanup
+GPUs once cap is exceeded, record `records_at_cap`).
+
+| arm | linear success | linear wall | linear @-cap? | lmetric success | lmetric wall |
+|---|---|---|---|---|---|
+| **colo** | 807/807 = **100 %** | 964 s | — | 807/807 = **100 %** | 1021 s |
+| **pd6 (6:2)** | **472/807 = 58 %** | 2280 s | ⊗ cap (706 dispatched) | 474/807 = 59 % | 3325 s |
+| **pd4 (4:4)** | **349/807 = 43 %** | 2281 s | ⊗ cap (577 dispatched) | 348/807 = 43 % | 6850 s |
+| **pd2 (2:6)** | **176/807 = 22 %** | 2280 s | ⊗ cap (521 dispatched) | 180/807 = 22 % | 19275 s |
+
+→ Figure: [`figs/v2/fig4_linear_vs_lmetric.png`](../../figs/v2/fig4_linear_vs_lmetric.png) ·
+data: [`fig4r_linear.json`](fig4r_linear.json)
+
+**Three clean conclusions from the wall-cap experiment:**
+
+1. **The success-rate ceiling is structural, not a routing artifact.** Linear and
+   LMetric — two very different scoring policies (one session-sticky cache-aware,
+   the other non-sticky pure load) — converge on **identical success rates**
+   (58 / 43 / 22 %) for every PD-disagg ratio. Routing has *zero* effect on the
+   completion ceiling. The bottleneck is the static P:D split itself.
+
+2. **LMetric's longer wall was wall *wasted on requests that will never succeed*.**
+   When the cap is enforced, linear hits the same ceiling in 2280 s as LMetric did
+   in 3300–19000 s — the extra wall just slowly times out the unreachable
+   requests at 600 s each.
+
+3. **The wall-cap is the right way to bench PD-disagg.** Reporting "completion %"
+   without a wall budget is misleading (the bench eventually completes if you wait
+   forever, but only by counting timeouts as failures over hours). The honest
+   metric is **success-in-2×-colo-wall**, which gives the same answer for both
+   routings and matches what an end user would observe on a real SLO.
+
+This **strengthens** the §5 D-pool capacity-ceiling thesis: even with
+session-affinity routing serving every request to a warm prefix cache (which
+*should* maximise PD's throughput), the static D-pool can't admit more than
+~22 / 43 / 58 % of the agentic trace within 2× the colo budget. Colo wins not
+because routing is smarter, but because its **elastic pool** absorbs whichever
+phase is hot — there's no cap to hit.
+
+---
+
+## Reproduce
+
+```bash
+# On dash1, from the main repo /home/admin/cpfs/wjh/agentic-kv:
+for TR in w600_r0.0015_st30.jsonl w600_r0.0015_st30_first600s.jsonl; do
+  TRACE=traces/$TR bash scripts/bench.sh --tag fig4l_lmetric_colo_${TR%.*} \
+    --mode baseline --policy lmetric
+  for r in 6:2 4:4 2:6; do
+    TRACE=traces/$TR bash scripts/bench.sh --tag fig4l_lmetric_${r/:/p}_${TR%.*} \
+      --mode pdsep --pd-ratio $r --policy lmetric
+  done
+done
+
+python microbench/fresh_setup/plot_fig4l_lmetric.py
+python microbench/fresh_setup/plot_fig4_linear_vs_lmetric.py
+```
+
+For the linear + 2× wall-cap variant, run colo first to get `wall_clock_s`,
+compute `CAP=2*wall`, then launch each PD-disagg arm in the background and
+`SIGTERM` it (so bench.sh's cleanup trap fires) once `date +%s` minus the
+arm's start time exceeds `CAP`. The capped runs lack `metrics.summary.json`
+(replayer was killed before it could write); compute the summary directly from
+`metrics.jsonl` (see the inline collector used to build
+`analysis/v2/fig4r_linear.json`).
+
+Source `bench.sh` cleans GPUs before each arm and writes `metrics.jsonl` +
+`metrics.summary.json` per tag. Aggregation script: see the inline JSON dump used
+to build `analysis/v2/fig4l_lmetric.json`.

analysis/v2/fig4l_lmetric.json

@@ -0,0 +1 @@
+[{"tag": "fig4l_lmetric_colo_first600s", "arm": "colo", "trace": "first600s", "n": 807, "req": 807, "e2e": {"count": 807.0, "mean": 11.066699584425269, "p50": 3.27055042097345, "p90": 28.745733462180937, "p99": 97.40008939541167}, "ttft": {"count": 807.0, "mean": 5.119651803458883, "p50": 1.2114678020589054, "p90": 14.777630288852365, "p99": 50.68302261995841}, "tpot": {"count": 807.0, "mean": 0.03004899278845205, "p50": 0.009643197803618922, "p90": 0.042092699501536976, "p99": 0.3919741264067197}, "wall": 1020.5351374909515, "tps": 226.12940164644368}, {"tag": "fig4l_lmetric_colo_full", "arm": "colo", "trace": "full", "n": 1214, "req": 1214, "e2e": {"count": 1214.0, "mean": 10.928977524270508, "p50": 3.1279119075043127, "p90": 30.011970606888667, "p99": 94.77313101590481}, "ttft": {"count": 1214.0, "mean": 5.533819193267678, "p50": 1.017395684029907, "p90": 17.36427243486981, "p99": 51.49416554694993}, "tpot": {"count": 1214.0, "mean": 0.02049970290344434, "p50": 0.009544484575988867, "p90": 0.032480608771520716, "p99": 0.26057810739537074}, "wall": 2993.276069591986, "tps": 125.38402448497122}, {"tag": "fig4l_lmetric_pd2_first600s", "arm": "2P+6D", "trace": "first600s", "n": 180, "req": 807, "e2e": {"count": 180.0, "mean": 380.2505690135715, "p50": 535.6594606440049, "p90": 579.5011055286858, "p99": 601.5567972306756}, "ttft": {"count": 180.0, "mean": 378.7133691522933, "p50": 534.4269686369807, "p90": 577.3534130641376, "p99": 596.404559875431}, "tpot": {"count": 180.0, "mean": 0.007975266077679418, "p50": 0.007166497974743372, "p90": 0.012511071875514153, "p99": 0.017508981961061446}, "wall": 19275.367093455978, "tps": 1.8895100582735462}, {"tag": "fig4l_lmetric_pd2_full", "arm": "2P+6D", "trace": "full", "n": 169, "req": 1214, "e2e": {"count": 169.0, "mean": 194.88523891245458, "p50": 6.817620265996084, "p90": 552.1569225640735, "p99": 595.3934216396092}, "ttft": {"count": 169.0, "mean": 193.4153314989016, "p50": 5.60239192598965, "p90": 549.3611521873856, "p99": 582.4436428000824}, "tpot": {"count": 169.0, "mean": 0.007747395842651413, "p50": 0.007691574401794991, "p90": 0.011201243427351017, "p99": 0.013311375577245894}, "wall": 33770.57413210906, "tps": 0.9869539045920406}, {"tag": "fig4l_lmetric_pd4_first600s", "arm": "4P+4D", "trace": "first600s", "n": 348, "req": 807, "e2e": {"count": 348.0, "mean": 202.63302869595395, "p50": 214.03008900902933, "p90": 477.40967412578175, "p99": 576.6393926549597}, "ttft": {"count": 348.0, "mean": 199.96385804087797, "p50": 213.50966987549327, "p90": 475.7766476540827, "p99": 559.6153268160638}, "tpot": {"count": 348.0, "mean": 0.008873619369764751, "p50": 0.007645836479973812, "p90": 0.013845969236959285, "p99": 0.02567216653158788}, "wall": 6850.181333696004, "tps": 15.00296050477674}, {"tag": "fig4l_lmetric_pd4_full", "arm": "4P+4D", "trace": "full", "n": 533, "req": 1214, "e2e": {"count": 533.0, "mean": 130.94711188977982, "p50": 8.219856544979848, "p90": 473.44134307731883, "p99": 533.2597587251009}, "ttft": {"count": 533.0, "mean": 127.83193208824007, "p50": 4.8246813879814, "p90": 467.54664219671395, "p99": 528.8304683346115}, "tpot": {"count": 533.0, "mean": 0.008886429490232585, "p50": 0.007981476340708988, "p90": 0.013570741891233497, "p99": 0.023050950961825044}, "wall": 13884.384965199977, "tps": 12.621372890425038}, {"tag": "fig4l_lmetric_pd6_first600s", "arm": "6P+2D", "trace": "first600s", "n": 474, "req": 807, "e2e": {"count": 474.0, "mean": 83.15809065495806, "p50": 6.7270191764691845, "p90": 391.6558471220078, "p99": 544.7372293809171}, "ttft": {"count": 474.0, "mean": 80.70155321074382, "p50": 4.1273433425230905, "p90": 390.00296151017517, "p99": 539.0574236416071}, "tpot": {"count": 474.0, "mean": 0.008519881756330928, "p50": 0.00803907146806204, "p90": 0.012583933303093976, "p99": 0.018606097790947705}, "wall": 3325.2749515309697, "tps": 39.705588838364164}, {"tag": "fig4l_lmetric_pd6_full", "arm": "6P+2D", "trace": "full", "n": 793, "req": 1214, "e2e": {"count": 793.0, "mean": 61.907526705667, "p50": 3.69814173609484, "p90": 308.2633092067672, "p99": 477.48038318102715}, "ttft": {"count": 793.0, "mean": 59.25069201986225, "p50": 1.402295546955429, "p90": 302.5604081378088, "p99": 475.3738951798529}, "tpot": {"count": 793.0, "mean": 0.009137289999448822, "p50": 0.008635683270933276, "p90": 0.013065757584108427, "p99": 0.01816783740464599}, "wall": 5662.029295974993, "tps": 39.24494000021532}]

analysis/v2/fig4r_linear.json

@@ -0,0 +1 @@
+[{"tag": "fig4r_linear_colo_first600s", "arm": "colo", "trace": "first600s", "policy": "linear", "n": 807, "req": 807, "dispatched": 807, "e2e": {"count": 807.0, "mean": 8.436370009274967, "p50": 2.5224755640374497, "p90": 22.65510415879542, "p99": 75.54369598095519}, "ttft": {"count": 807.0, "mean": 4.2332503390957195, "p50": 0.8872958200518042, "p90": 11.684667797433207, "p99": 44.98891795879462}, "tpot": {"count": 807.0, "mean": 0.020958194728517718, "p50": 0.00851320761584622, "p90": 0.026440129078245465, "p99": 0.30344440533287176}, "wall": 963.6191155100241, "tps": 239.4857016486815, "capped": false}, {"tag": "fig4r_linear_pd2_first600s", "arm": "2P+6D", "trace": "first600s", "policy": "linear", "n": 176, "req": 807, "dispatched": 521, "e2e": {"count": 176, "mean": 378.5561210460834, "p50": 536.7719694490079, "p90": 583.832092280034, "p99": 601.3415494390065}, "ttft": {"count": 176, "mean": 377.12570991374446, "p50": 536.1157373189926, "p90": 580.3465002350276, "p99": 598.0943597999867}, "tpot": {"count": 176, "mean": 0.007864906140929698, "p50": 0.007212154543958604, "p90": 0.011962352272927423, "p99": 0.017870794738764347}, "wall": 2280, "tps": 14.419736842105262, "capped": true}, {"tag": "fig4r_linear_pd4_first600s", "arm": "4P+4D", "trace": "first600s", "policy": "linear", "n": 349, "req": 807, "dispatched": 577, "e2e": {"count": 349, "mean": 264.8537863784421, "p50": 306.6853819829412, "p90": 488.64622142596636, "p99": 596.5830293919425}, "ttft": {"count": 349, "mean": 262.3163347712099, "p50": 299.75751709297765, "p90": 485.475125996978, "p99": 596.4081599479541}, "tpot": {"count": 349, "mean": 0.010442244895290958, "p50": 0.008213572105774598, "p90": 0.019443845545703716, "p99": 0.028178529054794}, "wall": 2281, "tps": 38.306882946076286, "capped": true}, {"tag": "fig4r_linear_pd6_first600s", "arm": "6P+2D", "trace": "first600s", "policy": "linear", "n": 472, "req": 807, "dispatched": 706, "e2e": {"count": 472, "mean": 118.632779156234, "p50": 12.702161715948023, "p90": 458.1609142010566, "p99": 526.5488834320568}, "ttft": {"count": 472, "mean": 115.80202843308507, "p50": 9.745031949947588, "p90": 455.81679951993283, "p99": 516.5850186559837}, "tpot": {"count": 472, "mean": 0.00950947083585719, "p50": 0.008435572332624966, "p90": 0.015233499645638644, "p99": 0.023447183093280886}, "wall": 2280, "tps": 61.69210526315789, "capped": true}]

analysis/working_set/README.md

@@ -0,0 +1,67 @@
+# KV-cache Working-Set Sizing — GLM-5.1-FP8 · TP=8 · 1× B300 node
+
+工具：`scripts/working_set_analysis.py`（可配置 GPU 型号 / 并行度 TP·PP·EP / 模型 config.json /
+KV dtype / 权重大小）。图：`figs/working_set/glm5_fp8_tp8_b300.png`。
+
+## 复现
+
+```bash
+.venv/bin/python scripts/working_set_analysis.py \
+  /home/gahow/phd/kvcache-simulator/bailian-traces/glm_coder_blksz_512_040915-040917.jsonl \
+  --model-config /home/gahow/phd/kvcache-simulator/models/GLM-5/config.json \
+  --gpu B300 --tp 8 --ep 8 --kv-dtype-bytes 1 --weight-gb 744 --min-ts 0 \
+  --out figs/working_set/glm5_fp8_tp8_b300.png
+```
+
+## 方法
+
+`hash_ids` 是全局内容寻址 block id（同内容=同 id，复用=同 id 再现）。vLLM prefix cache 是
+block 级，所以**集群级 KV footprint = 任一时刻必须常驻的 distinct block 数**，与 placement 无关
+（affinity 只搬运 block，不改总量）。三种 working set：
+- `W_all` 永不淘汰（真上界）
+- `W_oracle` 每 block 只在 `[首次, 末次复用]` 常驻（Belady 完美预知 → 满 APC 上界的最小 HBM）
+- `W_denning(T)` 滑窗 T 内被访问的 distinct block（现实 TTL-LRU）
+
+KV/token：MLA → `L·(kv_lora_rank+qk_rope_head_dim)·dtype`；GQA → `2·L·kv_heads·head_dim·dtype`
+（与 `kvcache-simulator/src/config.rs::kv_block_bytes` 一致）。
+
+## 配置
+
+| 项 | 值 |
+|---|---|
+| 模型 | GLM-5.1-FP8（MLA, L=78, kv_lora=512+rope=64） |
+| KV/token · KV/block(512) | **43.9 KiB** · **23.0 MB**（≈ Qwen3 GQA 96 KiB 的一半） |
+| 硬件 | 8× B300 (288 GB) = 2304 GB HBM/replica |
+| 预算 | FP8 权重 744 GB + act 32 GB → **KV pool = 1528 GB/node** |
+| trace | dash0 glm_coder，475k req，**1.25h active @ 106 QPS**，~40k tok/req（剔除 77 条负 ts 暖机） |
+| APC 上界 | **80.4%** |
+
+## 结果
+
+| 保留窗口 T | peak footprint | = 节点 (GPU) | APC@T |
+|---:|---:|---:|---:|
+| 2s（在飞下限）| 533 GB | 0.3 (3) | 1.7% |
+| 10s | 2,068 GB | 1.4 (11) | 15% |
+| 30s | 4,906 GB | 3.2 (26) | 42% |
+| 60s | 7,698 GB | 5.0 (40) | 56% |
+| 300s | 21,960 GB | 14.4 (115) | 74% |
+| **oracle（满 80.4%）** | **21,399 GB** | **14.0 (112)** | 80.4% |
+| retain-forever | 167,018 GB | 109 (874) | — |
+
+## 结论
+
+1. **Serving：1 节点绰绰有余。** 在飞 KV（τ≈2-5s）仅 533–1157 GB ≪ 单节点 1528 GB。
+   MLA + B300 大 HBM 让 live footprint 微不足道——跑起来根本不缺显存。
+2. **缓存全部复用（80.4%）：1 节点差 ~14×。** oracle 下限 21.4 TB = 14 节点（112 GPU），
+   真实 LRU ~2× → ~28 节点。单节点（1528 GB）只能 hold ~10s 窗口 → cache 侧 APC 仅 ~10-15%。
+   要 ~56% 需 5 节点，~74% 需 ~14 节点。
+3. **瓶颈在长尾，不在 live。** 把 APC 50%→80% 装进 GPU HBM 要 5→14 节点，极不经济
+   → offload/migration 到 CPU DRAM（每节点 ~1.5 TB）是定量动机。与 Qwen 结论方向一致。
+
+## 注意
+
+- footprint 是 TTL-LRU（最浪费）+ shared-cache 下限：真实 capacity-LRU 同容量下 APC 更高，
+  但分区/affinity 不均衡又抬高需求；oracle / retain-forever 给出下/上界。
+- GLM trace mean ~40k tok/req，是 Qwen trace（11k）的 ~3.5×（tokenizer + 抽取不同），
+  **绝对 GB 不可跨模型横比**，方法与定性结论可比。
+- EP 不改变 KV 总量（只影响 expert 权重分布），`--ep` 仅作标注。

analysis/workload_chars/README.md

@@ -0,0 +1,81 @@
+# Agentic workload characterization C1–C3 (full 051315 production trace)
+
+Date 2026-05-29. Source: `trace-glm5.1-formatted/051315-051317.jsonl` on dash1
+(release file, 2,114,220 requests / 1,307,276 sessions / 2h, type=100% `coder`).
+This release file **is the full cluster-level production trace** — session skew
+reproduces 46.5/66.5/74.6/87.5/96.0 exactly. Compute: `compute_chars.py`
+(2-pass, ~65s, `~/ali-trace/.venv` python). Numbers: `chars.json`.
+
+> ⚠️ **Cluster-level, not per-instance.** This is one cluster's aggregate stream.
+> Concurrent-session counts have NO denominator of "8 instances" — do not compare
+> them to a single deployment's instance count.
+
+These three are NOT in the existing 13 analyzer figures (which are single-variable
+marginals on the older 041x traces). C1–C3 are joint/temporal and argument-bearing.
+
+## C1 — the workload is a MIXTURE, not "multi-turn agentic" (`c1_session_mixture.png`)
+
+- **90.3%** of sessions are single-turn; mean 1.62 turns, p99=18, max=3091.
+- But multi-turn sessions (9.7%) = **44.2% of requests** and **66.9% of input
+  (prefill) mass**. Single-turn = **60.2% of output (decode) mass**.
+- Continuation hazard P(reach k+1 | reached k): turn1→2 only **10.2%**, but
+  turn2→3 50.6%, turn5→6 87%, turn12→13 **94.3%** (Lindy / Pareto).
+- Predictability of heaviness at cold-start is near-zero:
+  corr(turn1_input, session_mass)=0.15, corr(turn1_input, n_turns)=**0.04**.
+
+**Routing:** heaviness is unpredictable at session start → proactive placement
+cannot pre-empt hot-pin → a REACTIVE mechanism (observable-load routing /
+migration) is required. But once a session has shown depth, it almost surely
+continues → "observed accumulated load" is the signal that works (not turn-1
+features, not cost-model prediction). The single/multi optimal strategies are
+opposite (load-balance the 90% one-shot sea vs affinity-pin the deep tail) and
+you can't tell them apart at turn 1 → the only viable policy starts everyone
+load-balanced and becomes sticky as turns accrue. This is exactly LPWL's
+emergent behavior (`new_uncached≈input`→by-load; `new_uncached≈0`→sticks), so
+C1 explains *why* a cache-aware-load score is the right shape — it auto-segments
+the mixture with no classifier.
+
+## C2 — marginal work collapses while resident state explodes (`c2_work_amortization.png`)
+
+Per turn: resident context grows 11k→56k+ tokens while new prefill collapses
+2.7k→~200 tokens; per-turn reuse climbs 83%→**99.6%**; resident/new ratio
+("the PD tax") grows to ~250× by turn 12, ~450× by turn 30.
+
+**PD-colocation:** the dominant cost is keeping ~50k+ resident KV available for
+the next turn's tiny delta. Disaggregation physically splits a turn's prefill-KV
+(P) and decode-KV (D), and the next turn's prefix = [prevPrompt + prevAnswer]
+spans both → must be gathered/transferred; colocation keeps it local for free.
+**Routing:** route on delta (`input − cache_hit`), never total input — C2 is the
+trace-level justification for LPWL's score function.
+
+## C3 — prefill/decode BALANCE (honest reframe) (`c3_prefill_decode_balance.png`)
+
+- Token mass: 98.7% input / **1.3% output**; of input, 60% reused-prefix, 40%
+  new-prefill (28.6B new-prefill tokens vs 0.94B decode tokens).
+- **But tokens ≠ time.** Under a per-request latency model (prefill@7k tok/s,
+  TPOT 10ms), aggregate decode-time share ≈ **70% (robust 68–71% across
+  constants)** — each decode token costs ~70–140× a prefill token. So this is
+  NOT a "decode is negligible" workload.
+- Per-request the bottleneck FLIPS within a session: turn-1 (and the 90%
+  single-turn) is prefill-bound; turns ≥3 are strongly decode-bound.
+
+**PD-colocation (correct argument):** the workload has *substantial* work on both
+sides of the roofline — compute-bound prefill (~30% of time) and memory-bound
+decode (~70%). Colocation interleaves them on one GPU (chunked prefill +
+continuous batching) so compute and HBM bandwidth are both used; static
+disaggregation strands P-instances bandwidth-idle and D-instances compute-idle.
+The earlier "decode is 1.3% so nothing to isolate" instinct was WRONG (token vs
+time confusion) — C3b is the correction.
+
+**Caveat:** C3b's 70% is a per-request-latency-weighted estimate; batched decode
+throughput will shift it. Ground-truth needs `-raw.jsonl` (`usage.cached_tokens`
+for exact reuse; `backend_first_response_time_ms` / `total_cost_time_ms` for real
+prefill vs decode wall time). Sampling that 522GB file is the next step.
+
+## Goal mapping
+
+| | argue PD-colocation | guide routing |
+|---|---|---|
+| C1 mixture + hazard | both segments favor colo (diff reasons) | reactive + auto-segment ⇒ LPWL shape |
+| C2 resident/delta | the PD tax (transfer/split resident KV) | route on delta, not total |
+| C3 prefill/decode | roofline complementarity (interleave) | per-req bottleneck flips within session |

analysis/workload_chars/chars.json

@@ -0,0 +1,964 @@
+{
+  "mixture": {
+    "single_sessions": 1179990,
+    "multi_sessions": 127286,
+    "req_single_pct": 55.81207253738968,
+    "req_multi_pct": 44.187927462610325,
+    "in_single_pct": 33.12487590117447,
+    "in_multi_pct": 66.87512409882554,
+    "out_single_pct": 60.24502960903973,
+    "out_multi_pct": 39.75497039096027
+  },
+  "turns": {
+    "mean": 1.6172713336739908,
+    "p99": 18.0,
+    "max": 3091,
+    "single_turn_pct": 90.26326498765371
+  },
+  "hazard": {
+    "1": 0.102101621998721,
+    "2": 0.5062146469376287,
+    "3": 0.7351961756478754,
+    "4": 0.8113739305485657,
+    "5": 0.8723731546954472,
+    "6": 0.8669264241631353,
+    "7": 0.9093235352011023,
+    "8": 0.9240204920989971,
+    "9": 0.901725753553022,
+    "10": 0.9346178826585841,
+    "11": 0.9260597637248089,
+    "12": 0.9427685226874781,
+    "13": 0.91950119395065,
+    "14": 0.936865189289012,
+    "15": 0.9382160896883085,
+    "16": 0.9308646838684262,
+    "17": 0.9371561574269995,
+    "18": 0.9312862196131557,
+    "19": 0.9333279456925813,
+    "20": 0.9351459000779289,
+    "21": 0.9399074074074074,
+    "22": 0.9404984730568416,
+    "23": 0.9473132921336546,
+    "24": 0.9193940734188413,
+    "25": 0.9497294046903187,
+    "26": 0.9323793845764214,
+    "27": 0.9483906016569333,
+    "28": 0.9368466275239868,
+    "29": 0.9472638336900031
+  },
+  "token_mass": {
+    "total_input": 71116829368,
+    "total_output": 940765734,
+    "out_in_ratio_pct": 1.3228454394837104,
+    "new_prefill": 28616906067,
+    "reused_prefix": 42499923301,
+    "new_prefill_pct_of_input": 40.23928839532401
+  },
+  "decode_time_fraction": {
+    "optimistic_for_prefill": 0.6812079219496285,
+    "mid": 0.6970810590484581,
+    "pessimistic": 0.711448473592609
+  },
+  "per_turn": {
+    "turn": [
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11,
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19,
+      20,
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29,
+      30,
+      31,
+      32,
+      33,
+      34,
+      35,
+      36,
+      37,
+      38,
+      39,
+      40,
+      41,
+      42,
+      43,
+      44,
+      45,
+      46,
+      47,
+      48,
+      49,
+      50,
+      51,
+      52,
+      53,
+      54,
+      55,
+      56,
+      57,
+      58,
+      59,
+      60,
+      61,
+      62,
+      63,
+      64,
+      65,
+      66,
+      67,
+      68,
+      69,
+      70,
+      71,
+      72,
+      73,
+      74,
+      75,
+      76,
+      77,
+      78,
+      79,
+      80,
+      81,
+      82,
+      83,
+      84,
+      85,
+      86,
+      87,
+      88,
+      89,
+      90,
+      91,
+      92,
+      93,
+      94,
+      95,
+      96,
+      97,
+      98,
+      99,
+      100,
+      101,
+      102,
+      103,
+      104,
+      105,
+      106,
+      107,
+      108,
+      109,
+      110,
+      111,
+      112,
+      113,
+      114,
+      115,
+      116,
+      117,
+      118,
+      119,
+      120,
+      121,
+      122,
+      123,
+      124,
+      125,
+      126,
+      127,
+      128,
+      129,
+      130,
+      131,
+      132,
+      133,
+      134,
+      135,
+      136,
+      137,
+      138,
+      139,
+      140,
+      141,
+      142,
+      143,
+      144,
+      145,
+      146,
+      147,
+      148
+    ],
+    "med_resident_input": [
+      11035.0,
+      19505.0,
+      28059.0,
+      35089.0,
+      41215.0,
+      44750.0,
+      47419.5,
+      49874.0,
+      51905.0,
+      53068.0,
+      54782.0,
+      56414.0,
+      58229.0,
+      59123.5,
+      60434.5,
+      61320.0,
+      62243.0,
+      63411.0,
+      64510.5,
+      65423.0,
+      66942.5,
+      67965.0,
+      68826.0,
+      70165.5,
+      70052.0,
+      70936.0,
+      71547.0,
+      72648.0,
+      73406.0,
+      73844.0,
+      73604.0,
+      74937.5,
+      74778.0,
+      75460.0,
+      75029.0,
+      74978.0,
+      75933.0,
+      76590.0,
+      74695.0,
+      76813.0,
+      77079.5,
+      78310.0,
+      77848.0,
+      77549.0,
+      78203.0,
+      79102.0,
+      79202.0,
+      78821.0,
+      79868.0,
+      80229.5,
+      80912.0,
+      81620.0,
+      81612.5,
+      81836.5,
+      82506.0,
+      82948.0,
+      82633.0,
+      84107.5,
+      84176.0,
+      84441.0,
+      84101.0,
+      85192.0,
+      84127.0,
+      84783.5,
+      85087.0,
+      85771.5,
+      86110.0,
+      85374.5,
+      87137.0,
+      87677.0,
+      88587.0,
+      88656.0,
+      88882.0,
+      89284.0,
+      91512.0,
+      89850.0,
+      90596.0,
+      91244.0,
+      92102.0,
+      93431.0,
+      92333.5,
+      96682.0,
+      94999.0,
+      95226.5,
+      95173.0,
+      95910.0,
+      96528.0,
+      96508.0,
+      97270.0,
+      97301.0,
+      97076.5,
+      97105.0,
+      98032.0,
+      97962.5,
+      97968.5,
+      98310.0,
+      97061.0,
+      97631.0,
+      100126.0,
+      97765.0,
+      101076.0,
+      98198.5,
+      98678.0,
+      98307.0,
+      99174.0,
+      99882.0,
+      99974.0,
+      99757.0,
+      100065.5,
+      99943.0,
+      100612.0,
+      101138.0,
+      106738.0,
+      99621.0,
+      101980.0,
+      102252.0,
+      103018.0,
+      101238.0,
+      102005.0,
+      101897.0,
+      103576.0,
+      102159.5,
+      102695.5,
+      100590.5,
+      103236.0,
+      101812.0,
+      103074.0,
+      99966.0,
+      102183.5,
+      101882.0,
+      102572.5,
+      105622.5,
+      106066.0,
+      103974.0,
+      105443.5,
+      104716.0,
+      105041.0,
+      106628.0,
+      108320.0,
+      108022.5,
+      107621.5,
+      107664.0,
+      107913.0,
+      108630.0,
+      108382.0,
+      107216.5,
+      105731.0,
+      103986.0
+    ],
+    "med_new_prefill": [
+      11035.0,
+      2920.0,
+      1249.0,
+      767.0,
+      628.0,
+      485.0,
+      400.0,
+      359.0,
+      314.0,
+      274.0,
+      263.0,
+      258.0,
+      244.0,
+      231.0,
+      227.0,
+      222.0,
+      201.0,
+      200.0,
+      198.0,
+      189.0,
+      182.5,
+      184.0,
+      179.0,
+      188.0,
+      173.0,
+      180.0,
+      164.0,
+      167.0,
+      159.5,
+      168.0,
+      156.0,
+      174.0,
+      156.0,
+      159.0,
+      166.0,
+      165.0,
+      153.0,
+      158.0,
+      182.0,
+      149.0,
+      184.0,
+      172.0,
+      149.0,
+      167.0,
+      163.0,
+      152.0,
+      153.0,
+      171.0,
+      151.0,
+      146.0,
+      162.0,
+      153.0,
+      156.0,
+      164.0,
+      148.0,
+      143.0,
+      143.0,
+      149.0,
+      170.5,
+      159.0,
+      144.0,
+      168.0,
+      148.0,
+      144.5,
+      142.5,
+      146.5,
+      147.0,
+      157.0,
+      168.0,
+      153.0,
+      155.0,
+      127.5,
+      145.0,
+      143.0,
+      146.0,
+      123.0,
+      139.0,
+      137.0,
+      115.0,
+      139.5,
+      117.0,
+      154.0,
+      111.0,
+      124.0,
+      118.0,
+      90.0,
+      104.0,
+      116.0,
+      112.0,
+      76.5,
+      110.0,
+      101.0,
+      123.0,
+      114.0,
+      86.0,
+      92.0,
+      108.0,
+      85.0,
+      146.0,
+      77.5,
+      101.0,
+      102.0,
+      85.0,
+      77.0,
+      114.0,
+      66.0,
+      105.0,
+      90.0,
+      89.0,
+      100.0,
+      108.5,
+      100.0,
+      169.0,
+      89.0,
+      106.5,
+      78.0,
+      75.0,
+      90.0,
+      77.0,
+      88.0,
+      102.0,
+      83.5,
+      123.5,
+      116.5,
+      108.0,
+      119.0,
+      82.0,
+      80.0,
+      105.0,
+      90.0,
+      91.0,
+      113.0,
+      122.0,
+      102.0,
+      101.5,
+      64.0,
+      78.0,
+      52.5,
+      98.5,
+      72.0,
+      87.0,
+      102.0,
+      97.0,
+      123.0,
+      80.0,
+      132.5,
+      86.5,
+      111.0
+    ],
+    "med_output": [
+      63.0,
+      67.0,
+      111.0,
+      142.0,
+      158.0,
+      162.0,
+      164.0,
+      164.0,
+      159.0,
+      160.0,
+      159.0,
+      161.0,
+      160.0,
+      158.0,
+      154.0,
+      154.0,
+      154.0,
+      149.0,
+      146.0,
+      147.0,
+      142.0,
+      144.0,
+      143.0,
+      142.0,
+      140.0,
+      136.0,
+      137.0,
+      139.0,
+      136.0,
+      133.0,
+      130.0,
+      131.0,
+      125.0,
+      123.0,
+      122.0,
+      122.0,
+      118.0,
+      122.0,
+      114.0,
+      112.0,
+      115.0,
+      111.0,
+      109.0,
+      112.0,
+      109.0,
+      107.0,
+      111.0,
+      105.0,
+      108.0,
+      107.0,
+      100.0,
+      100.0,
+      95.0,
+      105.0,
+      103.0,
+      102.0,
+      100.0,
+      100.0,
+      98.0,
+      98.0,
+      101.0,
+      99.0,
+      101.0,
+      102.0,
+      97.0,
+      91.0,
+      100.0,
+      97.0,
+      94.0,
+      98.5,
+      92.5,
+      97.0,
+      102.0,
+      92.0,
+      95.0,
+      91.0,
+      91.0,
+      92.0,
+      85.0,
+      98.0,
+      96.0,
+      99.0,
+      94.0,
+      96.0,
+      90.0,
+      85.0,
+      99.0,
+      86.0,
+      99.0,
+      93.0,
+      92.0,
+      93.0,
+      87.0,
+      83.0,
+      87.5,
+      82.0,
+      80.0,
+      90.0,
+      92.0,
+      80.0,
+      77.0,
+      82.0,
+      87.0,
+      74.0,
+      83.0,
+      79.0,
+      84.0,
+      80.5,
+      79.0,
+      76.0,
+      78.5,
+      71.5,
+      81.0,
+      87.0,
+      82.0,
+      85.0,
+      87.0,
+      75.0,
+      75.0,
+      82.0,
+      86.0,
+      76.5,
+      77.5,
+      70.0,
+      78.0,
+      85.0,
+      77.0,
+      67.0,
+      76.5,
+      107.0,
+      92.0,
+      80.5,
+      85.0,
+      83.0,
+      77.0,
+      70.0,
+      84.0,
+      69.0,
+      97.0,
+      72.0,
+      81.0,
+      87.0,
+      89.0,
+      102.0,
+      83.0,
+      82.5,
+      91.0,
+      79.5
+    ],
+    "resident_over_new": [
+      1.0,
+      6.679794520547945,
+      22.46517213771017,
+      45.748370273794,
+      65.62898089171975,
+      92.26804123711341,
+      118.54875,
+      138.92479108635098,
+      165.30254777070064,
+      193.67883211678833,
+      208.29657794676805,
+      218.65891472868216,
+      238.64344262295083,
+      255.94588744588745,
+      266.23127753303964,
+      276.2162162162162,
+      309.6666666666667,
+      317.055,
+      325.81060606060606,
+      346.15343915343914,
+      366.8082191780822,
+      369.375,
+      384.5027932960894,
+      373.22074468085106,
+      404.9248554913295,
+      394.0888888888889,
+      436.2621951219512,
+      435.0179640718563,
+      460.2257053291536,
+      439.54761904761904,
+      471.8205128205128,
+      430.67528735632186,
+      479.34615384615387,
+      474.59119496855345,
+      451.98192771084337,
+      454.41212121212124,
+      496.29411764705884,
+      484.746835443038,
+      410.4120879120879,
+      515.5234899328859,
+      418.9103260869565,
+      455.2906976744186,
+      522.4697986577181,
+      464.36526946107784,
+      479.7730061349693,
+      520.4078947368421,
+      517.6601307189543,
+      460.94152046783626,
+      528.9271523178808,
+      549.5171232876712,
+      499.4567901234568,
+      533.4640522875817,
+      523.1570512820513,
+      499.0030487804878,
+      557.472972972973,
+      580.0559440559441,
+      577.8531468531469,
+      564.4798657718121,
+      493.7008797653959,
+      531.0754716981132,
+      584.0347222222222,
+      507.0952380952381,
+      568.4256756756756,
+      586.7370242214533,
+      597.1017543859649,
+      585.4709897610921,
+      585.7823129251701,
+      543.7866242038217,
+      518.672619047619,
+      573.0522875816994,
+      571.5290322580645,
+      695.3411764705883,
+      612.9793103448276,
+      624.3636363636364,
+      626.7945205479452,
+      730.4878048780488,
+      651.7697841726618,
+      666.014598540146,
+      800.8869565217391,
+      669.7562724014336,
+      789.1752136752136,
+      627.8051948051948,
+      855.8468468468468,
+      767.9556451612904,
+      806.5508474576271,
+      1065.6666666666667,
+      928.1538461538462,
+      831.9655172413793,
+      868.4821428571429,
+      1271.9084967320262,
+      882.5136363636364,
+      961.4356435643564,
+      797.0081300813008,
+      859.3201754385965,
+      1139.1686046511627,
+      1068.5869565217392,
+      898.7129629629629,
+      1148.6,
+      685.7945205479452,
+      1261.483870967742,
+      1000.7524752475248,
+      962.7303921568628,
+      1160.9176470588236,
+      1276.7142857142858,
+      869.9473684210526,
+      1513.3636363636363,
+      952.1333333333333,
+      1108.411111111111,
+      1124.3314606741574,
+      999.43,
+      927.2995391705069,
+      1011.38,
+      631.5857988165681,
+      1119.3370786516855,
+      957.5586854460093,
+      1310.923076923077,
+      1373.5733333333333,
+      1124.8666666666666,
+      1324.7402597402597,
+      1157.9204545454545,
+      1015.4509803921569,
+      1223.4670658682635,
+      831.5425101214574,
+      863.4377682403433,
+      955.8888888888889,
+      855.563025210084,
+      1257.0,
+      1249.575,
+      973.1761904761905,
+      1132.0222222222221,
+      1127.1703296703297,
+      934.712389380531,
+      869.3934426229508,
+      1019.3529411764706,
+      1038.8522167487686,
+      1636.1875,
+      1346.679487179487,
+      2031.009523809524,
+      1099.6954314720813,
+      1500.3125,
+      1237.028735632184,
+      1055.5294117647059,
+      1112.5051546391753,
+      883.170731707317,
+      1354.775,
+      809.1811320754717,
+      1222.3236994219653,
+      936.8108108108108
+    ],
+    "reuse_pct": [
+      0.0,
+      85.02947962061009,
+      95.5486653123775,
+      97.81412978426287,
+      98.47628290670872,
+      98.91620111731844,
+      99.1564651672835,
+      99.28018606889361,
+      99.39504864656584,
+      99.48368131453984,
+      99.5199153006462,
+      99.54266671393626,
+      99.5809648113483,
+      99.60929241333818,
+      99.62438673274372,
+      99.63796477495107,
+      99.67707212055974,
+      99.68459730961506,
+      99.69307322063851,
+      99.71111077144124,
+      99.72737797363409,
+      99.72927241962775,
+      99.73992386598088,
+      99.73206205328829,
+      99.75304059841261,
+      99.74625014097215,
+      99.7707800466826,
+      99.77012443563484,
+      99.78271530937526,
+      99.77249336438979,
+      99.78805499701103,
+      99.76780650542119,
+      99.79138249217684,
+      99.78929234031276,
+      99.77875221580989,
+      99.77993544773133,
+      99.7985065781676,
+      99.7937067502285,
+      99.75634245933462,
+      99.80602241808027,
+      99.76128542608606,
+      99.78036010726599,
+      99.80860137704244,
+      99.78465228436214,
+      99.79156809841055,
+      99.8078430381027,
+      99.80682306002375,
+      99.7830527397521,
+      99.81093804777883,
+      99.81802204924622,
+      99.79978247973106,
+      99.8125459446214,
+      99.8088528105376,
+      99.79960042279423,
+      99.82061910648923,
+      99.82760283551141,
+      99.82694565125313,
+      99.822845762863,
+      99.79744820376354,
+      99.81170284577398,
+      99.82877730348034,
+      99.80279838482487,
+      99.82407550489143,
+      99.82956589430727,
+      99.8325243574224,
+      99.82919734410615,
+      99.82928811984671,
+      99.81610434028896,
+      99.80720015607606,
+      99.82549585410084,
+      99.8250307607211,
+      99.85618570655116,
+      99.83686235683265,
+      99.83983692486895,
+      99.84045808200017,
+      99.86310517529216,
+      99.84657159256479,
+      99.84985314102846,
+      99.87513843347593,
+      99.85069195449047,
+      99.87328542728262,
+      99.84071492108149,
+      99.88315666480699,
+      99.8697841462198,
+      99.87601525642776,
+      99.90616202690022,
+      99.89225924084204,
+      99.87980271065611,
+      99.88485658476407,
+      99.9213779920042,
+      99.88668730331234,
+      99.89598887801864,
+      99.87453076546434,
+      99.88362893964528,
+      99.91221668189266,
+      99.90641847217984,
+      99.88872976787793,
+      99.91293748911718,
+      99.8541837285021,
+      99.92072827699074,
+      99.90007519094543,
+      99.89612875960428,
+      99.91386124566772,
+      99.92167393980083,
+      99.88505051727267,
+      99.93392202799302,
+      99.89497269290015,
+      99.90978076726445,
+      99.91105825684177,
+      99.89994296749147,
+      99.89215998091679,
+      99.90112519527774,
+      99.84166838426802,
+      99.91066140673152,
+      99.89556775838399,
+      99.92371787348903,
+      99.9271971888408,
+      99.91110057488295,
+      99.92451350423998,
+      99.91363828179436,
+      99.90152158801267,
+      99.91826506590185,
+      99.87974156608614,
+      99.8841838941053,
+      99.89538533069859,
+      99.883117903587,
+      99.92044550517105,
+      99.91997279074886,
+      99.89724368415645,
+      99.91166251153295,
+      99.91128226376466,
+      99.89301521929514,
+      99.88497727829841,
+      99.90189855156096,
+      99.9037399175862,
+      99.93888231024867,
+      99.92574328119497,
+      99.95076340173313,
+      99.90906573116692,
+      99.9333472193293,
+      99.91916113415999,
+      99.90526081141329,
+      99.91011277603255,
+      99.88677161005248,
+      99.92618700522226,
+      99.8764182751722,
+      99.91818861071965,
+      99.89325486123131
+    ]
+  }
+}

analysis/workload_chars/compute_chars.py

@@ -0,0 +1,180 @@
+import json, sys, math, statistics as st
+from collections import defaultdict, Counter
+import matplotlib; matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+
+PATH="/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl"
+OUT="/tmp/wlc_out"; import os; os.makedirs(OUT, exist_ok=True)
+BLOCK=512
+# --- transparent cost model for C3 (clearly-labeled estimate; raw-timing validation pending) ---
+PREFILL_TOK_S=7000.0     # MB1: 32k->4.5s ~7100 tok/s effective on H20 / 30B-A3B
+TPOT_S=0.010             # ~10ms/token decode (crossover unloaded ~5ms, loaded ~25ms)
+
+def pct(v,p):
+    if not v: return float('nan')
+    s=sorted(v);k=(len(s)-1)*p;f=int(k)
+    return s[f] if f+1>=len(s) else s[f]+(s[f+1]-s[f])*(k-f)
+
+# ---------- Pass A: structure (scalars only) ----------
+parents={}; recs={}; childcount=Counter()
+for line in open(PATH):
+    if not line.strip(): continue
+    d=json.loads(line); cid=d["chat_id"]; pid=d["parent_chat_id"]
+    parents[cid]=pid
+    recs[cid]=(float(d["timestamp"]),int(d["input_length"]),int(d["output_length"]),int(d["turn"]))
+    if pid!="-1": childcount[pid]+=1
+print(f"[A] records={len(recs)}", file=sys.stderr)
+
+root_of={}
+def root(cid):
+    path=[];c=cid
+    while True:
+        if c in root_of:r=root_of[c];break
+        p=parents.get(c,"-1")
+        if p=="-1" or p not in recs:r=c;break
+        path.append(c);c=p
+    for x in path:root_of[x]=r
+    root_of[cid]=r;return r
+sessions=defaultdict(list)
+for cid in recs: sessions[root(cid)].append(cid)
+seq={r:sorted(m,key=lambda c:(recs[c][3],recs[c][0])) for r,m in sessions.items()}
+print(f"[A] sessions={len(seq)}", file=sys.stderr)
+
+# ---------- C1: mixture + turn tail + hazard ----------
+sr=mr=sm=mm=so=mo=0
+turns_per=[]
+for r,s in seq.items():
+    multi=len(s)>1; turns_per.append(len(s))
+    for c in s:
+        _,inl,outl,_=recs[c]
+        if multi: mr+=1;mm+=inl;mo+=outl
+        else: sr+=1;sm+=inl;so+=outl
+tot_r=sr+mr; tot_in=sm+mm; tot_out=so+mo
+cnt_turn=Counter()
+for r,s in seq.items():
+    for c in s: cnt_turn[recs[c][3]]+=1
+hazard={k: (cnt_turn[k+1]/cnt_turn[k] if cnt_turn[k] else 0) for k in range(1,30)}
+
+# ---------- C2/C3: per-turn resident vs new-prefill (scalar) + hash_ids reuse ----------
+by_in=defaultdict(list); by_new=defaultdict(list); by_out=defaultdict(list)
+by_reuse_hash=defaultdict(list)  # hash-block prefix stability: reused/parent_blocks
+store={}  # cid -> (blockset, in, out) for chats with pending children
+tot_new_prefill=0; tot_reused=0
+for line in open(PATH):
+    if not line.strip(): continue
+    d=json.loads(line); cid=d["chat_id"]; pid=d["parent_chat_id"]
+    inl=int(d["input_length"]); outl=int(d["output_length"]); turn=int(d["turn"])
+    blocks=set(d["hash_ids"])
+    if pid in store:
+        pblk,pin,pout=store[pid]
+        new_prefill=max(0, inl - pin - pout)              # actual recompute (accounts for cached answer)
+        reused_blk=len(blocks & pblk)
+        by_reuse_hash[turn].append(reused_blk/len(pblk) if pblk else 0)
+        childcount[pid]-=1
+        if childcount[pid]<=0: del store[pid]
+        tot_reused += (inl-new_prefill)
+    else:
+        new_prefill=inl                                    # session start: all new (intra-session)
+    tot_new_prefill+=new_prefill
+    by_in[turn].append(inl); by_new[turn].append(new_prefill); by_out[turn].append(outl)
+    if childcount[cid]>0: store[cid]=(blocks,inl,outl)
+print(f"[B] done; store residual={len(store)}", file=sys.stderr)
+
+TURNS=[t for t in sorted(by_in) if len(by_in[t])>=50]
+med_in=[pct(by_in[t],.5) for t in TURNS]
+med_new=[max(pct(by_new[t],.5),1) for t in TURNS]
+med_out=[pct(by_out[t],.5) for t in TURNS]
+ratio=[med_in[i]/med_new[i] for i in range(len(TURNS))]
+reuse_pct=[(1-med_new[i]/med_in[i])*100 for i in range(len(TURNS))]
+# C3 time per turn (cost model)
+t_pref=[med_new[i]/PREFILL_TOK_S for i in range(len(TURNS))]
+t_dec=[med_out[i]*TPOT_S for i in range(len(TURNS))]
+
+# aggregate decode/prefill time fraction over a RANGE of constants
+def agg_time(prate,tpot):
+    tp=tot_new_prefill/prate; td=tot_out*tpot; return td/(tp+td)
+frac_lo=agg_time(13000,0.005); frac_mid=agg_time(7000,0.010); frac_hi=agg_time(3000,0.025)
+
+chars={
+ "mixture":{"single_sessions":sr if False else sum(1 for s in seq.values() if len(s)==1),
+   "multi_sessions":sum(1 for s in seq.values() if len(s)>1),
+   "req_single_pct":sr/tot_r*100,"req_multi_pct":mr/tot_r*100,
+   "in_single_pct":sm/tot_in*100,"in_multi_pct":mm/tot_in*100,
+   "out_single_pct":so/tot_out*100,"out_multi_pct":mo/tot_out*100},
+ "turns":{"mean":st.mean(turns_per),"p99":pct(turns_per,.99),"max":max(turns_per),
+   "single_turn_pct":sum(1 for x in turns_per if x==1)/len(turns_per)*100},
+ "hazard":hazard,
+ "token_mass":{"total_input":tot_in,"total_output":tot_out,"out_in_ratio_pct":tot_out/tot_in*100,
+   "new_prefill":tot_new_prefill,"reused_prefix":tot_reused,
+   "new_prefill_pct_of_input":tot_new_prefill/tot_in*100},
+ "decode_time_fraction":{"optimistic_for_prefill":frac_lo,"mid":frac_mid,"pessimistic":frac_hi},
+ "per_turn":{"turn":TURNS,"med_resident_input":med_in,"med_new_prefill":med_new,
+   "med_output":med_out,"resident_over_new":ratio,"reuse_pct":reuse_pct},
+}
+json.dump(chars, open(f"{OUT}/chars.json","w"), indent=2)
+
+# ================= FIGURES =================
+plt.rcParams.update({"figure.dpi":140,"font.size":10,"axes.grid":True,"grid.alpha":.3})
+
+# ---- C1 ----
+fig,ax=plt.subplots(1,3,figsize=(15,4.2))
+cats=["% sessions","% requests","% input\ntokens","% output\ntokens"];
+singv=[chars["mixture"]["single_sessions"]/len(seq)*100, chars["mixture"]["req_single_pct"],
+       chars["mixture"]["in_single_pct"], chars["mixture"]["out_single_pct"]]
+multv=[100-x for x in singv]
+x=np.arange(len(cats))
+ax[0].bar(x,singv,label="single-turn",color="#7fb3d5")
+ax[0].bar(x,multv,bottom=singv,label="multi-turn",color="#e74c3c")
+for i in range(len(cats)):
+    ax[0].text(i,singv[i]/2,f"{singv[i]:.0f}",ha="center",va="center",fontsize=9)
+    ax[0].text(i,singv[i]+multv[i]/2,f"{multv[i]:.0f}",ha="center",va="center",color="white",fontsize=9)
+ax[0].set_xticks(x);ax[0].set_xticklabels(cats);ax[0].set_ylabel("%");ax[0].set_ylim(0,100)
+ax[0].set_title("C1a Mixture: 90% sessions single-turn,\nbut multi-turn carries 2/3 prefill mass");ax[0].legend(loc="center right")
+# turn CCDF log-log
+tc=sorted(turns_per); n=len(tc); xs=sorted(set(tc))
+ccdf=[sum(1 for v in tc if v>=xx)/n for xx in xs]
+ax[1].loglog(xs,ccdf,marker=".",ms=3,color="#34495e")
+ax[1].set_xlabel("turns per session (k)");ax[1].set_ylabel("P(turns >= k)")
+ax[1].set_title(f"C1b Heavy-tailed session length\n(p99={chars['turns']['p99']:.0f}, max={chars['turns']['max']})")
+# hazard
+hk=list(range(1,20)); hv=[hazard[k]*100 for k in hk]
+ax[2].plot(hk,hv,marker="o",color="#16a085")
+ax[2].set_xlabel("reached turn k");ax[2].set_ylabel("P(continue to k+1) %");ax[2].set_ylim(0,100)
+ax[2].set_title("C1c Continuation hazard rises 10%->94%\n(unpredictable at start, Lindy after)")
+fig.tight_layout(); fig.savefig(f"{OUT}/c1_session_mixture.png"); plt.close(fig)
+
+# ---- C2 ----
+fig,ax=plt.subplots(1,3,figsize=(15,4.2))
+ax[0].semilogy(TURNS,med_in,marker="o",label="resident context (input)",color="#e74c3c")
+ax[0].semilogy(TURNS,med_new,marker="s",label="new prefill this turn",color="#2980b9")
+ax[0].set_xlabel("turn");ax[0].set_ylabel("tokens (median, log)");ax[0].legend()
+ax[0].set_xlim(1,30)
+ax[0].set_title("C2a Resident state explodes,\nmarginal work collapses")
+ax[1].plot(TURNS,ratio,marker="o",color="#8e44ad")
+ax[1].set_xlabel("turn");ax[1].set_ylabel("resident / new-prefill");ax[1].set_xlim(1,30)
+ax[1].set_title("C2b The PD tax = resident/delta\n(grows to ~250x by deep turns)")
+ax[2].plot(TURNS,reuse_pct,marker="o",color="#27ae60")
+ax[2].set_xlabel("turn");ax[2].set_ylabel("per-turn reuse %");ax[2].set_ylim(50,100);ax[2].set_xlim(1,30)
+ax[2].set_title("C2c Per-turn reuse climbs to 99.6%\n(deep turns are near-pure cache hits)")
+fig.tight_layout(); fig.savefig(f"{OUT}/c2_work_amortization.png"); plt.close(fig)
+
+# ---- C3 ----
+fig,ax=plt.subplots(1,2,figsize=(11,4.4))
+# token mass decomposition
+vals=[tot_reused/1e9, tot_new_prefill/1e9, tot_out/1e9]
+labs=[f"reused prefix\n{tot_reused/tot_in*100:.0f}% of input",
+      f"new prefill\n{tot_new_prefill/tot_in*100:.0f}% of input",
+      f"decode output\n{tot_out/tot_in*100:.1f}% of input"]
+ax[0].bar(range(3),vals,color=["#95a5a6","#2980b9","#e67e22"])
+ax[0].set_xticks(range(3));ax[0].set_xticklabels(labs,fontsize=8.5)
+ax[0].set_ylabel("tokens (billions)")
+ax[0].set_title("C3a Token mass: prefill-dominated\n(but tokens != time, see C3b)")
+# per-turn prefill vs decode TIME (cost model)
+ax[1].semilogy(TURNS,t_pref,marker="o",label="prefill time (new tok / 7k·s⁻¹)",color="#2980b9")
+ax[1].semilogy(TURNS,t_dec,marker="s",label="decode time (out·10ms)",color="#e67e22")
+ax[1].set_xlabel("turn");ax[1].set_ylabel("seconds (median, log)");ax[1].legend(fontsize=8);ax[1].set_xlim(1,30)
+ax[1].set_title(f"C3b Prefill→decode bottleneck flips within a session\n(agg decode-time share ≈ {frac_mid*100:.0f}%, range {frac_lo*100:.0f}–{frac_hi*100:.0f}%)")
+fig.tight_layout(); fig.savefig(f"{OUT}/c3_prefill_decode_balance.png"); plt.close(fig)
+print("FIGURES + chars.json written to", OUT)
+print(json.dumps({k:chars[k] for k in ["mixture","turns","token_mass","decode_time_fraction"]}, indent=2))

figs/mb5/CORRECTION.md

@@ -0,0 +1,19 @@
+# ⚠️ Correction notice for figs/mb5/ (2026-05-30)
+
+These figures back `microbench/fresh_setup/PD_DISAGG_RESULTS.md`. A producer-side
+contamination was later found in the stack that produced them: commit **`e13391e`**
+(deployed over the "fresh" pip vLLM by `scripts/deploy_vllm_patches.sh`) evicts a
+producer's prefix-cache blocks on every KV transfer, so a disaggregated producer
+could never keep a session's prefix warm. It is now gated behind
+`VLLM_EVICT_SENT_BLOCKS` (default off) and everything was re-run clean.
+
+| figure | section | status |
+|---|---|---|
+| `mb5_producer_hotspot.png` | §6.3 session-affinity hot-pinning | 🛑 **RETRACTED** — pure `e13391e` artifact. On the clean stack, session-routed producers reach APC parity with colo (71–82%); there is no 0%-util stall / hot-pin pathology. |
+| `mb5_latency_compare.png` | §3 round-robin headline | ✅ stands — RR's ~0% prefix-hit is a *routing* artifact (consecutive turns → different producers), not the eviction bug; reproduced clean. |
+| `mb5_kv_timeline.png`, `mb5_role_split.png`, `mb5_peak_utilization.png` | §5 per-role KV pool occupancy | ✅ D-pool capacity-ceiling mechanism stands (decode pegs while prefill strands). P-pool occupancy may read slightly low under eviction; the qualitative split is unaffected. |
+| `mb5_summary.csv` | aggregate | mixed — §3/§5 rows valid; any session-affinity rows superseded. |
+
+**Superseded by the corrected three-axis ablation:** [`../mb5_pd_ablation/`](../mb5_pd_ablation/)
+(reuse / shape / concurrency), data in [`../../analysis/mb5_pd_ablation/`](../../analysis/mb5_pd_ablation/).
+Raw §6 data `analysis/mb5/session_prod.json` is contaminated; `analysis/mb5/rr_prod.json` (round-robin) stands.

figs/mb5/mb5_summary.csv

@@ -0,0 +1,5 @@
+config,rep,n_requests,n_success,wall_clock_s,peak_pool_frac,steady_pool_frac,p_pool_peak_frac,p_pool_steady_frac,d_pool_peak_frac,d_pool_steady_frac,peak_waiting,latency_p50_s,latency_p90_s,latency_p99_s,ttft_p50_s,ttft_p90_s,ttft_p99_s,prefix_cache_hit_ratio
+8C,1,1214,1214,2994.218414353032,0.7174957362137578,0.3439702956225128,,,,,29,10.82550932947197,83.34998885790122,194.10265863158946,6.967104309005663,53.12018221841427,114.12611859919207,0.1937163528742694
+6P+2D,1,1214,1214,3419.065942236979,0.7726478112563957,0.42145750426378625,0.743272692817889,0.3082291074474133,0.9959636156907333,0.7434906196702672,128,44.48975181748392,91.82252187062406,147.70196208347772,40.95952733900049,86.68752026481089,142.84028979733685,0.0
+4P+4D,1,1214,1214,4170.666486939997,0.6997939169982945,0.45876918703808983,0.6438459351904491,0.28540363843092664,0.9753411028993746,0.5977686185332576,152,59.52004547297838,157.08703426021387,224.03997302683115,56.419772224500775,153.07864206891392,219.73412787001706,0.0
+2P+6D,1,1214,109,5761.816568834998,0.9698692438885731,0.9435119386014781,0.9969869243888573,0.9198408186469585,0.9620238772029562,0.9494504453287853,872,26.293884326005355,499.3484142678091,577.7122636228032,23.580788671970367,498.0334587502061,576.5306194114453,0.0

microbench/connector_tax/cache_sweep/MIGRATION_TRANSFER_COST.md

@@ -0,0 +1,178 @@
+# Why KV-transfer is slow during migration under real load
+
+**Question.** EAR's unified+A+B routing beats migration (v3) on agentic
+workloads. We wanted to know whether *layerwise* KV transfer would shrink
+migration's overhead enough to make it viable. Investigating that led to a
+sharper question: **in a real (loaded) cluster, when we migrate, the KV
+transfer is already slow — the effective bandwidth is far below the
+~10 GB/s wire rate. Why?**
+
+This doc answers that with instrumented measurements.
+
+**TL;DR.** Migration fires precisely when instances are *busy* (that's the
+trigger). But on a busy instance, KV transfer runs at **~3 GB/s instead of
+~10 GB/s**, because:
+
+1. **The RDMA write itself slows ~2× under compute load** — GPU-direct RDMA
+   (`batch_transfer_sync_write`) contends with the running attention/MLP
+   kernels for **HBM and PCIe bandwidth**. (idle 7.6 GB/s → busy 4.0 GB/s)
+2. **The connector's Python control plane gets GIL-starved** — mooncake's
+   ZMQ handshake + transfer orchestration run on asyncio threads inside the
+   engine process; when the engine's main thread is doing a long forward
+   pass (e.g. a 100k-token prefill), those threads stall for *seconds*.
+
+Both are **inherent to upstream vLLM 0.18.1 + mooncake** (reproduced on a
+clean fresh venv; the transfer path is byte-identical to upstream — our
+patches did not cause this), and both get **worse**, not better, with
+layerwise transfer. So the bandwidth gap is not a layerwise problem; it is a
+*transfer-on-a-busy-GPU* problem.
+
+---
+
+## 1. Evidence chain
+
+Three independent measurements, all on dash0 (8×H100, Qwen3-Coder-30B-A3B,
+TP=1), Mooncake `kv_both`.
+
+### 1a. Instrumented v3 trace replay — where does migration time go?
+
+Run `outputs/b3_v3_fullbreak_20260528_0338/`. Instruments:
+`instrument_dst_migration.py` (dst scheduler lifecycle) +
+`instrument_mooncake.py` (connector internals: `send_blocks` RDMA,
+`receive_kv` window, `ready_wait`).
+
+25 migrations fired over the trace. Dst-side migration overhead
+(`T_kv_pull` = scheduler marks `WAITING_FOR_REMOTE_KVS` → `finished_recving`):
+
+| component | share | what it is |
+|---|---:|---|
+| RDMA-actual (`batch_transfer_sync_write`) | **55%** (55.2 s) | the real RDMA write |
+| dst control-plane gap | **45%** (45.4 s) | scheduler↔receiver_loop dispatch + completion propagation |
+| `ready_wait` (src KV not committed) | 0% | 25/25 already committed — **ruled out** |
+
+- Pure RDMA aggregate rate: **2.03 GB/s** (vs MB2 idle 9.7 GB/s).
+- RDMA rate **collapses with transfer size**: <3 GiB → 4–9.5 GB/s,
+  >5 GiB → 0.9–2.6 GB/s.
+- The control-plane gap is **bimodal**: median 0.04 s, but a handful of
+  requests stall ~10 s. Those are small-KV transfers (0.18 s of actual RDMA)
+  whose `T_kv_pull` is 8–11 s — i.e. the dst's `receiver_loop` thread was
+  GIL-starved for ~10 s while the engine did a big forward pass.
+
+> Earlier (pre-instrumentation) we wrongly attributed ~90% of migration
+> overhead to "dst scheduler queueing" by estimating transfer at clean wire
+> speed. With real instrumentation, dst *scheduler admission* is ~0
+> (`T_admission_post_kv` = 0.003 s); the time is the transfer phase (RDMA +
+> connector control plane), both degraded by instance busy-ness.
+
+### 1b. MB6 controlled microbench — does busy-ness cause it?
+
+`microbench/fresh_setup/mb6_transfer_under_load.py` + `run_mb6.sh`: 2
+instances, transfer a fixed-size KV (prefill on A → migrate to B) while
+holding *N* background decode streams on both. Sweep N.
+
+Effective transfer bandwidth (65k-token KV ≈ 6 GiB), main venv:
+
+| background load | 65k transfer | eff bandwidth |
+|---|---:|---:|
+| **0 (idle)** | 747 ms | **8.76 GB/s** |
+| 8 (4/instance) | 2423 ms | 4.53 GB/s |
+| **24 (12/instance)** | 2015 ms | **3.33 GB/s** |
+
+Monotonic degradation with load. **The busy level (3.3 GB/s) matches the
+v3 trace's 3.3 GB/s median exactly** — because agentic instances run
+~10+ concurrent requests, i.e. the bg=24 regime.
+
+Decomposing the 65k transfer into RDMA-actual vs control-plane:
+
+| bg | RDMA rate | control-plane share |
+|---|---:|---:|
+| 0 (idle) | 7.56 GB/s | 13% |
+| 8 | 4.07 GB/s | 11% |
+| 24 (busy) | 3.97 GB/s | 15% |
+
+In the clean microbench the **RDMA write itself is the dominant degrading
+term** (7.6 → 4.0 GB/s). The ~10 s control-plane stalls seen in the trace
+(1a) don't reproduce here because steady decode forward passes are short;
+they require the long (100k-token) prefills that the real trace has.
+
+### 1c. Fresh-venv comparison — is it our patch?
+
+Same MB6 sweep on `agentic-kv-fresh/.venv` (clean upstream-style 0.18.1):
+
+| bg | 65k eff (fresh) | 65k eff (main/patched) |
+|---|---:|---:|
+| 0 | 8.73 GB/s | 8.76 GB/s |
+| 8 | 4.52 GB/s | 4.53 GB/s |
+| 24 | 3.27 GB/s | 3.33 GB/s |
+
+**Identical within noise.** Plus a static check: the v3 transfer path
+(`send_kv_to_decode`, `_send_blocks`/`batch_transfer_sync_write`,
+`_build_transfer_params`) is **byte-identical** to pristine upstream 0.18.1
+(commit `445e491`); `receive_kv_from_single_worker` differs only by a 4-line
+error branch. Our mooncake commits (`a7df84b` direct-read,
+`ea51497` partial-prefill, `e3a1d70` read→push) only touch a *separate*
+`direct_read` path that v3 does **not** use (v3 requests carry no
+`direct_read` flag → normal push path).
+
+→ **The slowdown is upstream/hardware-inherent, not introduced by us.**
+
+---
+
+## 2. Root cause
+
+Migration in agentic serving transfers KV **between instances that are
+concurrently busy with compute** — by construction, since v3 migrates *away
+from* a busy host. On a busy instance:
+
+- **HBM/PCIe contention (the dominant, irreducible part).** Mooncake's
+  transfer is GPU-direct RDMA: the NIC DMAs KV straight out of / into GPU
+  HBM. While the GPU runs attention+MLP kernels, those kernels saturate HBM
+  bandwidth, so the NIC's RDMA gets a smaller slice. Effective transfer
+  bandwidth roughly halves (7.6 → 4.0 GB/s at our load), and degrades
+  further for large multi-segment transfers.
+- **Control-plane GIL starvation (secondary, bursty).** The connector runs
+  its ZMQ handshake + `send_kv_to_decode`/`receive_kv` orchestration on
+  asyncio threads (`sender_loop`/`receiver_loop`) *inside the engine
+  process*. A long forward pass (100k-token prefill) holds the GIL for
+  seconds, stalling those threads → multi-second dispatch gaps even when the
+  actual transfer is 0.2 s.
+
+MB2 measured 9.7 GB/s precisely because both endpoints were **idle**. The
+real-workload gap is the difference between "idle benchmark" and "transfer
+while the GPU is doing the day job."
+
+---
+
+## 3. Implication: layerwise is the wrong lever; migration's tax is largely irreducible
+
+| lever | effect on the gap |
+|---|---|
+| **Model-level layerwise transfer** (push each layer's KV during prefill) | **Worse.** Prefill is the most HBM-intensive phase, so per-layer transfers contend *harder* for HBM (Cause 1); and they multiply the control-plane round-trips (Cause 2). |
+| **Control-plane fix** (move mooncake orchestration off the GIL-contended threads / separate process) | Addresses only the bursty ~10 s stalls (~15% in the clean case, up to ~45% of the trace tail). Does **not** touch the HBM-contention half. |
+| **Reduce bytes** (cache-aware target so less KV moves) | Helps linearly; v3 Mechanism B already does some. Orthogonal. |
+| **Migrate to/from idle instances** | Would restore ~10 GB/s — but defeats the purpose (we migrate *because* the host is busy). |
+
+The dominant cost (RDMA contending with compute for HBM on busy instances)
+is a **hardware reality**, not a software bug we can patch away, and not
+something layerwise improves. This reinforces
+[UNIFIED_ABLATION.md](UNIFIED_ABLATION.md): the unified no-migration path
+(A+B'+RaceFix) remains the right default; migration's transfer tax is
+structural on a loaded agentic cluster.
+
+---
+
+## 4. Repro / artifacts
+
+- Instrumented v3 breakdown: `outputs/b3_v3_fullbreak_20260528_0338/unified_v3/`
+  (`transfer_decomp.txt`, `dst_migration_breakdown.{csv,png}`,
+  `transfer_rootcause.png`)
+- MB6 main: `outputs/mb6_agentic-kv_20260528_0552/mb6_result.json`
+- MB6 fresh: `outputs/mb6_fresh_20260528_0559/mb6_result.json`
+- Instruments: `microbench/fresh_setup/instrument_dst_migration.py`,
+  `microbench/fresh_setup/instrument_mooncake.py`
+- Microbench: `microbench/fresh_setup/mb6_transfer_under_load.py` +
+  `run_mb6.sh` (`VENV=… bash run_mb6.sh`)
+- Analyzers: `analyze_dst_migration.py`, `analyze_transfer_decomp.py`
+
+All instruments apply/revert cleanly via `--apply`/`--revert`; both venvs
+were restored after the runs.

microbench/connector_tax/cache_sweep/UNIFIED_ABLATION.md

@@ -0,0 +1,431 @@
+# Unified routing ablation: A (tighter affinity) + B (decode-aware LMetric)
+
+Goal: judge whether `unified` (cache-aware hybrid affinity + LMetric fallback)
+has enough headroom to surpass v3 migration-based routing on agentic
+workloads, without invoking PD-sep migration.
+
+## Workload / baseline
+
+- Trace: `w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions)
+- Hardware: 8 × H100 (dash0), Qwen3-Coder-30B-A3B, TP=1, max_model_len=200000
+- Trace replay through `cache_aware_proxy.py` with policy `unified`
+- `b3_replay_20260527_0114/unified/` reference
+
+| Metric (ms) | baseline (`overload_factor=2.0`) |
+|---|---:|
+| TTFT p50 | 520 |
+| TTFT p90 | **8781** |
+| TTFT p99 | 47647 |
+| TPOT p90 | 17.8 |
+| E2E p90 | 19989 |
+| E2E p99 | 85841 |
+
+Reference points we're trying to beat / match:
+- v3 fixed rotation (cache-blind picker): TTFT p90 = 10828
+- v3 + Mechanism B (cache-rich picker): TTFT p90 = 9711
+- All v3 variants are +10–23% worse than `unified` baseline.
+
+## Tail-source diagnostic on baseline
+
+Decision split, baseline unified:
+
+| Decision | n | TTFT mean | TTFT p90 | TTFT p99 |
+|---|---:|---:|---:|---:|
+| affinity     | 852 | 3183 | 7011 | 47432 |
+| lmetric_fallback | 362 | 4285 | 12083 | 46036 |
+
+Long-tail (>20s, n=65):
+- 40 / 65 came from `affinity` decisions
+- 25 / 65 came from `lmetric_fallback`
+
+For the 40 slow `affinity` reqs:
+- only 12 / 40 were actually overloaded at decision time (`aff_num_req > avg_num_req`)
+- overload ratio at decision: mean=0.93, p50=0.87
+- **most slow affinity reqs looked fine when the picker stuck — load piled
+  on after dispatch**.
+
+This is a snapshot-based-routing limitation. Tightening
+`overload_factor` only helps the genuine cases above the new threshold —
+expected to be a 5-10% improvement at best.
+
+---
+
+## Direction A — tighten affinity overflow
+
+**Hypothesis.** `overload_factor=2.0` lets the picker stick to affinity
+even when `affinity.num_req` is up to 2× the cluster average. Reducing to
+1.3 forces earlier overflow to LMetric fallback, escaping busy affinity
+hosts before the tail blows up.
+
+**Change.** Single CLI flag: `--overload-factor 1.3`. No code change.
+
+**Run.** `unified_of13_20260527_1532/unified/`.
+
+### A vs baseline
+
+| Metric (ms) | baseline (of=2.0) | A (of=1.3) | Δ |
+|---|---:|---:|---:|
+| TTFT p50 | 520 | 495 | −5% |
+| TTFT p90 | 8781 | 8730 | ≈0 |
+| TTFT p99 | 47647 | 43059 | −10% |
+| TPOT p50 | 7.9 | 8.0 | ≈0 |
+| TPOT p90 | 17.8 | **15.5** | **−13%** |
+| E2E p50 | 1761 | 1824 | +4% |
+| E2E p90 | 19989 | 18407 | −8% |
+| E2E p99 | 85841 | **71396** | **−17%** |
+
+TTFT p90 is essentially unchanged but the **deeper tail (p99) and
+TPOT both improved meaningfully**. Net: A alone gives roughly −10% to
+−17% on the long tail without hurting medians.
+
+### Decision split, A vs baseline
+
+| Decision | baseline n / p90 | A n / p90 | Δ p90 |
+|---|---|---|---|
+| affinity | 852 / 7011 | 817 / **5817** | **−17%** ✅ |
+| lmetric_fallback | 362 / 12083 | 397 / **15360** | **+27%** ⚠️ |
+
+The picker now sticks to affinity 35 fewer times. The remaining affinity
+decisions are higher-quality (no longer "barely-fitting" cases), so their
+p90 drops 17%.
+
+But the 35 extra reqs that got pushed into fallback **got slower**:
+fallback p90 went from 12083 → 15360. The LMetric scorer is selecting a
+worse instance for them.
+
+### Per-worker TTFT under A (of=1.3)
+
+```
+port 8000: n=  94  mean=4424  p90=12290    port 8004: n=192  mean=2597  p90=6968
+port 8001: n= 135  mean=2779  p90= 5553    port 8005: n=202  mean=3102  p90=6113
+port 8002: n=  88  mean=5827  p90=15804    port 8006: n=136  mean=4006  p90=10899
+port 8003: n= 217  mean=2674  p90= 4598    port 8007: n=150  mean=3648  p90= 7025
+```
+
+Compared to baseline (88..217 reqs/port), A redistributes more evenly
+(88..217 still but distribution is fatter in the middle). port 8002
+remains slow (p90 15.8s) — its cache pool seems to keep getting cold
+work routed there by LMetric.
+
+### Why A alone isn't enough
+
+LMetric scorer (`unified_hybrid` fallback path):
+
+```python
+score = (pending_prefill_tokens + new_uncached_tokens) * num_requests
+```
+
+This **ignores `ongoing_decode_tokens`** entirely. An instance with no
+pending prefill but 200k tokens currently in decode looks "ideal"
+(score=0×num_req=0) — yet a new request landing there waits behind
+slow decode iters caused by the large batch KV reads.
+
+A pushes more requests into fallback, but fallback can't tell which
+instance is actually free. → Direction B is mandatory companion.
+
+---
+
+## Direction B — decode-aware LMetric
+
+**Hypothesis.** Adding a decode-load penalty to the LMetric score lets
+fallback distinguish "no prefill queued but heavy decode running" from
+"truly idle". Should restore fallback p90 ≤ 12s baseline level.
+
+**Change.**
+```python
+score = (pending_prefill + new + lmetric_decode_weight * ongoing_decode_tokens) * num_requests
+```
+- `lmetric_decode_weight=0.0` ⇒ original LMetric (control)
+- `lmetric_decode_weight=0.01` ⇒ first experiment (rationale: 1 decode token
+  in batch costs ~0.01 prefill-token-equivalent in scheduler iter time
+  on H100 + Qwen3-30B-A3B)
+
+CLI: `--lmetric-decode-weight 0.01`. Setting in code:
+`cache_aware_proxy.py:Settings.lmetric_decode_weight`.
+
+**Run.** `unified_of13_lmw001_20260527_1628/unified/`.
+
+### A+B vs baseline / A
+
+| Metric (ms) | baseline | A (of=1.3) | A+B (of=1.3, lmw=0.01) | Δ vs baseline |
+|---|---:|---:|---:|---:|
+| TTFT p50 | 520 | 495 | 514 | −1% |
+| **TTFT p90** | 8781 | 8730 | **8421** | **−4%** ✅ |
+| TTFT p99 | 47647 | 43059 | 44800 | −6% |
+| TPOT p50 | 7.9 | 8.0 | 7.9 | ≈0 |
+| TPOT p90 | 17.8 | 15.5 | 15.7 | −12% |
+| E2E p50 | 1761 | 1824 | 1870 | +6% |
+| E2E p90 | 19989 | 18407 | **21064** | **+5%** ⚠️ |
+| E2E p99 | 85841 | 71396 | **64344** | **−25%** ✅ |
+
+Long-tail counts:
+
+```
+thresh       baseline      A      A+B   v3 MechB
+>  5000ms        170      173      170        177
+> 10000ms        105      109      109        119
+> 20000ms         65       64       59         78
+> 30000ms         41       40       37         50
+> 50000ms          8        5        6         14
+```
+
+A+B is best on every long-tail-count threshold ≤30s, marginal worse at 50s.
+
+### Decision split (A+B vs A)
+
+| Decision | A (of=1.3) | A+B | Note |
+|---|---|---|---|
+| affinity p90 | 5817 | 5836 | ≈ same |
+| fallback p90 | **15360** | **13501** | B recovered some of A's fallback regression |
+
+B partially fixed fallback's selection (−12% on fallback p90 vs A alone),
+but still worse than baseline (12083).
+
+### Per-worker TTFT (A+B)
+
+```
+port 8000: n=134  mean=3495  p90=10967      port 8004: n=136  mean=3102  p90= 7906
+port 8001: n=143  mean=2981  p90=10189      port 8005: n=179  mean=1624  p90= 2735
+port 8002: n=221  mean=2355  p90= 3502      port 8006: n=137  mean=5356  p90= 9628
+port 8003: n=146  mean=3932  p90=10729      port 8007: n=118  mean=5210  p90=26798  ← new hotspot
+```
+
+A+B trades the baseline's 8002 hotspot (p90=35s) for a new 8007 hotspot
+(p90=26.8s). Lower amplitude but hotspot survives.
+
+### Why 8007 became a hotspot under A+B — **found a bug in B**
+
+8007 in A+B: 118 reqs, **53% affinity / 47% fallback** (vs other ports
+60–77% affinity), **cache_hit_mean=50.5% (lowest)**.
+
+Top-10 slowest at 8007: all are big-prompt (100k+ tokens) fallback decisions
+with `cached_tokens=0` (cold prefill). LMetric is pushing many cold-prefill
+fallbacks to 8007.
+
+Looking at the B formula:
+
+```python
+decode_pen = lmetric_decode_weight * ongoing_decode_tokens
+score = (pending_prefill + new + decode_pen) * num_requests   # ← BUG
+```
+
+When `num_requests = 0`, the entire score (including decode penalty) zeros
+out. So an idle-but-decoding host (num_req=0 because its last prefill
+finished but decode is still running) looks like score=0, beating every
+busy host.
+
+**Fix (B'):** multiply by `max(num_requests, 1)`:
+
+```python
+score = (pending_prefill + new + decode_pen) * max(num_requests, 1)
+```
+
+Now idle hosts with high decode load get score = decode_pen × 1 = real
+nonzero penalty, beating zero-load hosts only when decode is small.
+
+### A+B' — re-run with the fix
+
+**Run.** `unified_of13_lmw001_v2_20260527_1724/unified/`.
+
+| Metric (ms) | baseline | A+B (BUG) | A+B' (fix) | Δ vs baseline |
+|---|---:|---:|---:|---:|
+| TTFT p50 | 520 | 514 | **485** | −7% |
+| **TTFT p90** | 8781 | 8421 | **8287** | **−5.6%** ✅ |
+| TTFT p99 | 47647 | 44800 | **41876** | **−12%** ✅ |
+| TPOT p90 | 17.8 | 15.7 | 17.5 | −2% |
+| E2E p90 | 19989 | 21064 | 20625 | +3% |
+| E2E p99 | 85841 | 64344 | 77827 | −9% |
+
+A+B' **best of all variants on TTFT p90 (8287) and TTFT p99 (41876)**.
+Long-tail counts (>30s, >50s) also best across variants.
+
+vs v3 reference points:
+| | TTFT p90 | TPOT p90 | E2E p99 |
+|---|---:|---:|---:|
+| **A+B'** | **8287** | 17.5 | 77827 |
+| v3 fixed (cache-blind) | 10828 | 21.0 | 47610 |
+| v3 + Mech B | 9711 | 18.3 | 84492 |
+
+A+B' **beats v3 Mech B by 15% TTFT p90** with no migration overhead.
+
+### Per-worker (A+B' fixed)
+
+```
+8000: n=158  p90= 5688      8004: n=189  p90= 4249
+8001: n=159  p90= 7323      8005: n=116  p90=14598
+8002: n=114  p90= 8726      8006: n=180  p90= 6198
+8003: n=173  p90= 6715      8007: n=125  p90=22242   ← still hot
+```
+
+A+B' redistributed load more evenly (114..189) but **8007 still has p90=22s**.
+
+### 8007 deep-dive in A+B'
+
+```
+8007: n=125, affinity=69 (55%), fallback=56 (45%), cache_hit_mean=lowest
+```
+
+Top-15 slow at 8007:
+- 7 of them are session **1313181** turns 9–14 (130k+ tokens each, agentic
+  long context, ~50% cache hit)
+- Several others are cold-start turn-1 of large-prompt sessions
+- First two slow reqs arrived **0.7 s apart** — strong hint of concurrent
+  picker race
+
+### Iteration 3: race-condition fix
+
+**Diagnosis.** In `_handle_combined`:
+
+```python
+chosen, best_idx, decision = pick_instance_unified_hybrid(...)  # sync
+# ... sync breakdown updates ...
+return await _handle_local_request(...)   # ← await yields here
+                                          #   THEN reservation happens
+```
+
+`return await async_func(...)` evaluates the async call (creates coroutine)
+and yields to the event loop **before** the coroutine body executes. The
+reservation (`chosen.pending_prefill_tokens += new`, etc.) lives at the top
+of `_handle_local_request`, so between the picker and the reservation there
+is a **window where another coroutine can run and re-pick the same instance**.
+
+When two big-prompt reqs arrive within milliseconds, both run pick →
+both pick the "free" 8007 → both yield → both reserve. Result: 8007 gets
+back-to-back 130k-token cold prefills, each waiting for the other.
+
+**Fix.** Move the reservation **before** the await, inside `_handle_combined`:
+
+```python
+# Race fix: reserve atomically with pick, before any await.
+chosen.ongoing_tokens += input_length
+chosen.pending_prefill_tokens += estimated_new
+chosen.num_requests += 1
+return await _handle_local_request(..., _pre_reserved=True)
+```
+
+`_handle_local_request` skips its own reservation when `_pre_reserved=True`.
+PD-sep paths are unaffected (they have their own reservation).
+
+**Run.** Pending — `unified_of13_lmw001_racefix_*`. Hypothesis: 8007 p90
+drops to within ±3s of cluster median, since concurrent picks for the
+same "free" instance no longer happen.
+
+---
+
+## A+B'+RaceFix — results
+
+**Run.** `unified_of13_lmw001_racefix_20260527_1821/unified/`.
+
+| Metric (ms) | baseline | A+B' | A+B'+RF | Δ vs baseline |
+|---|---:|---:|---:|---:|
+| TTFT p50 | 520 | 485 | **478** | −8% |
+| **TTFT p90** | 8781 | 8287 | **7770** | **−11.5%** ✅ |
+| TTFT p99 | 47647 | 41876 | **42447** | −11% |
+| TPOT p90 | 17.8 | 17.5 | 18.0 | +1% |
+| E2E p90 | 19989 | 20625 | **18418** | −8% |
+| E2E p99 | 85841 | 77827 | **71227** | −17% |
+
+vs v3 reference:
+- **A+B'+RF TTFT p90 = 7770ms, vs v3 Mech B 9711ms → −20%** ✅
+
+Long-tail counts (best across all variants):
+```
+> 5s:   170 → 158         > 30s:   41 → 33
+>10s:   105 → 103         > 50s:    8 →  4
+>20s:    65 →  57         >100s:    0 →  0
+```
+
+### Decision split — race fix mainly helped affinity
+
+| Decision | baseline | A+B'+RF |
+|---|---:|---:|
+| affinity p90 | 7011 | **5042** ✅ (−28%) |
+| fallback p90 | 12083 | 13944 (+15%) |
+
+The race-condition was hurting affinity decisions the most. When two
+concurrent reqs both stuck to a "free-looking" affinity instance, they
+piled up and inflated affinity's tail. Fix removed this collision.
+
+### Per-worker
+
+```
+8000: n=86   p90=11541      8004: n=150  p90=11906
+8001: n=186  p90= 8307      8005: n=109  p90= 4798
+8002: n=105  p90=14540      8006: n=183  p90= 6258
+8003: n=264  p90= 3079      8007: n=131  p90=21850   ← still hot
+8000 spread now 86..264 — race fix did disperse routing
+```
+
+### 8007 still hot — but it's **workload-inherent, not a routing bug**
+
+Top sessions on 8007:
+```
+session 1279412: n=22  mean= 2208  max=18985  decisions: 91% affinity
+session 1313181: n=17  mean=17399  max=49089  decisions: 65% affinity
+session 1262354: n=15  mean=  622  max= 2325  decisions: 87% affinity
+session 1342921: n= 7  mean=17817  max=55589  decisions: 86% affinity
+session 1260327: n= 8  mean= 1636  max= 5382  decisions: 75% affinity
+session 1268831: n= 5  mean= 1443  max= 2673  decisions: 80% affinity
+```
+
+Sessions 1313181 and 1342921 are **long agentic contexts**: 100k–130k tokens
+per turn with ~50% cache hit (i.e. 50k new tokens prefill per turn). Even
+on a perfectly load-balanced instance, each turn is 7–15s of pure compute.
+
+Forcing these sessions to spread across instances would mean **cold prefill
+every turn (0% cache hit)** → each turn becomes 20–30s instead of 7–15s.
+Spreading is **net-negative**.
+
+→ The 8007 p90=22s is the floor imposed by these sessions' structure,
+not by routing policy. Unified is at its ceiling for this workload.
+
+---
+
+## Final ranking and take-aways
+
+| Policy | TTFT p90 (ms) | Δ vs baseline | Notes |
+|---|---:|---:|---|
+| baseline unified (of=2.0) | 8781 | — | reference |
+| A (of=1.3) | 8730 | ≈0 | affinity p90 -17%, fallback p90 +27% |
+| A+B (of=1.3, lmw=0.01, BUG) | 8421 | −4% | 8007 hotspot from `*num_req` zeroing bug |
+| A+B' (formula fix) | 8287 | −5.6% | Bug fixed, still 8007 mild hotspot |
+| **A+B'+RaceFix** | **7770** | **−11.5%** ✅ | **Best unified variant** |
+| v3 fixed | 10828 | +23% | PD-sep migration, cache-blind picker |
+| v3 + Mech B | 9711 | +11% | PD-sep + cache-rich target picker |
+
+### Conclusions
+
+1. **Unified path beats v3 PD-sep on this workload by 20%+ TTFT p90.**
+   PD-sep migration's fixed cost (src prefill + dst first-token waiting on
+   loaded scheduler) outweighs any decode-time savings for short-output
+   agentic turns.
+
+2. **Three orthogonal fixes compound for a 11.5% TTFT p90 win:**
+   - A (`overload_factor=1.3`): tighter affinity overflow → −0.6% but
+     much cleaner affinity decisions (p90 -17%)
+   - B' (`lmetric_decode_weight=0.01` with `max(num_req,1)`): decode-aware
+     fallback → −3.5%
+   - RaceFix (atomic reserve before await): kills concurrent-pick
+     collisions → −5.6%
+
+3. **Race condition was the biggest single hidden bug.** `return await
+   async_func(...)` yields to the event loop **before** the body of
+   `async_func` runs, so reservations done in the body don't take effect
+   in time to deter concurrent picks. This affects ANY async dispatch
+   with separate pick/reserve steps — worth checking other routing
+   policies.
+
+4. **8007 p90=22s is workload-inherent.** Sessions with 100k+ token turns
+   at 50% cache hit cannot finish faster than 7–15s per turn regardless
+   of routing. Forcing spread would hurt rather than help.
+
+5. **Migration (v3) is not necessary** when unified routing is tuned
+   well. Save the PD-sep mechanism for cases where it can be proven
+   net-positive (e.g. very-long-output sessions on extremely overloaded
+   prefill hosts) and use unified A+B'+RaceFix as the default.
+
+---
+
+## Direction A+B — run pending
+
+(Will be filled when `unified_of13_lmw001_*/unified/` finishes.)

microbench/connector_tax/cache_sweep/analyze_b3_replay.py

@@ -0,0 +1,169 @@
+#!/usr/bin/env python3
+"""B3 5-policy re-test analyser.
+
+Compute TTFT/TPOT/E2E mean/p50/p90/p99 for each policy from
+metrics.jsonl, compare against the historical b3_policy_comparison.json
+that drives fig_b3_latency_bars.png, and emit a side-by-side table
+plus a new figure with the same layout as the original.
+
+Usage:
+  python analyze_b3_replay.py --root <outroot> [--old-data <path>] [--figure <path>]
+"""
+
+import argparse
+import json
+import statistics
+from pathlib import Path
+
+
+POLICIES = ["lmetric", "load_only", "sticky", "unified", "unified_v2"]
+
+
+def pct(xs, p):
+    if not xs:
+        return None
+    xs = sorted(xs)
+    k = max(0, min(len(xs) - 1, int(p / 100.0 * (len(xs) - 1))))
+    return xs[k]
+
+
+def summarise(path):
+    rows = [json.loads(l) for l in open(path) if l.strip()]
+    ok = [r for r in rows if not r.get("error")]
+    ttft = [r["ttft_s"] * 1000 for r in ok if r.get("ttft_s") is not None]
+    tpot = [r["tpot_s"] * 1000 for r in ok if r.get("tpot_s")]
+    e2e = [r["latency_s"] * 1000 for r in ok if r.get("latency_s") is not None]
+    return {
+        "n_total": len(rows),
+        "n_ok": len(ok),
+        "ttft_mean_ms": statistics.mean(ttft) if ttft else None,
+        "ttft_p50_ms": pct(ttft, 50),
+        "ttft_p90_ms": pct(ttft, 90),
+        "ttft_p99_ms": pct(ttft, 99),
+        "tpot_mean_ms": statistics.mean(tpot) if tpot else None,
+        "tpot_p50_ms": pct(tpot, 50),
+        "tpot_p90_ms": pct(tpot, 90),
+        "tpot_p99_ms": pct(tpot, 99),
+        "e2e_mean_ms": statistics.mean(e2e) if e2e else None,
+        "e2e_p50_ms": pct(e2e, 50),
+        "e2e_p90_ms": pct(e2e, 90),
+        "e2e_p99_ms": pct(e2e, 99),
+    }
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--root", type=Path, required=True)
+    ap.add_argument("--old-data", type=Path,
+                    default=Path("analysis/characterization/window_1_results/b3_policy_comparison.json"))
+    ap.add_argument("--figure", type=Path, default=None)
+    args = ap.parse_args()
+
+    new = {}
+    for p in POLICIES:
+        path = args.root / p / "metrics.jsonl"
+        if not path.exists():
+            print(f"MISSING: {path}")
+            continue
+        new[p] = summarise(path)
+
+    old = {}
+    if args.old_data.exists():
+        d = json.load(open(args.old_data))
+        for r in d.get("rows", []):
+            old[r["policy"]] = {
+                "ttft_p50_ms": r["ttft_p50_s"] * 1000,
+                "ttft_p90_ms": r["ttft_p90_s"] * 1000,
+                "ttft_p99_ms": r["ttft_p99_s"] * 1000,
+                "tpot_p90_ms": r["tpot_p90_s"] * 1000,
+                "e2e_p90_ms":  r.get("e2e_p90_s", 0) * 1000,
+            }
+
+    def fmt(v): return f"{v:.0f}" if v is not None else "-"
+    def pctd(a, b):
+        if a is None or b is None or a == 0: return "-"
+        return f"{(b/a-1)*100:+.1f}%"
+
+    # Headline table
+    print(f"\n# NEW: today's re-test")
+    print(f"{'policy':<14}{'n_ok':>6}{'TTFTp50':>10}{'TTFTp90':>10}{'TTFTp99':>10}{'TPOTp90':>10}{'E2Ep90':>10}")
+    print("-" * 70)
+    for p in POLICIES:
+        if p not in new: continue
+        r = new[p]
+        print(f"{p:<14}{r['n_ok']:>6}{fmt(r['ttft_p50_ms']):>9}ms{fmt(r['ttft_p90_ms']):>9}ms{fmt(r['ttft_p99_ms']):>9}ms{fmt(r['tpot_p90_ms']):>9}ms{fmt(r['e2e_p90_ms']):>9}ms")
+
+    print(f"\n# OLD: window_1_results/b3_policy_comparison.json")
+    print(f"{'policy':<14}{'TTFTp50':>10}{'TTFTp90':>10}{'TTFTp99':>10}{'TPOTp90':>10}{'E2Ep90':>10}")
+    print("-" * 60)
+    for p in POLICIES:
+        if p not in old: continue
+        r = old[p]
+        print(f"{p:<14}{fmt(r['ttft_p50_ms']):>9}ms{fmt(r['ttft_p90_ms']):>9}ms{fmt(r['ttft_p99_ms']):>9}ms{fmt(r['tpot_p90_ms']):>9}ms{fmt(r['e2e_p90_ms']):>9}ms")
+
+    print(f"\n# DRIFT: today vs old (same policy)")
+    print(f"{'policy':<14}{'ΔTTFTp50':>10}{'ΔTTFTp90':>10}{'ΔTTFTp99':>10}{'ΔTPOTp90':>10}{'ΔE2Ep90':>10}")
+    print("-" * 60)
+    for p in POLICIES:
+        if p not in new or p not in old: continue
+        n, o = new[p], old[p]
+        print(f"{p:<14}{pctd(o['ttft_p50_ms'], n['ttft_p50_ms']):>10}"
+              f"{pctd(o['ttft_p90_ms'], n['ttft_p90_ms']):>10}"
+              f"{pctd(o['ttft_p99_ms'], n['ttft_p99_ms']):>10}"
+              f"{pctd(o['tpot_p90_ms'], n['tpot_p90_ms']):>10}"
+              f"{pctd(o['e2e_p90_ms'], n['e2e_p90_ms']):>10}")
+
+    # Relative ordering check
+    def ranks(values_dict, key):
+        items = [(p, r[key]) for p, r in values_dict.items() if r.get(key)]
+        items.sort(key=lambda x: x[1])
+        return [p for p, _ in items]
+
+    print(f"\n# TTFT p90 ranking (best → worst)")
+    for label, src in [("OLD", old), ("NEW", new)]:
+        if src:
+            order = ranks(src, "ttft_p90_ms")
+            print(f"  {label}: {' < '.join(order)}")
+
+    out = {"new": new, "old": old}
+    out_path = args.root / "b3_replay_summary.json"
+    out_path.write_text(json.dumps(out, indent=2))
+    print(f"\nWrote {out_path}")
+
+    # Bar plot (matplotlib)
+    if not args.figure:
+        args.figure = args.root / "fig_b3_latency_bars_new.png"
+    try:
+        import matplotlib
+        matplotlib.use("Agg")
+        import matplotlib.pyplot as plt
+
+        pols = [p for p in POLICIES if p in new]
+        metrics = [("TTFT p90 (s)", "ttft_p90_ms", 1000),
+                    ("TPOT p90 (ms)", "tpot_p90_ms", 1),
+                    ("E2E p90 (s)", "e2e_p90_ms", 1000)]
+        colors = {"lmetric": "tab:blue", "load_only": "tab:orange",
+                  "sticky": "tab:green", "unified": "tab:red",
+                  "unified_v2": "tab:purple"}
+        fig, axes = plt.subplots(1, 3, figsize=(14, 4.5))
+        for ax, (label, key, div) in zip(axes, metrics):
+            vals = [new[p][key] / div for p in pols]
+            bars = ax.bar(pols, vals,
+                          color=[colors.get(p, "gray") for p in pols],
+                          edgecolor="black", linewidth=0.5)
+            ax.set_title(label)
+            ax.tick_params(axis="x", rotation=20)
+            for b, v in zip(bars, vals):
+                ax.text(b.get_x() + b.get_width() / 2, v, f"{v:.1f}",
+                        ha="center", va="bottom", fontsize=9)
+            ax.grid(alpha=0.3, axis="y")
+        fig.suptitle(f"B3 5-policy re-test ({args.root.name})")
+        fig.tight_layout()
+        fig.savefig(args.figure, dpi=120)
+        print(f"Wrote {args.figure}")
+    except Exception as e:
+        print(f"(figure skipped: {e})")
+
+
+if __name__ == "__main__":
+    main()

microbench/connector_tax/cache_sweep/analyze_dst_migration.py

@@ -0,0 +1,333 @@
+#!/usr/bin/env python3
+"""Analyze dst-side migration breakdown for unified_v3 runs.
+
+Joins the proxy `breakdown.json` (per-request route + phase timestamps)
+with the dst engine per-PID logs written by
+`instrument_dst_migration.py` (`dm_mig_pid<pid>.jsonl`), to attribute
+each migration's dst-side wall-clock into:
+
+  T_relay              proxy decode-sent → dst arrival
+  T_admission_pre_kv   dst arrival → status=WAITING_FOR_REMOTE_KVS
+                       (waiting in dst's scheduler queue before KV pull
+                        is even initiated)
+  T_kv_pull            WAITING_FOR_REMOTE_KVS → finished_recving
+                       (the actual RDMA transfer + connector ack)
+  T_admission_post_kv  finished_recving → first time in self.running
+                       (KV ready, waiting for batch slot)
+  T_first_iter         first scheduled → first generated token
+                       (one decode-iter compute + sampler latency)
+
+Layerwise transfer can at best eliminate T_kv_pull. Everything else is
+queueing or compute that layerwise does not touch.
+
+Usage:
+    python analyze_dst_migration.py \
+        --proxy-breakdown <RUNDIR>/breakdown.json \
+        --dst-log-dir     <DST_LOG_DIR>
+        [--output         <RUNDIR>/dst_migration_breakdown.csv]
+        [--plot           <RUNDIR>/dst_migration_breakdown.png]
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import re
+import statistics
+import sys
+from pathlib import Path
+
+
+def _core_req_id(rid: str) -> str:
+    """Normalize a vLLM engine req_id back to the proxy's request_id.
+
+    vLLM wraps the proxy id `S:T:U:N` as `cmpl-S:T:U:N-<dp_rank>-<hex>`.
+    Strip the `cmpl-` prefix and the trailing `-<digits>-<hex>` suffix so
+    it joins against the proxy `breakdown.json` request_id.
+    """
+    if not rid:
+        return rid
+    s = rid
+    if s.startswith("cmpl-"):
+        s = s[len("cmpl-"):]
+    m = re.match(r"^(.*)-\d+-[0-9a-fA-F]+$", s)
+    if m:
+        s = m.group(1)
+    return s
+
+
+def _pct(vals: list[float], q: float) -> float:
+    if not vals:
+        return float("nan")
+    vs = sorted(vals)
+    i = max(0, min(len(vs) - 1, int(math.ceil(q * len(vs))) - 1))
+    return vs[i]
+
+
+def _summary(name: str, vals: list[float]) -> dict:
+    if not vals:
+        return {"name": name, "n": 0}
+    return {
+        "name": name,
+        "n": len(vals),
+        "mean_s": statistics.mean(vals),
+        "p50_s": _pct(vals, 0.5),
+        "p90_s": _pct(vals, 0.9),
+        "p99_s": _pct(vals, 0.99),
+        "max_s": max(vals),
+        "sum_s": sum(vals),
+    }
+
+
+def load_dst_log(dst_log_dir: Path) -> dict[str, dict]:
+    by_req: dict[str, dict] = {}
+    found_files = sorted(dst_log_dir.glob("dm_mig_pid*.jsonl"))
+    print(f"[analyze] dst log files: {len(found_files)} under {dst_log_dir}")
+    for f in found_files:
+        with f.open() as fh:
+            for line in fh:
+                try:
+                    rec = json.loads(line)
+                except Exception:
+                    continue
+                rid = rec.get("req_id")
+                if not rid:
+                    continue
+                key = _core_req_id(rid)
+                rec["_raw_req_id"] = rid
+                # If a req shows up twice (shouldn't, but be safe), prefer the
+                # one with t_first_token_unix populated.
+                prev = by_req.get(key)
+                if prev is None or (
+                    rec.get("t_first_token_unix") and
+                    not prev.get("t_first_token_unix")
+                ):
+                    by_req[key] = rec
+    print(f"[analyze] unique dst records: {len(by_req)}")
+    return by_req
+
+
+def load_proxy_breakdown(path: Path) -> list[dict]:
+    with path.open() as fh:
+        data = json.load(fh)
+    assert isinstance(data, list), f"unexpected breakdown.json shape: {type(data)}"
+    return data
+
+
+def decompose(proxy_recs: list[dict], dst_by_req: dict[str, dict]) -> list[dict]:
+    """Build per-migration breakdown rows by joining proxy + dst by req_id."""
+    rows: list[dict] = []
+    migrations = [x for x in proxy_recs if x.get("route_class") == "PD_SEP_V2"]
+    print(f"[analyze] proxy migrations: {len(migrations)} "
+          f"(of {len(proxy_recs)} total requests)")
+
+    miss_in_dst = 0
+    missing_phases = 0
+    for p in migrations:
+        rid = p.get("request_id")
+        dst = dst_by_req.get(rid)
+        if dst is None:
+            miss_in_dst += 1
+            continue
+        if dst.get("t_first_token_unix") is None:
+            missing_phases += 1
+            # still include the row but mark phases as NaN downstream
+        t_decode_sent = p.get("t_decode_sent_unix")
+        t_first_tok = p.get("t_first_token_unix")
+        t_arrival = dst.get("t_arrival_unix")
+        t_wait_kvs = dst.get("t_wait_for_kvs_unix")
+        t_kv_done = dst.get("t_kv_recv_done_unix")
+        t_first_sched = dst.get("t_first_scheduled_unix")
+        t_first_tok_dst = dst.get("t_first_token_unix")
+
+        def _diff(a, b):
+            if a is None or b is None:
+                return None
+            return float(a) - float(b)
+
+        rows.append({
+            "request_id": rid,
+            "session_id": p.get("session_id"),
+            "input_length": p.get("input_length"),
+            "v3_new_local": p.get("v3_new_local"),
+            "v3_target_idx": p.get("v3_target_idx") or p.get("v3_decode_target_idx"),
+            "arrival_n_running": (dst.get("arrival_state") or {}).get("n_running"),
+            "arrival_n_waiting": (dst.get("arrival_state") or {}).get("n_waiting"),
+            "arrival_pending_prefill_tok": (dst.get("arrival_state") or {}).get("pending_prefill_tok"),
+            "arrival_n_waiting_for_kvs": (dst.get("arrival_state") or {}).get("n_waiting_for_kvs"),
+            # Phase durations (seconds)
+            "T_proxy_total_dst_first_token_s": _diff(t_first_tok, t_decode_sent),
+            "T_relay_s": _diff(t_arrival, t_decode_sent),
+            "T_admission_pre_kv_s": _diff(t_wait_kvs, t_arrival),
+            "T_kv_pull_s": _diff(t_kv_done, t_wait_kvs),
+            "T_admission_post_kv_s": _diff(t_first_sched, t_kv_done),
+            "T_first_iter_s": _diff(t_first_tok_dst, t_first_sched),
+            # Raw timestamps for debugging
+            "t_decode_sent_unix": t_decode_sent,
+            "t_dst_arrival_unix": t_arrival,
+            "t_dst_wait_for_kvs_unix": t_wait_kvs,
+            "t_dst_kv_recv_done_unix": t_kv_done,
+            "t_dst_first_scheduled_unix": t_first_sched,
+            "t_dst_first_token_unix": t_first_tok_dst,
+            "t_proxy_first_token_unix": t_first_tok,
+        })
+
+    print(f"[analyze] missing in dst log: {miss_in_dst}")
+    print(f"[analyze] dst record incomplete (no t_first_token): {missing_phases}")
+    return rows
+
+
+def emit_summary(rows: list[dict]) -> None:
+    if not rows:
+        print("[analyze] no rows — nothing to summarize.")
+        return
+
+    phase_keys = [
+        "T_proxy_total_dst_first_token_s",
+        "T_relay_s",
+        "T_admission_pre_kv_s",
+        "T_kv_pull_s",
+        "T_admission_post_kv_s",
+        "T_first_iter_s",
+    ]
+
+    print()
+    print("=" * 88)
+    print(f"Migration dst-side phase breakdown  (n_migrations={len(rows)})")
+    print("=" * 88)
+    print(f"{'phase':<36} {'n':>4} {'mean(s)':>9} {'p50':>8} {'p90':>8} "
+          f"{'p99':>8} {'max':>8} {'sum(s)':>9}")
+    print("-" * 88)
+    for k in phase_keys:
+        vals = [r[k] for r in rows if r.get(k) is not None]
+        if not vals:
+            print(f"{k:<36} {'n/a':>4}")
+            continue
+        s = _summary(k, vals)
+        print(f"{k:<36} {s['n']:>4} {s['mean_s']:>9.3f} {s['p50_s']:>8.3f} "
+              f"{s['p90_s']:>8.3f} {s['p99_s']:>8.3f} {s['max_s']:>8.3f} "
+              f"{s['sum_s']:>9.2f}")
+
+    print()
+    print("Aggregate attribution (sum across all migrations):")
+    sums = {}
+    for k in ("T_relay_s", "T_admission_pre_kv_s", "T_kv_pull_s",
+              "T_admission_post_kv_s", "T_first_iter_s"):
+        sums[k] = sum(r[k] for r in rows if r.get(k) is not None)
+    total = sum(sums.values())
+    total_proxy = sum(r["T_proxy_total_dst_first_token_s"] for r in rows
+                      if r.get("T_proxy_total_dst_first_token_s") is not None)
+    print(f"  decomposed sum   : {total:>8.2f} s")
+    print(f"  proxy total sum  : {total_proxy:>8.2f} s   "
+          f"(should be ~equal; gap = uninstrumented)")
+    if total > 0:
+        for k, v in sums.items():
+            print(f"    {k:<28} {v:>8.2f} s  ({v/total*100:5.1f} %)")
+
+    # Headline: "How much could layerwise save?"
+    layerwise_addressable = sums.get("T_kv_pull_s", 0.0)
+    queue_residual = sum(v for k, v in sums.items() if k != "T_kv_pull_s")
+    print()
+    print("Layerwise-addressable vs queue-residual:")
+    print(f"  T_kv_pull_s (addressable by layerwise)   : {layerwise_addressable:>8.2f} s "
+          f"({layerwise_addressable / total * 100 if total else 0:5.1f} %)")
+    print(f"  everything else (queue/admission/iter)   : {queue_residual:>8.2f} s "
+          f"({queue_residual / total * 100 if total else 0:5.1f} %)")
+
+
+def write_csv(rows: list[dict], path: Path) -> None:
+    import csv
+    if not rows:
+        path.write_text("")
+        return
+    fields = list(rows[0].keys())
+    with path.open("w", newline="") as fh:
+        w = csv.DictWriter(fh, fieldnames=fields)
+        w.writeheader()
+        w.writerows(rows)
+    print(f"[analyze] wrote CSV: {path}  (n={len(rows)})")
+
+
+def maybe_plot(rows: list[dict], out_path: Path) -> None:
+    try:
+        import matplotlib
+        matplotlib.use("Agg")
+        import matplotlib.pyplot as plt
+    except Exception as e:
+        print(f"[analyze] matplotlib unavailable ({e}); skipping plot.")
+        return
+
+    if not rows:
+        return
+
+    rows_sorted = sorted(
+        rows,
+        key=lambda r: r.get("T_proxy_total_dst_first_token_s") or 0.0,
+    )
+    n = len(rows_sorted)
+    idx = list(range(n))
+
+    def col(k):
+        return [(r.get(k) or 0.0) for r in rows_sorted]
+
+    relay = col("T_relay_s")
+    pre = col("T_admission_pre_kv_s")
+    pull = col("T_kv_pull_s")
+    post = col("T_admission_post_kv_s")
+    first_iter = col("T_first_iter_s")
+
+    fig, ax = plt.subplots(figsize=(11, 5))
+    bot = [0.0] * n
+    for vals, label, color in [
+        (relay, "HTTP relay", "#cccccc"),
+        (pre, "admission pre-KV", "#f4a261"),
+        (pull, "KV pull (layerwise-addressable)", "#e76f51"),
+        (post, "admission post-KV", "#2a9d8f"),
+        (first_iter, "first decode iter", "#264653"),
+    ]:
+        ax.bar(idx, vals, bottom=bot, color=color, label=label, width=0.85)
+        bot = [b + v for b, v in zip(bot, vals)]
+
+    ax.set_xticks(idx)
+    ax.set_xticklabels([str(i + 1) for i in idx], rotation=0, fontsize=8)
+    ax.set_xlabel("Migrated request (sorted by total dst wait, ascending)")
+    ax.set_ylabel("Time (s)")
+    ax.set_title("Per-migration dst-side phase breakdown (v3 unified_v3 run)")
+    ax.legend(loc="upper left", fontsize=9)
+    ax.grid(axis="y", linestyle=":", alpha=0.5)
+    fig.tight_layout()
+    fig.savefig(out_path, dpi=120)
+    plt.close(fig)
+    print(f"[analyze] wrote plot: {out_path}")
+
+
+def main() -> None:
+    p = argparse.ArgumentParser()
+    p.add_argument("--proxy-breakdown", type=Path, required=True)
+    p.add_argument("--dst-log-dir", type=Path, required=True)
+    p.add_argument("--output", type=Path, default=None,
+                   help="CSV path (default: <run>/dst_migration_breakdown.csv)")
+    p.add_argument("--plot", type=Path, default=None,
+                   help="PNG path (default: <run>/dst_migration_breakdown.png)")
+    args = p.parse_args()
+
+    if not args.proxy_breakdown.is_file():
+        sys.exit(f"missing proxy breakdown: {args.proxy_breakdown}")
+    if not args.dst_log_dir.is_dir():
+        sys.exit(f"missing dst log dir: {args.dst_log_dir}")
+
+    run_dir = args.proxy_breakdown.parent
+    out_csv = args.output or (run_dir / "dst_migration_breakdown.csv")
+    out_png = args.plot or (run_dir / "dst_migration_breakdown.png")
+
+    proxy_recs = load_proxy_breakdown(args.proxy_breakdown)
+    dst_by_req = load_dst_log(args.dst_log_dir)
+    rows = decompose(proxy_recs, dst_by_req)
+    emit_summary(rows)
+    write_csv(rows, out_csv)
+    maybe_plot(rows, out_png)
+
+
+if __name__ == "__main__":
+    main()

microbench/connector_tax/cache_sweep/analyze_migration_log.py

@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+"""Per-migration log + per-instance summary for a v3 trace replay.
+
+Reads <run_dir>/breakdown.json and <run_dir>/metrics.jsonl and emits:
+  1. A row per migration showing src→dst, per-side state snapshots, and
+     the resulting TTFT.
+  2. Histograms: migrations received per inst, sent per inst, all
+     (src→dst) pairs.
+  3. Post-rotation tail: how many turns of migrated sessions ended up on
+     each inst (downstream impact of rotation).
+  4. Anti-hotspot signal: recent_mig_received_in_window at decision time.
+
+Run any v3 replay through this to spot pathological clustering of
+migrations on the same dst within a short window.
+
+Usage:
+    python analyze_migration_log.py <RUN_DIR>
+where <RUN_DIR> contains breakdown.json + metrics.jsonl (i.e. the proxy's
+per-policy output folder, e.g. .../b3_v3_20260527_1344/unified_v3).
+"""
+import json
+import sys
+from collections import Counter, defaultdict
+from pathlib import Path
+
+
+def main(run_dir: Path) -> None:
+    bd = json.load(open(run_dir / "breakdown.json"))
+    m = {json.loads(l)["request_id"]: json.loads(l)
+         for l in open(run_dir / "metrics.jsonl")}
+
+    mig = [e for e in bd if e.get("v3_migrate")]
+    mig.sort(key=lambda x: x.get("t_decision_unix", 0))
+
+    print(f"=== {len(mig)} migrations in {run_dir.name} ===\n")
+    cols = (
+        "#", "t_rel", "session", "turn",
+        "src", "dst", "src_nreq", "src_dec_tok",
+        "dst_nreq", "dst_cache", "dst_recent_recv",
+        "inlen", "self_ttft_ms",
+    )
+    print("  " + " ".join(f"{c:>13}" for c in cols))
+    print("-" * (15 * len(cols)))
+
+    t0 = mig[0]["t_decision_unix"] if mig else 0
+    for i, e in enumerate(mig):
+        rid = e["request_id"]
+        src_idx = e.get("v3_src_idx", e["chosen_idx"])
+        dst_idx = e.get("v3_target_idx", -1)
+        src_state = e.get("v3_src_state") or {}
+        dst_state = e.get("v3_target_state") or {}
+        cands = {c["idx"]: c for c in e.get("candidate_scores", [])}
+        # Fall back to candidate_scores if dedicated v3_*_state fields aren't present.
+        src_nreq = src_state.get("num_requests", cands.get(src_idx, {}).get("num_requests", "-"))
+        src_dec_tok = src_state.get("ongoing_decode_tokens",
+                                    cands.get(src_idx, {}).get("ongoing_decode_tokens", "-"))
+        dst_nreq = dst_state.get("num_requests", cands.get(dst_idx, {}).get("num_requests", "-"))
+        dst_cache = e.get("v3_target_cache_hit", dst_state.get("cache_hit_estimate", 0))
+        dst_recent = e.get("v3_target_recent_received",
+                            dst_state.get("recent_mig_received_in_window", "-"))
+        inlen = e.get("input_length") or m.get(rid, {}).get("input_length", 0)
+        ttft = m.get(rid, {}).get("ttft_s") or 0
+        t_rel = e["t_decision_unix"] - t0
+        turn = m.get(rid, {}).get("turn_id", "?")
+        print(
+            f"  {i+1:>13} {t_rel:>13.1f} {e['session_id']:>13} {turn:>13} "
+            f"{src_idx:>13} {dst_idx:>13} {src_nreq:>13} {src_dec_tok:>13} "
+            f"{dst_nreq:>13} {dst_cache:>13} {dst_recent:>13} "
+            f"{inlen:>13} {ttft*1000:>13.0f}"
+        )
+
+    # Aggregate counts
+    print("\n=== Migrations TO each instance ===")
+    to_count = Counter(e.get("v3_target_idx", -1) for e in mig)
+    for idx in range(8):
+        print(f"  inst_{idx}: {to_count.get(idx, 0)} migrations received")
+
+    print("\n=== Migrations FROM each instance ===")
+    from_count = Counter(e.get("v3_src_idx", e["chosen_idx"]) for e in mig)
+    for idx in range(8):
+        print(f"  inst_{idx}: {from_count.get(idx, 0)} migrations sent")
+
+    print("\n=== Migration pairs (src→dst, count) ===")
+    pair_count = Counter(
+        (e.get("v3_src_idx", e["chosen_idx"]), e.get("v3_target_idx", -1))
+        for e in mig
+    )
+    for (s, d), n in sorted(pair_count.items(), key=lambda x: -x[1]):
+        print(f"  {s} → {d}: {n}")
+
+    print("\n=== Sessions migrating multiple times ===")
+    sess_mig = defaultdict(list)
+    for e in mig:
+        sess_mig[e["session_id"]].append(
+            (e.get("t_decision_unix", 0),
+             e.get("v3_src_idx", e["chosen_idx"]),
+             e.get("v3_target_idx", -1))
+        )
+    multi = {s: ev for s, ev in sess_mig.items() if len(ev) > 1}
+    if not multi:
+        print("  (none)")
+    for sess, events in sorted(multi.items()):
+        chain = " → ".join(f"{s}->{d}" for _, s, d in sorted(events))
+        print(f"  session {sess}: {chain}")
+
+    # Recent-received hotspot signal — non-zero values mean the picker
+    # accepted a target that recently got another migration.
+    print("\n=== Anti-hotspot signal: dst.recent_mig_received_in_window ===")
+    rec = [e.get("v3_target_recent_received", 0) for e in mig]
+    if rec:
+        nonzero = [v for v in rec if v]
+        print(f"  total migrations: {len(rec)}, "
+              f"with recent_received > 0: {len(nonzero)}, "
+              f"max recent_received: {max(rec)}")
+
+    # Post-rotation tail: turns of migrated sessions after their LAST mig
+    print("\n=== Post-rotation tail per inst (turns of migrated sessions after last mig) ===")
+    tail = Counter()
+    for sess, events in sess_mig.items():
+        final_dst = sorted(events)[-1][2]
+        last_t = max(t for t, _, _ in events)
+        sess_turns = [mm for rid, mm in m.items() if mm["session_id"] == sess]
+        tail[final_dst] += sum(1 for mm in sess_turns
+                               if mm.get("t_dispatch_unix", 0) > last_t)
+    for idx in range(8):
+        print(f"  inst_{idx}: {tail.get(idx, 0)} tail turns")
+
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        print("usage: analyze_migration_log.py <run_dir>", file=sys.stderr)
+        sys.exit(1)
+    main(Path(sys.argv[1]))

microbench/connector_tax/cache_sweep/analyze_transfer_decomp.py

@@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+"""Decompose migration KV-transfer time into RDMA-actual vs control-plane.
+
+Joins three logs from an instrumented unified_v3 run:
+
+  proxy breakdown.json        — per-request route + phase timestamps
+  dst_mig_log/dm_mig_pid*.jsonl  — dst lifecycle (instrument_dst_migration.py)
+                                   gives T_kv_pull = wait_for_kvs -> recv_done
+  mooncake xfer/mb2_transfer_pid*.jsonl — connector internals
+                                   (instrument_mooncake.py):
+      send_blocks            : pure RDMA (total_bytes, duration_s)  [producer]
+      receive_kv_enter/finish: consumer-observed transfer window    [consumer]
+      ready_wait             : producer wait for src KV commit       [producer]
+      send_kv_to_decode_enter: producer received the pull request    [producer]
+
+Decisive question: of the 87% dst-side overhead that is T_kv_pull, how
+much is the actual RDMA write (`send_blocks`) vs control-plane
+(handshake / ready-wait / GIL starvation on the busy src)?
+
+  - send_blocks bandwidth ~ wire (10 GB/s) AND << T_kv_pull
+        => loss is control-plane; layerwise (which only moves WHEN the
+           RDMA fires) will NOT fix it.
+  - send_blocks bandwidth << wire
+        => the RDMA write itself is slow (NIC / src-side servicing);
+           characterize with a load microbench next.
+
+Usage:
+  python analyze_transfer_decomp.py \
+      --proxy-breakdown <RUN>/unified_v3/breakdown.json \
+      --dst-log-dir     <RUN>/dst_mig_log \
+      --xfer-log-dir    <RUN>/xfer_log
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import re
+import statistics
+import sys
+from pathlib import Path
+
+
+def _core_req_id(rid: str) -> str:
+    if not rid:
+        return rid
+    s = rid
+    if s.startswith("cmpl-"):
+        s = s[len("cmpl-"):]
+    m = re.match(r"^(.*)-\d+-[0-9a-fA-F]+$", s)
+    if m:
+        s = m.group(1)
+    return s
+
+
+def _pct(vals, q):
+    if not vals:
+        return float("nan")
+    vs = sorted(vals)
+    i = max(0, min(len(vs) - 1, int(math.ceil(q * len(vs))) - 1))
+    return vs[i]
+
+
+def _stat_line(name, vals, unit="s"):
+    if not vals:
+        print(f"{name:<34} n=0")
+        return
+    print(f"{name:<34} n={len(vals):>3}  mean={statistics.mean(vals):>8.3f}  "
+          f"p50={_pct(vals,0.5):>8.3f}  p90={_pct(vals,0.9):>8.3f}  "
+          f"max={max(vals):>8.3f}  sum={sum(vals):>8.2f} {unit}")
+
+
+def load_events(xfer_dir: Path):
+    files = sorted(xfer_dir.glob("mb2_transfer_pid*.jsonl"))
+    print(f"[xfer] log files: {len(files)} under {xfer_dir}")
+    send_blocks, recv_enter, recv_finish, ready_wait, send_enter = [], [], [], [], []
+    for f in files:
+        pid = f.stem.replace("mb2_transfer_pid", "")
+        with f.open() as fh:
+            for line in fh:
+                try:
+                    e = json.loads(line)
+                except Exception:
+                    continue
+                e["_pid"] = pid
+                ev = e.get("event")
+                if ev == "send_blocks":
+                    send_blocks.append(e)
+                elif ev == "receive_kv_enter":
+                    recv_enter.append(e)
+                elif ev == "receive_kv_finish":
+                    recv_finish.append(e)
+                elif ev == "ready_wait":
+                    ready_wait.append(e)
+                elif ev == "send_kv_to_decode_enter":
+                    send_enter.append(e)
+    print(f"[xfer] events: send_blocks={len(send_blocks)} "
+          f"recv_enter={len(recv_enter)} recv_finish={len(recv_finish)} "
+          f"ready_wait={len(ready_wait)} send_enter={len(send_enter)}")
+    return send_blocks, recv_enter, recv_finish, ready_wait, send_enter
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--proxy-breakdown", type=Path, required=True)
+    p.add_argument("--dst-log-dir", type=Path, required=True)
+    p.add_argument("--xfer-log-dir", type=Path, required=True)
+    args = p.parse_args()
+
+    for pth in (args.proxy_breakdown, args.dst_log_dir, args.xfer_log_dir):
+        if not pth.exists():
+            sys.exit(f"missing: {pth}")
+
+    proxy = json.load(args.proxy_breakdown.open())
+    migrations = [x for x in proxy if x.get("route_class") == "PD_SEP_V2"]
+    mig_ids = {x.get("request_id") for x in migrations}
+    print(f"[proxy] migrations: {len(migrations)} / {len(proxy)} total")
+
+    # dst lifecycle: T_kv_pull per migration (core req id)
+    dst_pull = {}
+    for f in sorted(args.dst_log_dir.glob("dm_mig_pid*.jsonl")):
+        for line in f.open():
+            try:
+                r = json.loads(line)
+            except Exception:
+                continue
+            tw = r.get("t_wait_for_kvs_unix")
+            td = r.get("t_kv_recv_done_unix")
+            if tw and td:
+                dst_pull[_core_req_id(r.get("req_id"))] = td - tw
+
+    sb, re_enter, re_finish, rw, se = load_events(args.xfer_log_dir)
+
+    # ---- 1. Pure RDMA bandwidth from send_blocks (the decisive number) ----
+    print("\n" + "=" * 90)
+    print("1. PURE RDMA WRITE rate (`send_blocks` = batch_transfer_sync_write)")
+    print("=" * 90)
+    bws, durs, bytes_l = [], [], []
+    for e in sb:
+        b = e.get("total_bytes", 0)
+        d = e.get("duration_s", 0)
+        if d and d > 0 and b > 0:
+            bws.append(b / 1e9 / d)
+            durs.append(d)
+            bytes_l.append(b)
+    if bws:
+        tot_b = sum(bytes_l)
+        tot_d = sum(durs)
+        print(f"  send_blocks calls: {len(bws)}")
+        print(f"  total bytes moved : {tot_b/2**30:.2f} GiB")
+        print(f"  total RDMA time   : {tot_d:.2f} s")
+        print(f"  AGGREGATE rate    : {tot_b/1e9/tot_d:.2f} GB/s  "
+              f"(MB2 idle-src steady-state = ~9.7-10 GB/s)")
+        _stat_line("  per-call rate (GB/s)", bws, unit="GB/s")
+        _stat_line("  per-call duration", durs)
+        # bandwidth vs size — small ops are latency-bound
+        print("\n  rate vs transfer size:")
+        pairs = sorted(zip(bytes_l, bws))
+        for b, w in pairs:
+            bar = "#" * int(min(40, w * 4))
+            print(f"    {b/2**20:>8.1f} MiB  {w:>6.2f} GB/s  {bar}")
+    else:
+        print("  no send_blocks events with positive duration")
+
+    # ---- 2. Producer ready-wait (src KV commit) ----
+    print("\n" + "=" * 90)
+    print("2. PRODUCER ready-wait (src KV not yet committed when pull arrived)")
+    print("=" * 90)
+    rw_vals = [e.get("ready_wait_s", 0) for e in rw if e.get("ready_wait_s") is not None]
+    already = sum(1 for e in rw if e.get("ready_already_set"))
+    _stat_line("  ready_wait", rw_vals)
+    print(f"  ready_already_set at entry: {already}/{len(rw)} "
+          f"(if most are True, src commit is not the bottleneck)")
+
+    # ---- 3. Consumer-observed receive_kv window ----
+    print("\n" + "=" * 90)
+    print("3. CONSUMER receive_kv window (enter->FINISH, ~most of T_kv_pull)")
+    print("=" * 90)
+    rf_vals = [e.get("duration_s", 0) for e in re_finish if e.get("duration_s")]
+    _stat_line("  receive_kv duration", rf_vals)
+
+    # ---- 4. Per-migration join: T_kv_pull vs receive_kv vs ready_wait ----
+    print("\n" + "=" * 90)
+    print("4. PER-MIGRATION join (T_kv_pull from dst vs connector internals)")
+    print("=" * 90)
+    # index connector events by core req id
+    rf_by_req = {}
+    for e in re_finish:
+        for rid in e.get("req_ids", []):
+            rf_by_req[_core_req_id(rid)] = e.get("duration_s")
+    rw_by_req = {}
+    for e in rw:
+        rw_by_req[_core_req_id(e.get("d_req_id", ""))] = e.get("ready_wait_s")
+
+    joined = 0
+    sum_pull = sum_recv = sum_rw = 0.0
+    rows = []
+    for m in migrations:
+        core = m.get("request_id")
+        pull = dst_pull.get(core)
+        recv = rf_by_req.get(core)
+        rwv = rw_by_req.get(core)
+        if pull is None and recv is None:
+            continue
+        joined += 1
+        if pull: sum_pull += pull
+        if recv: sum_recv += recv
+        if rwv: sum_rw += rwv
+        rows.append((core, m.get("input_length"), m.get("v3_target_cache_hit"),
+                     pull, recv, rwv))
+    print(f"  joined migrations: {joined}")
+    print(f"  Σ T_kv_pull (dst)        = {sum_pull:8.2f} s")
+    print(f"  Σ receive_kv (consumer)  = {sum_recv:8.2f} s")
+    print(f"  Σ ready_wait (producer)  = {sum_rw:8.2f} s")
+    # The RDMA share: best-effort total send_blocks time
+    sum_rdma = sum(durs) if durs else 0.0
+    print(f"  Σ send_blocks RDMA       = {sum_rdma:8.2f} s   (all transfers, "
+          f"not just migrations)")
+    if sum_pull > 0:
+        print(f"\n  RDMA-actual / T_kv_pull   ≈ {sum_rdma/sum_pull*100:5.1f} %")
+        print(f"  ready-wait / T_kv_pull    ≈ {sum_rw/sum_pull*100:5.1f} %")
+        resid = sum_pull - sum_rdma - sum_rw
+        print(f"  control-plane residual    ≈ {resid/sum_pull*100:5.1f} %  "
+              f"(handshake / ZMQ / GIL starvation)")
+
+    print("\n  per-migration detail:")
+    print(f"  {'req_id':<22} {'in_len':>7} {'dst_hit':>8} {'kv_pull':>8} "
+          f"{'recv_kv':>8} {'rdy_wait':>8}")
+    for core, il, hit, pull, recv, rwv in sorted(
+            rows, key=lambda r: -(r[3] or 0)):
+        def s(v): return f"{v:.2f}" if v is not None else "  --"
+        print(f"  {core:<22} {str(il):>7} {str(hit):>8} {s(pull):>8} "
+              f"{s(recv):>8} {s(rwv):>8}")
+
+
+if __name__ == "__main__":
+    main()

microbench/connector_tax/cache_sweep/run_5policy_600s.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# 5-policy comparison on the first-600s trace, fresh vLLM (cold APC) each arm.
+#   leastwork    LPWL (parameter-free)
+#   unified_ab   unified hybrid, A+B' tuned (of=1.3, lmw=0.01)
+#   unified_def  unified hybrid, defaults  (of=2.0, lmw=0.0)
+#   lmetric      P_tokens x BS, no affinity
+#   sticky       hard session affinity
+set -uo pipefail
+export ROOT=/home/admin/cpfs/wjh/agentic-kv
+export MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct
+TRACE="${TRACE:-$ROOT/traces/w600_r0.0015_st30_first600s.jsonl}"
+DATE="$(date +%Y%m%d_%H%M)"
+OUT="${OUTROOT:-$ROOT/outputs/policy5_600s_$DATE}"
+mkdir -p "$OUT"
+echo "OUTROOT=$OUT" | tee "$OUT/RUNINFO.txt"
+echo "TRACE=$TRACE  ($(wc -l < "$TRACE") reqs)" | tee -a "$OUT/RUNINFO.txt"
+date | tee -a "$OUT/RUNINFO.txt"
+
+run_arm() {  # name policy extra_args
+    local name="$1" policy="$2" extra="$3"
+    echo "===== ARM: $name (policy=$policy args='$extra') =====" | tee -a "$OUT/RUNINFO.txt"
+    local t0=$(date +%s)
+    EXTRA_PROXY_ARGS="$extra" bash "$ROOT/scripts/b3_isolated_policy.sh" \
+        "$policy" "$TRACE" "$OUT/$name" > "$OUT/$name.log" 2>&1
+    echo "$name rc=$? rows=$(wc -l < "$OUT/$name/metrics.jsonl" 2>/dev/null) dur=$(( $(date +%s) - t0 ))s" | tee -a "$OUT/RUNINFO.txt"
+}
+
+# Headline contrasts first (so a late failure still leaves the key arms).
+run_arm leastwork   leastwork ""
+run_arm unified_ab  unified   "--overload-factor 1.3 --lmetric-decode-weight 0.01"
+run_arm unified_def unified   "--overload-factor 2.0 --lmetric-decode-weight 0.0"
+run_arm lmetric     lmetric   ""
+run_arm sticky      sticky    ""
+
+echo "===== ALL DONE: $OUT =====" | tee -a "$OUT/RUNINFO.txt"
+date | tee -a "$OUT/RUNINFO.txt"

microbench/connector_tax/cache_sweep/run_5policy_both_modes.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+# 5-policy comparison in BOTH dispatch modes on the SAME ttp-annotated trace,
+# so the only variable is dispatch-mode (tracets vs thinktime). Fresh vLLM
+# (cold APC) per arm via run_5policy_600s.sh -> b3_isolated_policy.sh.
+set -uo pipefail
+ROOT="${ROOT:-/home/admin/cpfs/wjh/agentic-kv}"
+TRACE_FILE="${TRACE_FILE:-$ROOT/traces/w600_r0.0015_st30_first600s_ttp.jsonl}"
+RUN5="$ROOT/microbench/connector_tax/cache_sweep/run_5policy_600s.sh"
+DATE="$(date +%Y%m%d_%H%M)"
+
+echo "=== 5policy x {tracets,thinktime} | trace=$(basename "$TRACE_FILE") | $DATE ==="
+for MODE in tracets thinktime; do
+    OUT="$ROOT/outputs/policy5_600s_${MODE}_${DATE}"
+    echo "############ MODE=$MODE  OUT=$OUT  $(date) ############"
+    TRACE="$TRACE_FILE" REPLAY_DISPATCH_MODE="$MODE" OUTROOT="$OUT" \
+        bash "$RUN5"
+    echo "dispatch_mode=$MODE" >> "$OUT/RUNINFO.txt"
+    echo "trace=$TRACE_FILE" >> "$OUT/RUNINFO.txt"
+done
+echo "=== ALL DONE (both modes) $(date) ==="

microbench/connector_tax/cache_sweep/run_b3_replay.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# B3 routing-policy reproducibility re-test.
+#
+# Re-runs the 5 routing policies from fig_b3_latency_bars.png on the same
+# trace, in a single same-day session, to check whether the ordering
+# (unified < load_only < sticky etc.) still holds today.
+#
+# Policies (in run order):
+#   lmetric        plain — cache-aware P_tokens × BS
+#   load_only      plain — pure min-num_requests
+#   sticky         plain — hard session affinity
+#   unified        plain — hybrid affinity + LMetric fallback
+#   unified_v2     Mooncake kv_both + selective PD-sep (with DR-fix applied)
+#
+# unified_v2 is run with VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC=1 so we
+# get the "best Mooncake state" we have today (DR-fix on top of the
+# already-fixed mainline after e3a1d70 etc.). The other 4 policies don't
+# load any connector so the patch is irrelevant.
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+TRACE="${TRACE:-$PROJ_DIR/traces/w600_r0.0015_st30.jsonl}"
+DATE="$(date +%Y%m%d_%H%M)"
+OUTROOT="${OUTROOT:-$PROJ_DIR/outputs/b3_replay_${DATE}}"
+PYTHON="$PROJ_DIR/.venv/bin/python"
+DR_FIX_SCRIPT="$PROJ_DIR/microbench/connector_tax/cache_sweep/apply_direct_read_fix.py"
+VLLM_ROOT="${VLLM_ROOT:-$PROJ_DIR/.venv/lib/python3.12/site-packages/vllm}"
+
+mkdir -p "$OUTROOT"
+echo "=== B3 5-policy re-test ==="
+echo "Trace : $TRACE"
+echo "Out   : $OUTROOT"
+echo "Order : lmetric → load_only → sticky → unified → unified_v2 (DR-fix on)"
+echo ""
+
+cleanup_all() {
+    pkill -9 -f cache_aware_proxy 2>/dev/null || true
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 5
+    "$PYTHON" "$DR_FIX_SCRIPT" --revert --vllm-root "$VLLM_ROOT" 2>/dev/null || true
+}
+trap cleanup_all EXIT
+cleanup_all
+
+# Apply DR-fix once — it's env-gated so only unified_v2 (with env=1) sees it
+echo "[stage 0] applying CT_DR_FIX (env-gated, only activates when VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC=1)"
+"$PYTHON" "$DR_FIX_SCRIPT" --apply --vllm-root "$VLLM_ROOT"
+
+run_policy() {
+    local policy="$1"
+    local skip_dr="$2"
+    local rundir="$OUTROOT/$policy"
+    mkdir -p "$rundir"
+
+    echo ""
+    echo "====== $policy ; DR_SYNC_DISABLED=$skip_dr ======"
+
+    if [ "$skip_dr" = "1" ]; then
+        export VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC=1
+    else
+        unset VLLM_MOONCAKE_DISABLE_DIRECT_READ_SYNC
+    fi
+
+    bash "$PROJ_DIR/scripts/b3_isolated_policy.sh" "$policy" "$TRACE" "$rundir" \
+        2>&1 | tee "$rundir/orchestrator.log" | tail -30
+    rc="${PIPESTATUS[0]}"
+    if [ "$rc" != "0" ]; then
+        echo "[FAIL] policy $policy rc=$rc"
+    fi
+    # Belt-and-braces cleanup between policies
+    pkill -9 -f cache_aware_proxy 2>/dev/null || true
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 10
+    return 0
+}
+
+run_policy "lmetric"    "0"
+run_policy "load_only"  "0"
+run_policy "sticky"     "0"
+run_policy "unified"    "0"
+run_policy "unified_v2" "1"  # uses Mooncake kv_both; activate DR-fix
+
+echo ""
+echo "[stage Z] reverting CT_DR_FIX"
+"$PYTHON" "$DR_FIX_SCRIPT" --revert --vllm-root "$VLLM_ROOT"
+
+echo ""
+echo "Done. Artifacts: $OUTROOT"
+for p in lmetric load_only sticky unified unified_v2; do
+    echo "  $p: $OUTROOT/$p/metrics.jsonl"
+done

microbench/connector_tax/cache_sweep/run_smoke_nixl.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+# Smoke test for Nixl-based PD-sep migration (NVLink intra-node via UCX).
+#
+# Drops 2 vLLM kv_both NixlConnector instances on GPU 0,1 and runs
+# smoke_test_migrate_cache.py against them with the kv_transfer_params
+# format Nixl expects (only do_remote_decode on src; proxy must forward
+# kv_transfer_params from src's response to dst).
+#
+# Since smoke_test_migrate_cache.py is currently hard-coded for Mooncake
+# (transfer_id + remote_bootstrap_addr), we use a tiny Python in-line
+# variant here that does the Nixl response-forward handshake directly.
+
+set -uo pipefail
+
+PROJ_DIR="${PROJ_DIR:-/home/admin/cpfs/wjh/agentic-kv}"
+MODEL="${MODEL:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
+VENV="$PROJ_DIR/.venv/bin"
+LOGS_DIR="${LOGS_DIR:-$PROJ_DIR/outputs/smoke_nixl_$(date +%Y%m%d_%H%M%S)}"
+mkdir -p "$LOGS_DIR"
+
+cleanup() {
+    echo "[smoke-nixl] cleaning up vLLM..."
+    pkill -9 -f "vllm serve" 2>/dev/null || true
+    pkill -9 -f "EngineCore" 2>/dev/null || true
+    sleep 2
+}
+trap cleanup EXIT
+cleanup
+
+echo "[smoke-nixl] starting 2 vLLM kv_both NixlConnector on GPU 0,1"
+for i in 0 1; do
+    port=$((8000 + i))
+    nixl_port=$((5600 + i))
+    master=$((29500 + i))
+    PYTHONHASHSEED=42 \
+    VLLM_NIXL_SIDE_CHANNEL_PORT=$nixl_port \
+    CUDA_VISIBLE_DEVICES=$i \
+    MASTER_PORT=$master \
+    nohup "$VENV/vllm" serve "$MODEL" \
+        --host 0.0.0.0 --port "$port" \
+        --tensor-parallel-size 1 \
+        --trust-remote-code --enable-prefix-caching \
+        --dtype auto --gpu-memory-utilization 0.9 \
+        --max-model-len 200000 \
+        --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' \
+        --enable-prompt-tokens-details \
+        > "$LOGS_DIR/vllm_inst_${i}_gpu${i}.log" 2>&1 &
+    disown
+    sleep 2
+done
+
+echo "[smoke-nixl] waiting for health on 8000 and 8001 ..."
+for port in 8000 8001; do
+    tries=0
+    while ! curl -sf "http://127.0.0.1:$port/health" >/dev/null 2>&1; do
+        tries=$((tries+1))
+        if [ $tries -gt 240 ]; then
+            echo "[smoke-nixl] FATAL: $port not ready"; exit 1
+        fi
+        sleep 2
+    done
+    echo "  port=$port ready"
+done
+
+echo "[smoke-nixl] running smoke_nixl_migrate.py"
+"$VENV/python" "$PROJ_DIR/microbench/connector_tax/cache_sweep/smoke_nixl_migrate.py" \
+    --src-port 8000 --dst-port 8001 \
+    ${EXTRA_SMOKE_ARGS:-} \
+    2>&1 | tee "$LOGS_DIR/smoke_output.log"
+
+ec=${PIPESTATUS[0]}
+echo "[smoke-nixl] test exit=$ec, logs at $LOGS_DIR"
+exit $ec