Compare commits
1 Commits
main
...
4d669ae2d8
| Author | SHA1 | Date | |
|---|---|---|---|
| 4d669ae2d8 |
9
.gitignore
vendored
9
.gitignore
vendored
@@ -3,14 +3,7 @@ __pycache__/
|
||||
.venv/
|
||||
*.egg-info/
|
||||
outputs/
|
||||
traces/*
|
||||
# ship the anonymized sampled trace + its provenance (metadata only, no cleartext)
|
||||
!traces/w600_r0.0015_st30.jsonl
|
||||
!traces/README.md
|
||||
traces/
|
||||
*.log
|
||||
.claude/
|
||||
# third_party/vllm tracked in git for patch management
|
||||
!traces/w600_r0.0015_st30_first600s.jsonl
|
||||
# + time_to_parent_chat annotation (for --dispatch-mode thinktime); same anon data
|
||||
!traces/w600_r0.0015_st30_ttp.jsonl
|
||||
!traces/w600_r0.0015_st30_first600s_ttp.jsonl
|
||||
|
||||
65
MEETING.md
65
MEETING.md
@@ -6,26 +6,18 @@
|
||||
|
||||
## 1. 关键洞察：Dispatch Coupling
|
||||
|
||||
每个 turn 间有一段外部 gap `T_external`（chatbot 是人类读+想+打字；agentic 是 tool 执行）。**Little's Law `L = Λ · N · (W_turn + T_external)`** 在两种 workload 下都成立 —— 差异在于 `T_external` 的分布相对于 `W_turn` 的位置：
|
||||
- `T_external ≫ W_turn` → 开环 regime：scheduler 退一步不动 L
|
||||
- `T_external ≲ W_turn` → 闭环 regime：`W_turn(L)` 因 KV 竞争耦合到 L，反馈环把 scheduler 的 ε 退步放大几倍
|
||||
Chatbot：turn 间有人类 think-time，系统快慢 ⊥ 下一 turn 到达率。
|
||||
Agentic：turn 间只有 tool-call 返回 (≈0)，**系统跑慢 → session 停留长 → 并发多 → KV pool 紧 → 更慢**。
|
||||
|
||||
**Production trace 实测 `T_external` 分布**（next.start − prev.end，formatted session 链作 ground truth）：
|
||||
Little's Law 隐式方程：
|
||||
|
||||
|
||||
```
|
||||
L = Λ · N · W_turn(L) # agentic, T_human≈0
|
||||
```
|
||||
|
||||
| | Agentic | Chatbot |
|
||||
|---|---:|---:|
|
||||
| p50 | **1.6s** | **7.2s** |
|
||||
| gap < 1s | **39%** | 4% |
|
||||
| gap < 5s | 67% | 29% |
|
||||
| p99 | 738s | 43s |
|
||||
小扰动分析：amplification = `1 / (1 − Λ·N·W'(L*))`，系统接近 KV 饱和时发散。
|
||||
|
||||
两个分布形状完全不同：chatbot unimodal 集中在 5–15s（人类节奏）；agentic bimodal —— **39% 的 gap 在 sub-second 里（autonomous tool-call mode）**，外加 13% > 30s 的长尾。**Agentic 的 sub-second mass 是 chatbot 没有的**，正是 dispatch coupling 激活的来源。
|
||||
|
||||
**实测 regime**：在 unified（TTFT p90 = 7.3s）下，**73% 的 agentic turn 把系统推进闭环**（W_turn > T_external），chatbot 仅 32%。在 lmetric（15.7s）下 agentic 80%、**chatbot 也到 88%** —— lmetric 把 chatbot 自己也拖进闭环，这就是它在两种 workload 都 underperform 的根因。
|
||||
|
||||
**结果**：lmetric 跑 600s trace 用 49 min wall-clock = **8x amplification**。**per-turn W 的小差异被放大成 wall-clock 数量级差距** —— locality 不是 nice-to-have，是 dominant lever。
|
||||
**实测**：lmetric 跑 600s trace 用 49 min wall-clock = **8x amplification**。同硬件 unified 比 lmetric session 清空速度 ~3x。**per-turn W 的小差异被放大成 wall-clock 数量级差距** —— 这意味着 locality 不是 nice-to-have，是 dominant lever。
|
||||
|
||||
---
|
||||
|
||||
@@ -34,8 +26,8 @@
|
||||
| | 数据 | 图 |
|
||||
|---|---|---|
|
||||
| KV reuse 几乎只在 session 内 | intra 93.2% / cross 5.7% / shared 1.1% |  |
|
||||
| Session 极度偏斜 | production trace 上 top 1% / 5% / 10% / 25% / 50% = **46.5% / 66.5% / 74.6% / 87.5% / 96.0%** input mass |  |
|
||||
| 单请求 KV footprint 大，单 instance KV pool 很快被占满 | per-instance KV pool ≈ **38 GiB**（0.4 × 96 GiB H20，剩 50% params + 10% activation）；p99 req 11.5 GiB → 一个 instance 只装 **3 个 p99 decode**；4P+4D 让系统 decode 容量直接减半 |  |
|
||||
| Session 极度偏斜 | top 1% = 46.5% input mass |  |
|
||||
| 单请求 KV footprint 已经很大 | p99 = 11.8 GiB ≈ H20 12% |  |
|
||||
|
||||
理论 APC 上界 = intra-session 79.6% / any-session 80.3%，差 <1pp。**任何不 affinity 的调度都丢绝大部分 reuse。**
|
||||
|
||||
@@ -47,9 +39,7 @@
|
||||
|
||||
|
||||
|
||||
LMetric 56.9%、load_only 54.1% APC，远低于 79.6% 上界。23pp 缺口直接来自跨 instance 路由丢的 intra-session hit。
|
||||
|
||||
注意 LMetric 比 load_only 只好 **+3.3pp**：LMetric 的 score = `(pending_prefill + input − cache_hit) × num_requests`，cache_hit 只作 cost-model 减项，但 score 是**乘性**的 —— 一个有 affinity 的 instance 因 num_requests 高被乘式吃掉 cache 收益，LMetric 仍然会选冷 instance。sticky 把 cache 作硬约束直接拉到 77.2%。**结论：cache-aware-load routing 不够 —— affinity 必须是独立路由路径，不能折叠进 load cost 里**。
|
||||
LMetric 56.9%、load_only 54.1%、capped 31.6% APC，远低于 79.6% 上界。23pp 缺口直接来自跨 instance 路由丢的 intra-session hit。
|
||||
|
||||
### 静态 PD-disagg：D 侧 KV 容量墙
|
||||
|
||||
@@ -57,18 +47,11 @@ LMetric 56.9%、load_only 54.1% APC，远低于 79.6% 上界。23pp 缺口直接
|
||||
|
||||
agentic 平均请求 33.6k token 需 3.3GB KV；4P+4D / 6P+2D 在 agentic regime 都穿过 90% 内存墙。**TTFT p50 暴涨 62-72x，成功率 99.5% → 52-68%**。
|
||||
|
||||
### Pure sticky：全员被 hot session 拖累
|
||||
### Pure sticky / current unified：hot pin
|
||||
|
||||
|
||||
|
||||
|
||||
我们刻意 **用 (median, max) 两个绝对数**衡量 worker 不平衡，不用 `max/median` 单一比值 —— 比值会把 unified（一个 worker 牺牲、其他 7 个快）算成比 sticky（全员一起慢）更不平衡，与系统 e2e p90 实际排序反向。下面是绝对数：
|
||||
|
||||
| | median worker TTFT p90 | max worker | system e2e p90 |
|
||||
|---|---:|---:|---:|
|
||||
| sticky | **20.3s** | 55.4s | **34.6s** |
|
||||
| unified | **10.3s** | 37.7s | **18.0s** |
|
||||
|
||||
机制：production trace 上 top 1% 的 session 占 46.5% input 量、且 hot session 数量远多于 instance 数（8 个），sticky 的 hash 绑定让 **每个 worker 都自己承接一份 hot session**，median worker 也被拖慢。Unified 用 LMetric fallback 把 cold/new session 重路由到非 hot worker，保留 7/8 worker 的速度。系统 p90 由大多数请求决定，所以 unified 几乎 2x 快。
|
||||
APC 拉到 77-79%（接近上界），但 hotspot index 翻倍：sticky 2.73、unified 3.66 vs lmetric 2.25、load_only 1.29。skew 中的大 session 被锁在单 instance 上，造成 prefill-decode 干扰。
|
||||
|
||||
---
|
||||
|
||||
@@ -80,16 +63,9 @@ agentic 平均请求 33.6k token 需 3.3GB KV；4P+4D / 6P+2D 在 agentic regime
|
||||
新 session 用 load-balance 分配 host；后续 turn 按 session→host binding 路由。
|
||||
→ 这就是当前 `unified` 算法（hybrid LMetric + high-cache affinity），APC 79.4%，达到上界 97%。
|
||||
|
||||
**Pillar 2 — Hot-triggered session migration（end-to-end 实证待补，substrate 已验证）**
|
||||
**Pillar 2 — Hot-triggered session migration（实证待补）**
|
||||
当 host 的 `pending_prefill_tokens > T_hot`，把整个 session 的 KV 通过 mooncake `kv_connector` migrate 到更轻 instance；session binding 更新；后续 turn 路由到新 host。
|
||||
|
||||
> 🆕 **2026-05-27 数据**（commit `ef9e010`）：之前认为是 migration blocker 的 `kv_both` substrate overhead 已经不存在。在 8×TP1 trace replay 上 A/B/C 对比：
|
||||
> - plain unified: TTFT p90 = 11.97s
|
||||
> - unified + `kv_both`（未 DR-fix）: 9.74s（**−18.6%** vs plain）
|
||||
> - unified + `kv_both` + DR-fix: 7.58s（**−36.6%** vs plain）
|
||||
>
|
||||
> 即原 elastic_migration_v2 论文里 "+45% kv_both penalty" 已 obsolete；当前 substrate 是 **net positive**（connector mode 的 `delay_free_blocks=True` 在 93% intra-session-reuse trace 上把跨 turn cache hit 窗口拉长）。Migration 之前 4 次 revert 的主因消失。
|
||||
|
||||
关键 design：
|
||||
- Target 选择用 **observable pending prefill tokens**，**不用** cost-model prediction（实测 mooncake cost model 误差 10-21x，绕过）
|
||||
- Per-session cooldown 防 thrashing
|
||||
@@ -107,7 +83,6 @@ agentic 平均请求 33.6k token 需 3.3GB KV；4P+4D / 6P+2D 在 agentic regime
|
||||
- Pillar 1 affinity routing 已实现并测过（current `unified` 算法）
|
||||
- Dispatch coupling 的 Little's Law 形式化推导
|
||||
- `replayer/replay.py` patched 输出 `amplification`
|
||||
- 🆕 **kv_both substrate validation**（commit `ef9e010`）：trace replay A/B/C 证明 substrate 已经是 net positive（TTFT p90 −18.6% / DR-fix 后 −36.6% vs plain），原 +45% penalty obsolete
|
||||
|
||||
### 🟢 不依赖 migration 可以现在做
|
||||
|
||||
@@ -116,17 +91,17 @@ agentic 平均请求 33.6k token 需 3.3GB KV；4P+4D / 6P+2D 在 agentic regime
|
||||
3. λ / skew / KV pool 三轴 sensitivity
|
||||
4. Draft §1-§4 正文（数据已齐）
|
||||
|
||||
### 🚧 待 migration end-to-end validation
|
||||
### 🚧 待 migration validation
|
||||
|
||||
- §4.3 migration mechanism 的 e2e trigger + target selection 实验（substrate 已通，只缺策略层）
|
||||
- §4.3 migration mechanism 在 `connector_tax` DR-fix 之上重测
|
||||
- Full ablation（migration-only + both）
|
||||
- §5.6 migration microbench
|
||||
|
||||
### 风险
|
||||
|
||||
- Migration 之前 4 次尝试（`6b255fa`, `e991960/5772149`, `cc6e562`, `4c583f2`）都被 transfer overhead 吞掉而 revert —— **该 overhead 已在 2026-05-27 验证不再存在**（substrate net positive）
|
||||
- 仍未直接验证 e2e migration 策略层（trigger + target 选择）能在反馈环里产生正收益；中间还有"决策错误 + cooldown thrashing"两类风险，独立于 substrate
|
||||
- 即便 migration e2e 仍 marginal，affinity-only pillar 的实证已经独立成立，paper 至少有 strong-affinity 的 storyline 可写
|
||||
- Migration 之前 4 次尝试（`6b255fa`, `e991960/5772149`, `cc6e562`, `4c583f2`）都被 transfer overhead 吞掉而 revert
|
||||
- 最近 DR-fix 把 `build_connector_meta` slope +81 → -0.7 μs/1k blocks，但**未在 trace replay 上验证**
|
||||
- 若 migration validation fail，paper 可 pivot 成 "affinity-only is enough" —— 仍然能发，强度降一档
|
||||
|
||||
---
|
||||
|
||||
|
||||
553
PAPER_OUTLINE.md
553
PAPER_OUTLINE.md
@@ -1,362 +1,359 @@
|
||||
# GPU-Hit-First: Serving Agentic LLM Workloads by Keeping the Working Set in HBM
|
||||
# EAR: Elastic Affinity Routing for Agentic LLM Serving
|
||||
|
||||
> **Thesis (one-liner)**: 对 agentic LLM 负载，用户感受到的端到端 metric 是 **request latency / TPS / GPU
|
||||
> utilization**，而它们由一件事主导 —— **KV cache 命中是否发生在 GPU HBM 上**。Agentic 的 KV reuse 93% 在
|
||||
> session 内、且活跃 working set 小到一个节点就能常驻 HBM；命中层级 `GPU ≫ CPU-local > remote-RDMA-store ≫
|
||||
> recompute` 的代价差随 context 拉大。由此得到一条统一原则 —— **GPU-hit-first**：把活跃 working set 留在 HBM，
|
||||
> 而不是建深的 CPU/storage hierarchy 去追长尾。三个推论分别修复现有系统的三处失配：(3.1) 让 PD-colocation 重新
|
||||
> 成为默认；(3.2) 在全局路由里做 biased KV-cache-awareness；(3.3) 用 KV migration 而非 replication 做跨实例
|
||||
> GPU 去重。
|
||||
|
||||
> **Framing note (2026-05-30)**：本 outline 取代早期的 "EAR: Elastic Affinity Routing" 版本（保留在 git
|
||||
> 历史里）。原 EAR 的 dispatch-coupling 形式化在此 **降级为 §2 的 metric 论证**（解释"为什么是 request latency
|
||||
> 而不是 TTFT/TPOT"），不再是 headline；headline 升格为 GPU-hit-first 原则，affinity routing / migration
|
||||
> 成为该原则的两个推论（§3.2 / §3.3）。
|
||||
> **One-liner**: Agentic LLM workload 的 KV reuse 93% 是 intra-session 的，且 turn 间 tool-call 反馈耦合把单 request 的延迟差放大成 throughput 差距 —— locality 因此成为主导调度杠杆；现有 load-balance 丢 locality、静态 PD-disagg 撞 D 侧 KV 墙、pure session-sticky 造 hot pin，我们提出 session-affinity routing + hot-instance 触发 session migration 的调度器 **EAR (Elastic Affinity Router)**，单一方案同时拿到 locality 和 balance。
|
||||
|
||||
---
|
||||
|
||||
## 📊 Validation Status (2026-05-30)
|
||||
## 📊 Validation Status (2026-05-27)
|
||||
|
||||
| 章节 | 论点 | 证据 | 状态 |
|
||||
|---|---|---|---|
|
||||
| §1 | 背景：PD-colo / PD-disagg / KV storage hierarchy | — | 写作 |
|
||||
| §2 metric | request latency 而非 TTFT/TPOT；TPS；GPU util | dispatch coupling + amplification；`bench_report.py` | 🟡 论证全，wall-clock sweep 待补 |
|
||||
| §2.1 | KV hit 普遍且关键 | `f2a` intra 93.2%、APC 上界 79.6%、C1/C2 | ✅ |
|
||||
| **§2.2** | **GPU hit > CPU hit > RDMA-store hit ≫ miss** | **`v2/figs/exp_a_tier_latency.png`（四层实测）** | ✅ **NEW** |
|
||||
| **§2.2 Ev#1** | **GPU 足以常驻"有价值的" working set** | **`v2/figs/exp_b_capacity_knee.png`（knee）+ working_set + cluster-scale 校正** | ✅ **NEW** |
|
||||
| §3.1 | Make PD-colocation great again | `PD_DISAGG_RESULTS`、`crossover_pd_advantage`、MB1/MB2、C2/C3 | ✅ |
|
||||
| §3.2 | Biased KV-aware global routing | LPWL（TTFT p90 −31%）、LMetric 乘性稀释、sticky hot-pin、ES ablation | ✅ |
|
||||
| §3.3 | GPU dedup via migration not replication | substrate net-positive（−18.6%/−36.6%）；correctness smoke tests | 🟡 substrate 通，policy e2e 待验证 |
|
||||
| §4 | 集成系统端到端 eval | 散落 mb1/mb2/mb5/crossover/lpwl，需统一 | 🚧 |
|
||||
| §5 | Related work（含 storage hierarchy 正面回应）| — | 写作 |
|
||||
| 部分 | 现有数据 | 待补 |
|
||||
|---|---|---|
|
||||
| §2 Workload characterization | ✅ 完整 (3 张图复用) | — |
|
||||
| §3.1 Load-balance 丢 locality | ✅ 完整 (`f4a`) | — |
|
||||
| §3.2 静态 PD-disagg 撞 KV 墙 | ✅ 完整 (`f4b`) | — |
|
||||
| §3.3 Sticky 造 hot pin | ✅ 完整 (`f4c`, `f4d`) | — |
|
||||
| §4.1-2 Affinity routing | ✅ 已实现（current `unified` 算法）| — |
|
||||
| §4.3 Migration mechanism | 🚧 **DEFERRED** | 待 connector_tax fix 后重测 |
|
||||
| §5.2 End-to-end | ⚠️ 5/6 baseline 有数据 (`f6`) | 缺 static PD-disagg；EAR 列待 migration |
|
||||
| §5.3 Ablation | 🚧 **PARTIAL DEFER** | 仅 affinity-only 现可做，full 待 migration |
|
||||
| §5.4 Dispatch coupling validation | 🚧 **NEW DATA NEEDED** | 5 baseline wall-clock 重跑（Phase 1 patch 后）|
|
||||
| §5.5 Sensitivity | 🚧 **PARTIAL DEFER** | λ/skew/KV pool 可做；`T_hot`/`T_cool` 待 migration |
|
||||
| §5.6 Migration microbench | 🚧 **FULL DEFER** | 完全依赖 migration validation |
|
||||
|
||||
**前提背景**：team 之前 4 次尝试 migration 都因 transfer overhead 被还原（见 `analysis/unified_routing_fix_review.md`）；最近 `connector_tax` 工作的 DR-fix 把 build_connector_meta 的 1.4ms/step overhead 降到接近 0，但还未跑过完整 migration 实验。**EAR 的 migration 部分目前是 design intent，待重测后写入实证。**
|
||||
|
||||
---
|
||||
|
||||
## §1 Background and System Setup
|
||||
## §1 Introduction
|
||||
|
||||
### §1.1 LLM 与 KV cache
|
||||
Agentic LLM workload —— 由 LLM 通过 tool call 自驱、多 turn 完成任务 —— 已经成为推理系统的主导负载，但现有为 chatbot 设计的调度策略在 agentic 下普遍失败。本文先刻画 agentic 与 chatbot 的本质区别，然后说明为什么三类主流调度都不够，最后给出 EAR 设计。
|
||||
|
||||
Transformer 自回归推理分两段：**prefill**（一次性算完 prompt 的全部 KV，compute-bound）与 **decode**（逐 token
|
||||
生成，memory-bandwidth-bound）。每个 token 的 KV 常驻 GPU HBM 才能被后续 attention 复用。Prefix caching（APC）让
|
||||
相同 prompt 前缀直接命中已算好的 KV，省掉重复 prefill —— 这是本文全部优化的物理基础。
|
||||
**Contributions**:
|
||||
|
||||
> Qwen3-Coder-30B-A3B（GQA, 48 层, 4 KV heads, head_dim 128, bf16）：**KV = 96 KiB/token**，1 GiB = 10,923
|
||||
> token，block(16 tok) = 1.573 MB。
|
||||
- **C1 Dispatch coupling 论证**：我们形式化一个 agentic workload 独有的反馈环 —— 单 turn 服务时间通过 Little's Law 隐式方程影响并发 session 数，从而把 per-request 延迟差放大成 throughput 差距。实测：load-balance baseline 在 600s trace 上跑出 **8x** wall-clock amplification；EAR 跑出 **TBDx**。
|
||||
- **C2 EAR 设计**：两个 pillar 的调度器 —— affinity-default routing 抓 intra-session locality，hot-instance 触发的 session migration 在 hotspot 出现时把整个 session 的 KV 搬到更轻的 instance，避免 hot pin。
|
||||
- **C3 评估**：在真实 Qwen3-Coder agentic trace 上，EAR 同时 dominate 5 个 baseline 的 TTFT、TPOT、APC、hotspot index、wall-clock 五个维度。
|
||||
|
||||
### §1.2 Agentic workflow
|
||||
|
||||
Agentic 负载 = LLM 通过 tool-call 自驱、多 turn 完成任务。与 chatbot 的本质差异：每个 turn 由上一个 turn 的
|
||||
tool-call 结果触发（无人类 think-time），prefill-dominated（input/output ≈ 75×）。
|
||||
|
||||
**但它是一个 mixture，不是"全多轮"**（C1, `figs/workload_chars/c1_session_mixture.png`）：
|
||||
|
||||
- **90.3%** 的 session 是单轮（mean 1.62 turns）；但多轮 session（9.7%）= **44.2% 的请求**、**66.9% 的
|
||||
prefill 质量**。
|
||||
- Continuation hazard（Lindy）：turn1→2 仅 **10.2%**，turn5→6 87%，turn12→13 **94.3%** —— heaviness 在
|
||||
cold-start 几乎不可预测（corr(turn1_input, n_turns) = **0.04**）。
|
||||
|
||||
> **Routing 含义（贯穿 §3.2）**：heaviness 在 session 起点不可预测 → 必须 **reactive**（观测累计负载），不能
|
||||
> proactive 预判；单轮海洋与深尾的最优策略相反（前者 load-balance、后者 affinity-pin），且 turn-1 无法区分 →
|
||||
> 唯一可行的策略是"人人 load-balanced 起步、随 turn 累积变 sticky" —— 正是 LPWL 的 emergent 行为。
|
||||
|
||||
### §1.3 Serving agents in the wild
|
||||
|
||||
- **PD colocation（8C）**：每个 instance 对称，prefill 与 decode 在同一 GPU 上由 chunked-prefill +
|
||||
continuous batching 交错。弹性 KV 池，无静态分区。
|
||||
- **PD disaggregation**：把 instance 静态分成 prefill 池（P）与 decode 池（D），物理隔离两个阶段
|
||||
（DistServe / Splitwise）。
|
||||
- **KV cache storage hierarchy**：GPU HBM → CPU DRAM → 远端 pooled store（RDMA/SSD，如 Mooncake Store /
|
||||
LMCache）。把被淘汰/跨实例的 KV 下沉到更慢但更大的层，用传输换重算。
|
||||
|
||||
> 三者各被本文一节回应：PD-colo 在 §3.1 被"复活"为默认；PD-disagg 在 §3.1 被证否（agentic regime）；storage
|
||||
> hierarchy 在 §2.2 被定量地"限位"（GPU 命中远胜下层，且活跃 working set 本就装得下）。
|
||||
**Figure 1: Teaser — wall-clock vs trace-time across schedulers** — `figs/f1_teaser.png` **🚧 TBD (NEW DATA NEEDED)**
|
||||
> Needs Phase 3 measurements: 5 baselines × 3 runs of trace replay, extract `amplification = wall_clock_s / trace_span_s` from each summary (Phase 1 patch already exposes the field). Plot as bar chart with y=1 reference line. EAR row 暂为 TBD（待 migration validation）。
|
||||
|
||||
---
|
||||
|
||||
## §2 GPU memory hit is the key to serving agents
|
||||
## §2 Background and Workload Characterization
|
||||
|
||||
### §2.0 正确的 metric：request latency / TPS / GPU utilization（不是 TTFT/TPOT）
|
||||
### §2.1 Agentic Workload Primer
|
||||
|
||||
**为什么不是 per-request TTFT/TPOT**：agentic 的 turn 之间有反馈环，单 turn 的延迟会跨 turn **复利**成 session
|
||||
端到端时间与系统吞吐差距。只有 **request/session latency、tokens-per-second、GPU utilization** 能 capture 这件事。
|
||||
Agentic workload 与 chatbot 的三个本质差异：
|
||||
|
||||
**Dispatch coupling（降级为本节论证）**。每个 turn 间有外部 gap `T_external`（chatbot 是人在读/想/打字；agentic
|
||||
是 tool 执行）。Little's Law：`L = Λ · N · (W_turn(L) + T_external)`。当 `W_turn ≳ T_external` 时
|
||||
`W_turn(L)` 经 KV 竞争耦合到并发 L，scheduler 的 ε 退步被反馈环放大成 wall-clock 数倍差。
|
||||
- **Multi-turn, programmatic continuation**：每个 turn 由上一个 turn 的 tool-call 结果触发，没有人类 think-time
|
||||
- **Prefill-dominated**：input/output token ratio **75x**，98% 计算在 prefill 阶段（chatbot 为 1-10x）
|
||||
- **Skewed sessions**：top 1% session 贡献 **46.5%** input token 量
|
||||
|
||||
production trace 实测 `T_external` CDF（`figs/f3a_inter_turn_gap.png`）：
|
||||
平均 session 长度 TBD turn、TBD 输入 token；p99 单请求 KV 占用 **11.49 GiB**（H20 96GB HBM 的 12%）。
|
||||
|
||||
| | Agentic | Chatbot |
|
||||
|---|---:|---:|
|
||||
| p50 | **1.6 s** | 7.2 s |
|
||||
| gap < 1 s | **39%** | 4% |
|
||||
| gap < 5 s | 67% | 29% |
|
||||
| p99 | 738 s | 43 s |
|
||||
### §2.2 KV Cache Reuse Topology
|
||||
|
||||
agentic 有一段 chatbot 没有的 **sub-second tool-call mass（39% vs 4%）**，几乎天然 `W_turn ≫ T_external` → 闭环。
|
||||
**实测**：lmetric 跑 600s trace 用 49 min wall-clock = **8× amplification**。**结论：per-turn 延迟的小差被放大成
|
||||
端到端数量级差 → 必须用 request latency / TPS / GPU util 衡量。**
|
||||
Trace 上 KV reuse 的分解：
|
||||
|
||||
- **TPS / GPU util** 的工具：`microbench/fresh_setup/bench_report.py`（TTFT/TPOT/E2E 全分位 + TPS +
|
||||
per-worker GPU util）。PD stall 时 GPU ~0% util vs colo 34%（§3.1）即是 GPU-util 作为 metric 的直接体现。
|
||||
| Class | Share |
|
||||
|---|---|
|
||||
| Intra-session | **93.2%** |
|
||||
| Cross-session | 5.7% |
|
||||
| Shared prefix | 1.1% |
|
||||
|
||||
> **🚧 待补**：5 baseline × ≥3 runs 的 wall-clock amplification sweep（replayer 已输出 `amplification` 字段），
|
||||
> 钉死本节实证 closure。优先级高。
|
||||
理论 APC 上界：any-session **80.3%**，intra-session-only **79.6%**，差距 <1pp。**cache 本质上是 session-local 的**；任何不保留 session affinity 的调度都丢掉绝大部分 reuse 机会。
|
||||
|
||||
### §2.1 KV$ hit is common and critical
|
||||
**Figure 2: Workload characterization (3 panels)** — 现有数据可复用：
|
||||
|
||||
Trace 上 KV reuse 的分解（`figs/f2a_reuse_topology.png`）：**intra-session 93.2% / cross-session 5.7% /
|
||||
shared-prefix 1.1%**。理论 APC 上界：intra-only **79.6%** vs any-session 80.3%，差 <1pp —— **cache 本质上是
|
||||
session-local 的**。
|
||||
|
||||
|
||||
per-turn 视角（C2, `figs/workload_chars/c2_work_amortization.png`）：resident context 11k→56k+ token 增长而
|
||||
new-prefill 从 2.7k 坍缩到 ~200 token，per-turn reuse 爬到 **99.6%**，resident/new（"PD tax"）到 turn 12 ≈
|
||||
250×、turn 30 ≈ 450×。**绝大部分 prefill 工作是可被命中省掉的**；命中与否直接决定 TTFT。
|
||||
|
||||
|
||||
### §2.2 Hits on GPU is more important than the CPU
|
||||
|
||||
|
||||
**命中层级的代价是实测的，不是断言的**（Qwen3-Coder-30B-A3B / H20）。TTFT(s, p50) 服务一段长 L 的复用前缀，
|
||||
来自每个 KV 层：
|
||||
> 📝 Layout TBD：三张拼成 1×3 还是分散到 §2.1/§2.2/§2.4 各一张。
|
||||
|
||||
|
||||
### §2.3 Dispatch Coupling — Why Locality Dominates
|
||||
|
||||
这是本文最依赖直觉的论证，单独成节。
|
||||
|
||||
| prefix L | miss(recompute) | **remote RDMA store** | CPU-local(DRAM,PCIe) | GPU(HBM) | miss/RDMA | RDMA/CPU | CPU/GPU |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 8k | 0.588 | 0.151 | 0.076 | 0.053 | 3.9× | 2.0× | 1.5× |
|
||||
| 16k | 1.547 | 0.262 | 0.105 | 0.063 | 5.9× | 2.5× | 1.7× |
|
||||
| 32k | 4.604 | 0.680 | 0.158 | 0.080 | 6.8× | 4.3× | 2.0× |
|
||||
| **64k** | **15.23** | **0.97** | **0.27** | **0.11** | **15.8×** | **3.6×** | **2.4×** |
|
||||
**直觉**：chatbot 里每个 turn 后人要读、想、打字，**外部时钟**控制下一个 turn 何时到达；agentic 里 LLM 一拿到 tool-call 结果立刻发下一个 request，**系统自己的速度决定下一个 turn 何时到达**。所以一个慢策略不仅让单请求变慢，还让 session 在系统里停留更久 → 并发 session 更多 → KV 竞争更激烈 → 每个 turn 更慢 —— 反馈环。
|
||||
|
||||
- **GPU hit ~flat**（42→111 ms / 1k→64k）：命中即整段前缀在 HBM，只重算最后一个 token。
|
||||
- **CPU-local hit** transfer-bound（PCIe H2D 实测 ~54 GB/s）；CPU-hit ≈ GPU-hit + KV/PCIe + ~0.15s 开销。
|
||||
（native KV offload，命中经 `vllm:external_prefix_cache_hits` 100% 验证。）
|
||||
- **remote RDMA-store hit** = Mooncake-Store 机制（实测：两 instance，B 用 `do_remote_prefill` 经 RDMA 从 A 拉取
|
||||
缓存前缀而非重算；`mb2_kv_transfer.py` / `v2/.../run_rdma.sh`）。对 recompute 是大赢（**最高 16×**，与 blog 的
|
||||
46× 同向），但付 **NIC 税**（有效 ~5–7 GB/s，cf. MB2 raw ~9.7 GB/s；multi-NIC pooling 可抬高），故比 CPU-local
|
||||
慢 3.6×、比 GPU 慢 ~9×（64k），**代价差随 context 拉大**。
|
||||
- **结论 —— 层级严格且随 context 拉大：`GPU < CPU-local < remote-RDMA-store ≪ miss`**。global KV store 确实
|
||||
有用（这也是该路线存在的理由），但每靠近 GPU 一层就再省 1.4–4× TTFT。**最值钱的复用是 GPU-resident 的那种。**
|
||||
**具体例子**：一个 coding agent 跑 20 turn 的任务。
|
||||
|
||||
#### Evidence #1：GPU is sufficient to hold most KV requests
|
||||
- 快策略：每 turn 2s，session 共 40s，平均并发 10 个 session
|
||||
- 慢策略（线性估算）：每 turn 3s，session 共 60s，应该并发 15 个
|
||||
- 慢策略（实际）：15 个并发让每 turn 被推到 4s，session 80s，并发 20 个，turn 再推到 5s …… 直到撞墙或落到一个远更糟的新平衡
|
||||
|
||||
**realized APC 与 latency 在很小的 GPU 容量就饱和**（closed-loop 多轮负载，并发 4，扫 GPU KV 容量）：
|
||||
对照 chatbot：每 turn 后人读 30s。turn 从 2s 变 3s，session 从 32s 变 33s，3% 差距，几乎无反馈。
|
||||
|
||||
|
||||
**形式化**。记 `Λ` = session 到达率（trace 给定），`N` = 每 session 的 turn 数，`W_turn(L)` = 单 turn 服务时间，是当前并发 session 数 `L` 的递增函数（并发越多、KV 竞争越激烈、`W_turn` 越大）。
|
||||
|
||||
| GPU KV (GB) | realized APC | TTFT p90 |
|
||||
|---:|---:|---:|
|
||||
| 1.2 | 7.4% | 13.00 s |
|
||||
| 2.4 | 36.3% | 4.62 s |
|
||||
| **3.6** | **80.3%** | **0.53 s** |
|
||||
| 9.7 | 72.9% | 0.65 s |
|
||||
| 14.5| 72.9% | 0.65 s |
|
||||
Chatbot 的 Little's Law:
|
||||
```
|
||||
L = Λ · N · (W_turn(L) + T_human)
|
||||
```
|
||||
被大常数 `T_human` 主导，`W_turn(L)` 的扰动几乎不动 `L`。
|
||||
|
||||
**Knee 出现在 3.6 GB = 恰好 = 活跃 working set（4 session × 0.91 GB）**：APC 饱和到上界、TTFT p90 从 13.0s 坍缩
|
||||
到 0.53s，之后 dead-flat。**超过 working set 的 HBM 买不到额外收益；为追长尾而建的 CPU/storage tier 同理 ≈ 0。**
|
||||
Agentic 的 Little's Law（`T_human ≈ 0`）:
|
||||
```
|
||||
L = Λ · N · W_turn(L)
|
||||
```
|
||||
这是关于 `L` 的隐式方程。设策略变化让 `W_turn` 整体放大 `(1+ε)` 倍，小扰动分析得到：
|
||||
```
|
||||
dL/dε|_{ε=0} = L* / (1 − Λ · N · W'_turn(L*))
|
||||
```
|
||||
**分母接近 0** 时（系统接近 KV 饱和），放大系数发散。这就是为什么 lmetric 在 600s trace 上跑出 8x wall-clock 放大。
|
||||
|
||||
**Cluster-scale 校正（关键）**：working_set 分析（`analysis/working_set/`，`figs/working_set/`）显示"装下整个
|
||||
2h cluster 的全部 reuse 尾巴需 ~14 节点"，**这不构成 CPU offload 的动机** —— 那是用 1 个 replica 容量去装**整个
|
||||
cluster** 的 reuse；产出该 trace 的真实 cluster 远不止 14 节点（trace 是 cluster 聚合，见
|
||||
`project-trace-is-cluster-level`）。在 per-cluster 的 HBM 总量下，活跃 working set 本就 GPU-resident（live KV
|
||||
533–1157 GB ≪ 单节点 1528 GB）。**knee 位置随并发线性增长 = 随 cluster GPU 数增长**，而 cluster 本就提供了它。
|
||||
**Figure 3: Dispatch coupling schematic** — `figs/f3_coupling_schematic.png` **🚧 TBD (CUSTOM DRAW)**
|
||||
> 需要新画一张示意图：上半 chatbot timeline（`system → T_human → system → T_human → ...`），下半 agentic timeline（`system → ε → system → ε → ...`），右侧叠一个反馈环箭头 `W_turn → Λ → L → W_turn`。适合用 TikZ / draw.io / matplotlib annotate。
|
||||
|
||||
> ⚠ **Scope**：本小节"装得下"指的是**活跃 working set 产生的近期高价值复用**，不是"全部 reuse 尾巴"。冷
|
||||
> session 长 gap 后回来的深尾命中（既低价值/byte 又贵于 fetch）正是 storage-hierarchy 派追的东西；本文论点是
|
||||
> 在 agentic 下这条尾巴不值得为之建深层级。§5 正面回应该派。
|
||||
### §2.4 Takeaway
|
||||
|
||||
### §2.3 Takeaway
|
||||
|
||||
正确的 metric（§2.0）+ 命中集中在 GPU 才便宜（§2.2）+ 活跃 working set 装得下 HBM（Ev#1）⇒ **GPU-hit-first**：
|
||||
设计目标是最大化活跃 working set 的 **GPU 常驻 + 命中**，而非建深 CPU/storage hierarchy。§3 给出三个推论。
|
||||
三个性质 —— intra-session locality dominant (§2.2)、long context + prefill-heavy (§2.1)、dispatch coupling (§2.3) —— 共同决定了 agentic workload 的调度必须以 **locality 为主导**，并能容忍 skew 带来的 instance 间负载不均。
|
||||
|
||||
---
|
||||
|
||||
## §3 Optimizing agent serving with the GPU-Hit-First Principle
|
||||
## §3 Why Existing Schedulers Don't Fit
|
||||
|
||||
### §3.1 Make PD-colocation great again
|
||||
三类现有调度各自撞上 §2 三个性质中的一个：
|
||||
|
||||
静态 PD-disaggregation 对 chatbot 有效，对 agentic 结构性失败 —— colocation 才应是默认。
|
||||
### §3.1 Load-balanced routing 丢 locality
|
||||
|
||||
**端到端证据**（`microbench/fresh_setup/PD_DISAGG_RESULTS.md`，8×H20，trace replay）：**没有任何静态 P/D 比能赢
|
||||
8-way colocation（8C）**，且失败模式随比例移动：
|
||||
Round-robin 和 load-aware routing（如 LMetric, OSDI'26）最大化 instance 利用率，但忽略 session affinity。**实测 APC 跌到 56.9%**（vs 上界 79.6%），23pp 的差距直接来自丢失的 intra-session cache hit。违反 §2.2。
|
||||
|
||||
| Metric | **8C** | 6P+2D | 4P+4D | 2P+6D |
|
||||
|---|---:|---:|---:|---:|
|
||||
| completion | **100%** | 100% | 100% | **9%** 💀 |
|
||||
| wall-clock | **2994 s** | 3419 | 4171 | 5762 |
|
||||
| prefix-cache hit | **19.4%** | 0% | 0% | 0% |
|
||||
| TTFT p50 | **7.0 s** | 41.0 | 56.4 | 23.6 |
|
||||
| E2E p90 | **83.3 s** | 91.8 | 157.1 | 499 |
|
||||
### §3.2 静态 PD-disaggregation 撞 D 侧 KV 墙
|
||||
|
||||
- **D-heavy（4P+4D）**：decode 池饱和 **97.5%**、prefill 池 ~30% —— 半个 cluster 的 KV 被困在错的一侧；agentic
|
||||
请求大（p99 KV **11.5 GiB**），4D 让系统 decode 容量直接减半（24→12 并发，`figs/f2c_kv_footprint_cdf.png`、
|
||||
`figs/f4b_pdsep_kv_wall.png`）。
|
||||
- **P-heavy（2P+6D）**：prefill 池 jam 99.7%，872 请求堆积，**91% 永不完成**。
|
||||
- **更聪明的路由救不了**（§6.x）：给 P 侧加 session-affinity 反而更差（4P+4D completion 100%→36%），GPU ~0%
|
||||
util，cluster 卡在 KV-transfer 协调而非 compute —— 复现 producer hot-pinning。
|
||||
静态把 instance 分成 P pool 和 D pool 对 chatbot 有效，对 agentic 失败：agentic 请求平均 33.6k token，需要 **3.3GB** KV；4D 方案下 p90 请求占 D 侧 KV pool **69%**，p99 直接 **溢出 138%**。结果：**TTFT p50 暴涨 62-72x**，成功率从 99.5% 跌至 **52-68%**。违反 §2.1（prefill-dominant + 长 context）。
|
||||
|
||||
**为什么 colo 赢（正确论证，C2/C3 支撑）**：
|
||||
### §3.3 Pure session-sticky 造 hot pin
|
||||
|
||||
- **时变 P:D 需求**：agentic 同时在 roofline 两侧有实质工作 —— compute-bound prefill（~30% 时间）+
|
||||
memory-bound decode（**~70% 时间**，C3 token≠time 校正，`figs/workload_chars/c3_prefill_decode_balance.png`）。
|
||||
colo 的弹性池吸收当下热的那一相；静态分区让 P-instance 带宽闲、D-instance 算力闲。
|
||||
- **resident KV 本地化**（C2）：下一 turn 的 prefix = [prevPrompt+prevAnswer] 横跨 P/D 两侧，disagg 必须
|
||||
gather/transfer，colo 免费本地保留。
|
||||
- **transfer 不便宜且拓扑无关**（MB2，`figs/mb2_transfer_time_compare.png`）：Mooncake `batch_transfer_sync_write`
|
||||
恒走 RDMA NIC（~9.7 GB/s），intra ≈ inter；PD-disagg 的 per-request transfer 税无法靠拓扑买回。
|
||||
- **phase-isolation 是 disagg 唯一的真赢面但被压倒**（MB1，`figs/mb1_interference.png`：32k prefill 让
|
||||
per-stream TPOT 退化 52×，131k → 183×）—— 但被 D 侧容量天花板压倒（见上）。
|
||||
硬 session-instance 绑定恢复 locality（APC **77.2%**，达到上界 97%），但把 skew 中的大 session 锁在单 instance 上，**interference index 从 LMetric 的 6.53 翻倍到 13.65**（同 trace 同硬件）。违反 §2.4 的 skew 容忍要求。
|
||||
|
||||
**边界（不 overclaim）**：crossover sweep（`analysis/crossover/`，`figs/crossover_pd_advantage.png`）给出 colo
|
||||
停止占优的 input 长度 —— colo 在 agentic 工作点赢，且我们知道边界在哪。
|
||||
**Figure 4: Three baselines, three failure modes** — 拆成三个子图，分别放在 §3.1/§3.2/§3.3：
|
||||
|
||||
### §3.2 Biased KV-cache-awareness in global routing
|
||||
§3.1 — APC 实测 vs 理论上界 79.6% (lmetric 56.9%, load_only 54.1%, capped 31.6%, sticky 77.2%, unified 79.4%)：
|
||||
|
||||
|
||||
GPU-hit-first 在路由层 = **把 cache-awareness 作为带偏置的独立路由路径**，而不是折叠进 load cost。
|
||||
§3.2 — D 侧 KV pool 占用 vs per-request KV footprint，4P+4D 和 6P+2D 在 agentic regime 都穿过 90% 内存墙：
|
||||
|
||||
|
||||
**反例：load-balanced / 朴素 cache-aware-load 丢 locality**（`figs/f4a_apc_loss.png`）。LMetric（OSDI'26）打分
|
||||
`P = pending_prefill + (input − cache_hit)`，`score = P × num_requests` —— cache 只作 cost-model **减项**，而 score
|
||||
是**乘性**的，有 affinity 的 instance 因 num_requests 高被乘式吃掉 cache 收益：
|
||||
§3.3 — APC vs hotspot index 散点（unified/sticky 在高 APC 高 hotspot 区，lmetric/load_only 在低 APC 低 hotspot 区）：
|
||||
|
||||
|
||||
| 策略 | APC | vs load_only | cache 处理方式 |
|
||||
|---|---:|---:|---|
|
||||
| load_only | 53.9% | — | 无 |
|
||||
| LMetric | 57.2% | **+3.3pp** | cost-model 减项（被稀释）|
|
||||
| sticky | 77.7% | **+23.8pp** | 硬约束 |
|
||||
| unified | 78.7% | **+24.8pp** | 硬+软混合 |
|
||||
> 📝 可选支撑图 — Prefill-decode 干扰（同 GPU 8k prefill 让 TPOT 退化 66x），放 §3.3 支撑 sticky 的 interference 论证：
|
||||
|
||||
|
||||
`load_only→LMetric` 的 +3.3pp 几乎可忽略；**+20.5pp 的回报来自把 cache 作独立路由路径**。
|
||||
### §3.4 Takeaway
|
||||
|
||||
**本文方法：LPWL（least-prefill-work，parameter-free）**（`project-lpwl-policy`、`analysis/lpwl_5policy_600s.md`）：
|
||||
按 `new_uncached ≈ input − cache_hit` 路由 —— `new_uncached≈input`（冷/新 session）自动按负载分散，`new_uncached≈0`
|
||||
（暖 session）自动 stick。**零旋钮**，在 600s trace 上击败 tuned unified+A+B（TTFT p90 **−31%**），full w600 上打平。
|
||||
这正是 C1 mixture 要的形状：无需 classifier 自动切分单轮海洋与深尾。
|
||||
|
||||
**两条被否的杠杆（节省读者时间）**：
|
||||
|
||||
- **real-time engine state 不是路由杠杆**（ES ablation，`project-es-ablation-sweep`）：只对一次性 placement 有用
|
||||
（sticky −26%），对 per-req load-chasing 有害（load_only +27%）。
|
||||
- **derived-κ decode 项 net-negative**：decode-awareness 是错的杠杆。
|
||||
|
||||
**Pure sticky 的失败用绝对 per-worker latency 衡量**（`figs/f4c_per_worker_ttft.png`，不能用 max/median 比值）：
|
||||
|
||||
| | median worker TTFT p90 | max worker | system e2e p90 |
|
||||
|---|---:|---:|---:|
|
||||
| sticky | **20.3 s** | 55.4 s | **34.6 s** |
|
||||
| unified | **10.3 s** | 37.7 s | **18.0 s** |
|
||||
|
||||
hot session 数 ≫ instance 数（8），sticky 的 hash 绑定让**每个 worker 都自接一份 hot session**，median 也被拖慢；
|
||||
biased 路由把 cold/new 重路由到非 hot worker，保 7/8 worker 速度 → e2e p90 ~2× 快。**这引出 §3.3：sticky 的残余
|
||||
hot-pin 需要 migration 解。**
|
||||
|
||||
### §3.3 GPU KV-cache dedup with migration instead of replication
|
||||
|
||||
**视角**：多个 instance 各自缓存同一段 prefix = GPU 容量被 **replication** 浪费；GPU-hit-first 要求**全局只留一份 +
|
||||
把 session 搬到那一份**（migration/dedup），既保 GPU 命中又均衡负载、修复 §3.2 的残余 hot-pin。
|
||||
|
||||
- **Trigger**：host 的 `pending_prefill > T_hot` 且 session 在 `T_cool` 内未迁移过。
|
||||
- **Target**：用 **observable pending prefill tokens** 选最轻 instance，**不用** cost-model 预测（mooncake cost
|
||||
model 误差 10–21×，by construction 绕过）。
|
||||
- **Mechanism**：当前 request 重定向到 target，session KV 经 Mooncake `kv_connector` 迁移；binding 更新；后续
|
||||
turn 按 affinity 路由到新 host。迁移成本在该 session 剩余 turn 上摊销。
|
||||
- **Thrashing prevention**：per-session cooldown `T_cool`。
|
||||
|
||||
> **状态**：**substrate 已验证 net-positive**（`kv_both` connector 在 trace replay 上 TTFT p90 **−18.6%**，
|
||||
> DR-fix 后 **−36.6%** vs plain；之前认为是 blocker 的 transfer overhead 已不存在，4 次历史 revert 的主因消失）。
|
||||
> migration correctness smoke tests 通过。**policy 层 e2e（trigger + target 在反馈环里的真实收益）仍未直接验证**
|
||||
> —— 这是全文最弱的一环，独立风险是"决策错误 + cooldown thrashing"。affinity-only pillar（§3.2 LPWL）已独立成立，
|
||||
> 即便 migration 仍 marginal，paper 也有 strong-routing 主线。
|
||||
**问题不是任何单一 baseline 太弱，而是没有一个方案同时满足 §2 的三个性质**：保留 locality、尊重 D 侧 KV 容量、容忍 skew 带来的负载不均。EAR 是据我们所知第一个三件事同时做到的调度器。
|
||||
|
||||
---
|
||||
|
||||
## §4 System Evaluation
|
||||
## §4 Design: EAR
|
||||
|
||||
> **🚧 关键缺口**：目前证据散落在 mb1/mb2/mb5/crossover/lpwl/v2；§4 需要一个**集成系统**（colocation +
|
||||
> biased routing + dedup-migration，统一命名）跑端到端、用 §2.0 的新 metric（request latency / TPS / GPU util）
|
||||
> 评测，并把 §3.1/§3.2/§3.3 做成 ablation。
|
||||
### §4.1 Architecture
|
||||
|
||||
### §4.1 Setup
|
||||
- **Trace**：真实 Qwen3-Coder agentic trace（cluster-level，见 `project-trace-is-cluster-level`）；日常迭代
|
||||
`w600_r0.0015_st30`（~850 req / ~13 min / peak QPS ~1.6 / APC headroom ~76%）。
|
||||
- **Hardware**：8× H20 (96 GB)；vLLM 0.18.1 (V1, chunked-prefill) + Mooncake。
|
||||
- **Baselines**：① round-robin ② LMetric (OSDI'26) ③ kvcache-aware+load 线性混合 ④ sticky ⑤ static PD-disagg
|
||||
(4P/4D) ⑥ **本文系统**（colo + LPWL + dedup-migration）。
|
||||
- **Metrics**：request latency (mean/p50/p90/p99)、**TPS**、**GPU util (median/max worker)**、APC、
|
||||
wall-clock amplification（不单看 TTFT/TPOT，§2.0）。
|
||||
EAR 是位于 N 个同质 instance 之上的 router。每个 instance 是对称的 PD-colocated，没有静态 P/D 分区。每个 session 在 router 内维护一个 **host binding** —— 当前持有该 session KV 状态的 instance。Binding 在常态下稳定，仅在 hotspot 触发时通过 migration 改变。
|
||||
|
||||
### §4.2 End-to-end
|
||||
- `figs/f6_e2e_latency_bars.png` / `f6_e2e_latency_full_grid.png`（现有 4–5 baseline；🚧 补 static PD-disagg
|
||||
列 + 本文系统列）。
|
||||
**Figure 5: EAR architecture and request flow** — `figs/f5_architecture.png` **🚧 TBD (CUSTOM DRAW)**
|
||||
> 组件图：router (含 session→host table) → N 个 symmetric instances；affinity 路径实线，migration path 虚线。适合 TikZ / draw.io。
|
||||
|
||||
### §4.3 Ablation（GPU-hit-first 三推论各关一个）
|
||||
- −colocation（→ static PD-disagg，§3.1）
|
||||
- −biased routing（→ load-balance / LMetric，§3.2）
|
||||
- −dedup-migration（→ pure sticky，§3.3）
|
||||
- 🚧 migration-only / full 待 policy e2e 验证。
|
||||
### §4.2 Pillar 1: Affinity-Default Routing
|
||||
|
||||
### §4.4 Microbench 支撑（已有，复用）
|
||||
- §2.2 四层命中：`v2/figs/exp_a_tier_latency.png`
|
||||
- §2.2 容量 knee：`v2/figs/exp_b_capacity_knee.png`
|
||||
- §3.1 PD-disagg 失败：`figs/f4b`、`PD_DISAGG_RESULTS`、`crossover_pd_advantage`
|
||||
- transfer cost：`figs/mb2_*`；phase interference：`figs/mb1_interference.png`
|
||||
- **Cold start**：新 session 到达时，router 用 load-balance（选 pending prefill tokens 最少的 instance）分配初始 host
|
||||
- **Warm path**：已建立 session 的后续每个 turn 一律路由到当前 host
|
||||
- **效果**：intra-session KV reuse 被构造性保留，APC 接近 §2.2 的上界 79.6%
|
||||
|
||||
### §4.5 Sensitivity
|
||||
- 到达率 λ / skew(Zipf α) / KV pool size（不依赖 migration，可做）；`T_hot`/`T_cool`（依赖 migration，deferred）。
|
||||
### §4.3 Pillar 2: Hot-Triggered Session Migration 🚧 DEFERRED VALIDATION
|
||||
|
||||
避免 Pillar 1 退化成 pure sticky 的关键 mechanism。
|
||||
|
||||
> **状态**：Design 描述完整，但实证数据待 `connector_tax` DR-fix 之后重测。之前 4 次 migration 尝试（`6b255fa`, `e991960/5772149`, `cc6e562`, `4c583f2`）都因 transfer overhead 被还原 —— 直到 DR-fix 之前，migration 的实测收益始终被 overhead 吞掉。新一轮验证未跑。
|
||||
|
||||
#### §4.3.1 Trigger signal
|
||||
|
||||
EAR 实时监控每个 instance 的 **pending prefill tokens**。新 request 到达且按 affinity 应该路由到 host H 时，router 先检查：
|
||||
- `H.pending_prefill > T_hot`？（hotspot 检测）
|
||||
- session 在过去 `T_cool` 秒内未发生过 migration？（thrashing prevention，§4.3.4）
|
||||
|
||||
两个条件同时满足才考虑触发 migration。`T_hot` 和 `T_cool` 的取值见 §5.5 sensitivity。
|
||||
|
||||
#### §4.3.2 Target selection
|
||||
|
||||
候选集：所有 instance 中 (a) 剩余 KV 容量能装下 session 现有 context、(b) `pending_prefill` 严格小于 H 的。选 `pending_prefill` 最低者。
|
||||
|
||||
**关键设计点**：我们用 **observable current load** 而不是 **predicted transfer time** 排序。文献和 colleague 数据均显示 mooncake cost model 的预测误差达 10-21x；而 pending prefill tokens 是 router 直接观察到的数值，accuracy by construction。
|
||||
|
||||
若候选集为空（所有其他 instance 都装不下，或都比 H 更忙），EAR 保留当前 binding，继续在 H 上处理请求 —— **migration 是 opportunistic，不是 mandatory**。
|
||||
|
||||
#### §4.3.3 Mechanism
|
||||
|
||||
Migration 触发时：
|
||||
|
||||
1. 当前 request 直接重定向到 target instance T
|
||||
2. session 累计的 KV 状态从 source H 通过 Mooncake `kv_connector` 传输到 T
|
||||
3. session 的 host binding 更新为 T；后续 turn 按 affinity 自动路由到 T
|
||||
|
||||
KV transfer 发生在触发该 migration 的 request 的 critical path 上，但被该 session 剩余的 TBD turn 摊销。
|
||||
|
||||
#### §4.3.4 Thrashing prevention
|
||||
|
||||
每个 session 维护 `last_migration_timestamp`。在 cooldown `T_cool` 内被禁止再次 migrate。Cooldown 把 migration 行为限制在 O(session_lifetime / T_cool) 量级。
|
||||
|
||||
### §4.4 Implementation
|
||||
|
||||
基于 vLLM 0.18.1 + Mooncake (vanilla kv_connector)。EAR 是一个 router 进程，~TBD LoC。Session→host 表用 TBD（in-memory dict / Redis）维护。
|
||||
|
||||
---
|
||||
|
||||
## §5 Related Work
|
||||
## §5 Evaluation
|
||||
|
||||
- **Serving systems**：vLLM, Mooncake, SGLang, DistServe, Splitwise。本文基于 vLLM+Mooncake，与
|
||||
DistServe/Splitwise 不同在于**不做静态 P/D 分区**（§3.1 给出 agentic regime 的证否 + crossover 边界）。
|
||||
- **Cache-aware routing**：LMCache, Production-Stack, LMetric (OSDI'26)。本文指出 cache 信号不能折叠进乘性 load
|
||||
score（§3.2 LMetric 稀释分析），须作独立 biased 路由路径。
|
||||
- **KV storage hierarchy / offload（主要对手，正面回应）**：Mooncake Store, LMCache, AttentionStore 等把 KV 下沉
|
||||
到 CPU/SSD/远端 pool。**本文不否认其对 recompute 的收益**（§2.2 实测 remote-RDMA 命中比 recompute 快达 16×，
|
||||
与 Mooncake-Store blog 的 46× 同向）；但论证在 **agentic** 下 (i) 命中层级 GPU ≫ CPU ≫ RDMA-store（§2.2），
|
||||
(ii) 活跃 working set 本就 GPU-resident（§2.2 Ev#1）⇒ 深层级的边际收益低，应优先 GPU 常驻而非建深 hierarchy。
|
||||
- **Stateful migration**：Pollux, Gandiva（RL training 的 migration-as-rebalancing）。本文把该思路迁到 LLM KV
|
||||
cache 的 dedup（§3.3）。
|
||||
### §5.1 Setup
|
||||
|
||||
- **Trace**: 真实 Qwen3-Coder agentic trace，TBD requests / TBD seconds / r=0.0015 st=0.3，peak QPS ~1.6，APC headroom ~76%
|
||||
- **Hardware**: TBD × H20 (96GB HBM)
|
||||
- **Engine**: vLLM 0.18.1 + Mooncake `kv_connector`
|
||||
- **Baselines** (6 个):
|
||||
1. `load-balance` —— round-robin
|
||||
2. `LMetric` —— OSDI'26 load-aware routing
|
||||
3. `kvcache-aware + load-balance` —— linear combination of cache score and load score
|
||||
4. `sticky` —— 硬 session-instance pinning
|
||||
5. `static PD-disagg` —— 4P / 4D 静态分区
|
||||
6. `EAR` —— 本文
|
||||
- **Metrics**: TTFT (mean/p50/p90/p99)、TPOT (同上)、E2E、APC、hotspot index、wall-clock vs trace-time
|
||||
|
||||
### §5.2 End-to-end Performance
|
||||
|
||||
**Figure 6: End-to-end performance** — ✅ (PARTIAL，缺 PD-disagg 列)
|
||||
|
||||
|
||||
|
||||
> **🚧 TBD (NEW DATA)**：上图缺 `static PD-disagg` 那一列；EAR 列也是 TBD（需 migration validation）。要再补一张同样格式但包含全 6 个 baseline 的图。
|
||||
|
||||
| Scheduler | TTFT p50 | TTFT p90 | TPOT p90 | APC | Hotspot idx | Wall-clock factor |
|
||||
|---|---|---|---|---|---|---|
|
||||
| load-balance | TBD | TBD | TBD | TBD | TBD | TBD |
|
||||
| LMetric | TBD | TBD | TBD | 56.9% | 6.53 | ~8x |
|
||||
| kvcache+load | TBD | TBD | TBD | TBD | TBD | TBD |
|
||||
| sticky | TBD | 18.02s | TBD | 77.2% | 13.65 | TBD |
|
||||
| static PD-disagg | 62.8s | TBD | TBD | TBD | TBD | TBD |
|
||||
| **EAR** | TBD | **7.35s** | TBD | **79.4%** | TBD | TBD |
|
||||
|
||||
(粗体数字来自现有 "unified" 原型测量。)
|
||||
|
||||
### §5.3 Ablation 🚧 PARTIAL DEFER
|
||||
|
||||
我们独立关闭两个 pillar:
|
||||
|
||||
- **EAR (affinity only)**: 等价于 pure sticky；衡量 locality 单独贡献
|
||||
- **EAR (migration only)**: cold-balance + reactive migration，无 affinity；衡量 migration 能否独立成立
|
||||
- **EAR (full)**: 两个 pillar 都开
|
||||
|
||||
**Figure 7: Ablation** — `figs/f7_ablation.png` **🚧 TBD — DEFERRED (BLOCKED ON MIGRATION VALIDATION)**
|
||||
> 完整 ablation 需要 migration-only / both / affinity-only 三个配置。Migration-only 和 both 都依赖 migration 重测。现阶段可先做 affinity-only vs load-balance 的两点对比（已有数据：unified 79.4% APC vs lmetric 56.9% APC）。
|
||||
|
||||
预期结论：affinity-only 拿到 locality 但 interference 翻倍；migration-only 抓不住 locality；两者都必须。
|
||||
|
||||
### §5.4 Dispatch Coupling Validation
|
||||
|
||||
闭环 §2.3 的论证。对每个 baseline 测量：
|
||||
|
||||
- 单 turn 平均服务时间 `W_turn`（x 轴）
|
||||
- Wall-clock / trace-time amplification（y 轴）
|
||||
|
||||
**Figure 8: Wall-clock amplification vs per-turn service time** — `figs/f8_coupling_measured.png` **🚧 TBD (NEW DATA)**
|
||||
> 散点：x = 平均 per-turn `W_turn`（从 per-request metrics 算 TTFT + decode_time），y = amplification (`wall_clock / trace_span`，Phase 1 patch 已暴露)。每个 baseline 一个点。理论曲线 `L*/(1 − Λ·N·W'(L*))` 叠加（可选）。这是 §2.3 论证的实证 closure，**优先级最高**。
|
||||
|
||||
预期：EAR 在 `W_turn` 最小且放大系数最低的角上。
|
||||
|
||||
### §5.5 Sensitivity
|
||||
|
||||
| 参数 | 范围 | 检验 |
|
||||
|---|---|---|
|
||||
| 到达率 λ | TBD | EAR 在低/高负载下是否稳定 dominate |
|
||||
| Skew 程度 (Zipf α) | TBD | sticky 与 EAR 的差距是否随 skew 拉开 |
|
||||
| KV pool size | TBD | static PD-disagg 撞墙边界 |
|
||||
| `T_hot` (migration threshold) | TBD | 触发太宽 → thrash，太严 → 错过 |
|
||||
| `T_cool` (cooldown) | TBD | 同上 |
|
||||
|
||||
**Figure 9: Sensitivity heatmaps** — `figs/f9_sensitivity.png` **🚧 TBD (NEW DATA, PARTIAL DEFER)**
|
||||
> Arrival rate / skew / KV pool size 这三轴可现在做（不依赖 migration）；`T_hot` / `T_cool` 两轴依赖 migration validation，deferred。
|
||||
|
||||
### §5.6 Migration Microbenchmark 🚧 FULL DEFER
|
||||
|
||||
刻画 EAR 内部 migration 行为：
|
||||
|
||||
- Migration 触发率（% of requests）
|
||||
- 平均 KV transfer 时间
|
||||
- Migration accuracy：迁移后 target instance 在接下来 TBD 个 turn 内保持非 hot 的比例
|
||||
- Thrashing rate：cooldown 窗口内多次迁移的 session 占比（应为 0）
|
||||
|
||||
**Figure 10: Migration timeline** — `figs/f10_migration_timeline.png` **🚧 TBD — DEFERRED (BLOCKED ON MIGRATION VALIDATION)**
|
||||
> 时间轴上每个 instance 的 pending prefill tokens heatmap，migration 事件以箭头标出。完全依赖 migration 重测。
|
||||
|
||||
---
|
||||
|
||||
## §6 Conclusion
|
||||
## §6 Discussion and Limitations
|
||||
|
||||
对 agentic LLM 负载，用户感受的 request latency / TPS / GPU util 由 **KV 命中是否在 GPU HBM 上** 主导。命中层级
|
||||
`GPU ≫ CPU > RDMA-store ≫ recompute` 且代价差随 context 拉大（§2.2），而活跃 working set 小到本就 GPU-resident
|
||||
（§2.2 Ev#1）。**GPU-hit-first** 原则由此统一三件事：复活 PD-colocation（§3.1）、在路由里做 biased
|
||||
KV-awareness（§3.2）、用 migration 而非 replication 做 GPU 去重（§3.3）。
|
||||
- **Extreme skew**: 若单个 session 自己就把任意 instance 撑成 hot，EAR 退化为 sticky。我们未在该 regime 做 stress test。
|
||||
- **Cost model accuracy**: EAR 用 observable load 绕过了预测误差问题。但未来若引入 predictive admission control，需要解决 mooncake cost model 10-21x 误差。
|
||||
- **Heterogeneous hardware / multi-model**: EAR 假设 instance 同质。混合模型 / 混合 GPU 池需要扩展 binding 模型。
|
||||
- **Per-instance batch tuning (future)**: 动态调整 `max_batched_tokens` 可能进一步降低 instance 内部 prefill-decode 干扰，留作 future work。
|
||||
|
||||
---
|
||||
|
||||
## §7 Related Work
|
||||
|
||||
- **LLM serving systems**: vLLM, Mooncake, SGLang, DistServe, Splitwise. EAR 基于 vLLM + Mooncake 实现，与 DistServe/Splitwise 不同之处在于不做静态 P/D 分区。
|
||||
- **Cache-aware routing**: LMCache, Production-Stack, LMetric (OSDI'26)。这些工作最小化 cross-instance cache miss，但不迁移状态。
|
||||
- **Stateful service migration**: Pollux, Gandiva (RL training)。EAR 借鉴 migration-as-rebalancing 思路，将其迁移到 LLM inference 的 KV cache 场景。
|
||||
|
||||
---
|
||||
|
||||
## §8 Conclusion
|
||||
|
||||
对 agentic LLM workload，locality 是主导调度杠杆。EAR 用 session-affinity routing 抓住它，用 hot-triggered session migration 保护它，单一方案在 TTFT、APC、hotspot、wall-clock throughput 四个维度同时 dominate 五个 baseline。
|
||||
|
||||
---
|
||||
|
||||
## Work Plan
|
||||
|
||||
### ✅ Done
|
||||
- §1/§2 背景与 metric 论证；dispatch-coupling 数学；inter-turn gap CDF（`f3a`）
|
||||
- §2.1 reuse topology / C1–C3（`f2a`、`workload_chars/*`）
|
||||
- **§2.2 四层命中实测（`v2/exp_a_tier_latency`：GPU/CPU/RDMA/miss）**
|
||||
- **§2.2 Ev#1 容量 knee（`v2/exp_b_capacity_knee`）+ working_set + cluster-scale 校正**
|
||||
- §3.1 PD-disagg 证否（`PD_DISAGG_RESULTS`、crossover、MB1/MB2）
|
||||
- §3.2 LPWL（−31%）、LMetric 稀释、sticky hot-pin、ES ablation
|
||||
- §3.3 migration substrate net-positive + correctness smoke tests
|
||||
|
||||
### 🟢 不依赖 migration，现在可做
|
||||
- §2.0 wall-clock amplification sweep（5 baseline × ≥3 runs）— **优先级最高**
|
||||
- §4 集成系统命名 + 端到端 baseline 矩阵（含 static PD-disagg 列）
|
||||
- §4.5 λ / skew / KV pool sensitivity
|
||||
- 草拟 §1–§3 正文（证据/图已齐）
|
||||
- [x] §1 anchor sentence + contribution bullets
|
||||
- [x] §2 outline + reuse existing characterization figures (`f2a`/`f2b`/`f2c`)
|
||||
- [x] §3.1/§3.2/§3.3 outline + reuse existing baseline failure figures (`f4a`/`f4b`/`f4c`/`f4d`)
|
||||
- [x] §4 design description (§4.3 待实证)
|
||||
- [x] §5.2 partial figure (`f6` 5/6 baselines)
|
||||
- [x] `replayer/replay.py` patched to emit `trace_span_s` + `amplification` in summary
|
||||
|
||||
### 🚧 Deferred（待 migration policy e2e）
|
||||
- §3.3 migration trigger + target 的反馈环收益验证
|
||||
- §4.3 full / migration-only ablation
|
||||
- §4.5 `T_hot` / `T_cool` sensitivity
|
||||
### 🟢 Can do without migration (paper writing now possible)
|
||||
|
||||
### 🎨 待画
|
||||
- §1.3 storage-hierarchy 示意（GPU HBM → CPU DRAM → RDMA store）
|
||||
- §2.0 dispatch-coupling schematic（chatbot vs agentic timeline + 反馈环）
|
||||
- 集成系统 architecture 图
|
||||
- [ ] Draft §1-§4 正文（数据全有，figures 已 copy 完）
|
||||
- [ ] §2.3 dispatch coupling 那一节的正文 draft（数学已经在 conversation 里推完）
|
||||
- [ ] §3 三个失败模式正文 draft
|
||||
- [ ] §5.4 wall-clock amplification 实测（5 baseline × ≥3 runs）— **优先级最高**，这是 §2.3 的实证 closure
|
||||
- [ ] §5.2 把 static PD-disagg 补进 `f6` 那张图（重跑或合并现有 PD-sep 数据）
|
||||
- [ ] §5.5 sensitivity 的 λ / skew / KV pool 三轴
|
||||
- [ ] §3 三张子图各自独立的 latex/markdown layout 决定
|
||||
|
||||
### ❓ Open
|
||||
- 集成系统最终命名（GPU-hit-first 是原则；系统名待定）
|
||||
- §4 instance 数 / trace 总长定稿
|
||||
### 🚧 Deferred (待 migration validation)
|
||||
|
||||
- [ ] §4.3 migration mechanism 重测（`connector_tax` DR-fix 之后跑）
|
||||
- [ ] §5.3 full ablation (migration-only + both 两个配置)
|
||||
- [ ] §5.5 `T_hot` / `T_cool` 两轴 sensitivity
|
||||
- [ ] §5.6 migration microbench 全部
|
||||
- [ ] §1 teaser 图 (`f1`) EAR 那一列
|
||||
- [ ] §5.2 表里 EAR 那一行
|
||||
- [ ] §4.3.1 / §4.3.4 的 `T_hot` 和 `T_cool` 取值
|
||||
|
||||
### 🎨 Custom drawings (paper-writing 阶段)
|
||||
|
||||
- [ ] `f3_coupling_schematic.png` —— chatbot vs agentic timeline + 反馈环
|
||||
- [ ] `f5_architecture.png` —— EAR 组件图
|
||||
|
||||
### ❓ Open design decisions
|
||||
|
||||
- [ ] §4.4 session→host 表的存储介质（in-memory dict vs Redis）
|
||||
- [ ] §5.1 instance 数量、trace 总长度的最终定稿
|
||||
|
||||
77
README.md
77
README.md
@@ -1,77 +0,0 @@
|
||||
# agentic-kv
|
||||
|
||||
Serving agentic LLM workloads by keeping the KV working set in GPU HBM
|
||||
(GPU-hit-first). Research outline: [`PAPER_OUTLINE.md`](PAPER_OUTLINE.md).
|
||||
Evidence + experiments: [`v2/`](v2/).
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Benchmarking methodology — read this first
|
||||
|
||||
> **Replay agentic traces with `--dispatch-mode thinktime`, not the default
|
||||
> `tracets`.** It is the faithful, more realistic load — and the dispatch mode
|
||||
> materially changes the performance you measure.
|
||||
|
||||
The replayer offers two ways to time each turn:
|
||||
|
||||
| mode | turn-k dispatched at | what it models |
|
||||
|---|---|---|
|
||||
| `tracets` (default) | `max(prev_turn_finished, trace_ts)` | absolute production schedule |
|
||||
| **`thinktime` (use this)** | `prev_turn_finished + time_to_parent_chat` | real closed-loop agent pacing |
|
||||
|
||||
**Why it matters.** `tracets` collapses the inter-turn think-time to ~0 whenever
|
||||
the system falls behind (it fires the next turn immediately because the trace
|
||||
timestamp is already in the past). That manufactures **artificial request
|
||||
bursts** — spiking instantaneous concurrency → KV-pool pressure → preemption →
|
||||
inflated tail latency and wasted throughput. `thinktime` keeps each turn's real
|
||||
gap (tool-exec + agent think), so the offered load is what a real agent produces.
|
||||
|
||||
**Measured (w600 first-300s window, 8×H20, round-robin, 100% completion):**
|
||||
|
||||
| metric (N=8) | `tracets` (Mode 1) | **`thinktime` (Mode 2)** | Δ |
|
||||
|---|---:|---:|---:|
|
||||
| E2E p90 | 102.8 s | **73.5 s** | **−28%** |
|
||||
| E2E p99 | 245 s | **227 s** | −7% |
|
||||
| TTFT p90 | 56.1 s | **39.7 s** | **−29%** |
|
||||
| system TPS | 111.8 | **119.3** | **+7%** |
|
||||
| wall-clock | 967 s | **787 s** | −19% |
|
||||
| TPOT p90 | 0.174 s | 0.188 s | ~flat |
|
||||
|
||||
So under realistic capacity, `tracets` makes the system look **~30% worse on
|
||||
tail latency** than it actually is. Tell-tale: scaling 6→8 instances barely helped
|
||||
`tracets` (975→967 s — its bursts re-saturate regardless of capacity) but helped
|
||||
`thinktime` a lot (1125→787 s). Under heavy saturation (N=6) the two converge
|
||||
(E2E p90 ≈ 118–120 s), since there is no slack for bursts to harm. Decode (TPOT)
|
||||
is dispatch-independent everywhere.
|
||||
|
||||
**Recommendation:** benchmark with `--dispatch-mode thinktime`; use `tracets`
|
||||
only as an explicit bursty stress case. Full ablation:
|
||||
[`v2/exp_c_dispatch_ablation/`](v2/exp_c_dispatch_ablation/).
|
||||
|
||||
### How to use it
|
||||
|
||||
```bash
|
||||
# 1. annotate a trace with the real per-turn gap (one-time; scans the raw trace)
|
||||
python scripts/add_time_to_parent.py traces/w600_r0.0015_st30.jsonl traces/w600_ttp.jsonl
|
||||
|
||||
# 2. replay closed-loop with faithful think-time
|
||||
python -m replayer --trace traces/w600_ttp.jsonl --endpoint <eps> \
|
||||
--model <model> --dispatch-mode thinktime
|
||||
```
|
||||
|
||||
`time_to_parent_chat = this_turn.request_ready_time_ms − parent_turn.request_end_time_ms`,
|
||||
computed from the raw trace and stored per request; turn-1 has none (fires at its
|
||||
trace arrival). Traces without the field fall back to `tracets`.
|
||||
|
||||
---
|
||||
|
||||
## Project map
|
||||
|
||||
- [`PAPER_OUTLINE.md`](PAPER_OUTLINE.md) — GPU-hit-first paper outline (the thesis).
|
||||
- [`v2/`](v2/) — evidence experiments:
|
||||
- `exp_a_tier_latency/` — KV-hit cost by tier (GPU < CPU-local < remote-RDMA < miss).
|
||||
- `exp_b_capacity_knee/` — realized APC / latency knee vs GPU capacity.
|
||||
- `exp_c_dispatch_ablation/` — the replay-mode study above.
|
||||
- `replayer/` — trace replayer (`--dispatch-mode`, closed-loop think-time).
|
||||
- `scripts/add_time_to_parent.py` — trace annotation for `thinktime`.
|
||||
- `microbench/`, `analysis/` — PD-disagg, routing, workload characterization.
|
||||
@@ -1,134 +0,0 @@
|
||||
# 目前已成立的结论（2026-05-27）
|
||||
|
||||
EAR 项目目前能用实测数据支撑的论点汇总。每条都标了对应的图/数据路径。
|
||||
|
||||
---
|
||||
|
||||
## 1. Workload 性质（§2）
|
||||
|
||||
Production trace = Qwen3-Coder agentic，1.3 M sessions / 2.1 M reqs / 7200 s。
|
||||
|
||||
| 性质 | 数据 | 实证 |
|
||||
|---|---|---|
|
||||
| **KV 复用几乎全在 session 内** | intra 93.2% / cross 5.7% / shared 1.1%；理论 APC 上界 79.6% | `figs/f2a_reuse_topology.png` |
|
||||
| **Session 极度偏斜** | top 1%/5%/10%/25%/50% = 46.5%/66.5%/74.6%/87.5%/**96.0%** input mass | `figs/f2b_session_skew.png` |
|
||||
| **单请求 KV 已经很大** | p50 1.8 GiB / p90 8.0 / p95 9.6 / **p99 11.5 GiB**；KV pool 38 GiB/instance（0.4 × H20 96 GiB）→ p99 req 只能装 **3 个/instance** | `figs/f2c_kv_footprint_cdf.png` |
|
||||
|
||||
**结论**：cache 是 session-local 的，scheduling 必须保留 session affinity；单 request KV 接近 pool 上限，**PD-disagg 4P+4D 让系统 decode 容量直接减半**。
|
||||
|
||||
## 2. Dispatch Coupling（§2.3）
|
||||
|
||||
| 数据 | Agentic (Qwen3-Coder) | Chatbot (qwen3-max) |
|
||||
|---|---:|---:|
|
||||
| Inter-turn `T_external` p50 | **1.6 s** | 7.2 s |
|
||||
| `gap < 1 s` 比例 | **39%** | 4% |
|
||||
| `gap < 5 s` 比例 | 67% | 29% |
|
||||
| p99 | 738 s | 43 s |
|
||||
|
||||
参考图：`figs/f3a_inter_turn_gap.png`。
|
||||
|
||||
**结论**：agentic 有一段 chatbot 没有的 **sub-second tool-call mode**（39% vs 4%）。当 `W_turn ≳ T_external`（任何 W_turn > 1 s 的 scheduler 在 agentic 上都满足这条件），Little's Law `L = Λ · N · (W_turn(L) + T_external)` 进入闭环 regime，scheduler 的 ε 退步通过 KV 竞争反馈环被放大成 wall-clock 数倍差距。**实测**：lmetric 跑 600 s trace 用 49 min wall-clock = **8x amplification**。
|
||||
|
||||
## 3. 现有调度的三类失败（§3）
|
||||
|
||||
| Baseline | 失败模式 | 数据 |
|
||||
|---|---|---|
|
||||
| **load-balance / LMetric** | 丢 locality | lmetric APC **56.9%**（vs 上界 79.6%）；LMetric 比 load_only 只好 +3.3pp，因为 cache 信号在乘性 score `(pending+input−hit) × num_req` 里被 num_req 吞掉 |
|
||||
| **静态 PD-disagg** | D 侧 KV 容量墙 + transfer 成本 | 见 §4 cost-vs-benefit |
|
||||
| **Pure sticky** | 全员被 hot session 拖累，不是单一热点 | sticky median worker 20.3 s vs unified 10.3 s；system e2e p90 sticky 34.6 s vs unified 18.0 s（**用 max/median ratio 衡量是误导**，§3.3 用 absolute per-worker latency）|
|
||||
|
||||
参考图：`figs/f4a_apc_loss.png`、`figs/f4b_pdsep_kv_wall.png`、`figs/f4c_per_worker_ttft.png`、`figs/f6_e2e_latency_bars.png`、`figs/f6_e2e_latency_full_grid.png`。
|
||||
|
||||
## 4. Static PD-disagg 为什么失败（§3.2）—— 容量问题，不是 cost-benefit 问题
|
||||
|
||||
⚠ **2026-05-27 纠正**：本节前一版本论证"PD-disagg 因为 transfer cost > phase isolation benefit 而失败"。**这个论证算错了**。正确的 phase-isolation benefit 是**每个 prefill 事件 × D 个 concurrent stream** 的总和（≈ `D × T_prefill`），不是单个 request 的 decode 时长。用正确公式，PD-disagg 在 phase-isolation 这一维上**赢 colo 一两个数量级**。Static PD-disagg 在 agentic 上失败的**真正根因是 D 侧 KV pool 容量**，不是这一维。
|
||||
|
||||
### 4.1 真正的失败模式：D 侧 KV 容量天花板
|
||||
|
||||
| | 8C colo | 4P+4D PD-disagg |
|
||||
|---|---:|---:|
|
||||
| Per-D-instance KV pool（0.4 × 96 GiB） | 38 GiB | 38 GiB |
|
||||
| 系统总 decode 容量（D 实例数 × 单池） | 8 × 38 = **304 GiB** | 4 × 38 = **152 GiB** |
|
||||
| p99 单请求 KV = 11.5 GiB → 最多并发 decode | 24 | **12（减半）** |
|
||||
|
||||
Colleague 4P+4D 实测：TTFT p50 0.91 s → **62.8 s（62×）**、success rate **99.5% → 52%**。失败模式：**D 池溢出 + 排队**，不是 transfer 延迟。
|
||||
|
||||
参考图：`figs/f4b_pdsep_kv_wall.png`（pdf 版本是高质量 paper figure）。
|
||||
|
||||
### 4.2 MB2 — KV transfer cost（per-request 一次性成本，**不**是 dominant cost）
|
||||
|
||||
dash1 GPU 0+1（intra）和 dash1 ↔ dash2（inter, 200 Gbps RoCE）扫 9 个 size × 5 reps。
|
||||
|
||||
| 路径 | 稳态带宽（≤ 3 GiB） | p99 agentic 请求 11.5 GiB transfer |
|
||||
|---|---|---|
|
||||
| Intra-node | **9.7 GB/s** | p50 **1.9 s** · max 10 s |
|
||||
| Inter-node | **10.0 GB/s**（差 <3%） | p50 **1.7 s** · max 9.2 s |
|
||||
|
||||
**新发现**：intra/inter 几乎重合 → **Mooncake `batch_transfer_sync_write` 永远走 RDMA NIC**，不走 NVLink。200 Gbps NIC 是天花板。**PD-disagg transfer cost 与拓扑无关**。
|
||||
|
||||
参考图：`figs/mb2_transfer_time_compare.png`、doc `analysis/mb2/README.md`。
|
||||
|
||||
### 4.3 MB1 — Phase interference（PD-disagg 的潜在 benefit 上界）
|
||||
|
||||
dash1 GPU 0 单 instance（无 kv_connector），chunked-prefill 默认开启，D × P 扫描。D=8 结果：
|
||||
|
||||
| Prefill | T_prefill | per-stream TPOT during | penalty |
|
||||
|---|---:|---:|---:|
|
||||
| 2k tok | 143 ms | 32 ms | 4× |
|
||||
| 32k tok | 4.5 s | 388 ms | **52×** |
|
||||
| 131k tok | 57 s | 1419 ms | **183×** |
|
||||
|
||||
**Decode 在 prefill 期间被几乎完全 halted**，单 stream 损失 ≈ `T_prefill` 的时间。**每个 prefill event 总 decode 损失 ≈ `D × T_prefill`**。
|
||||
|
||||
参考图：`figs/mb1_interference.png`、doc `analysis/mb1/README.md`。
|
||||
|
||||
### 4.4 联合 cost-benefit（per-prefill event）
|
||||
|
||||
| Prefill (KV size) | T_prefill | Cost = T_transfer | Benefit = D × T_prefill (D=8) | Cost / Benefit |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 2k tok (192 MiB) | 0.14 s | 8 ms | 1.1 s | **0.7%** |
|
||||
| 33k tok (3 GiB, trace mean) | 4.5 s | 0.32 s | 36 s | **0.9%** |
|
||||
| 125k tok (12 GiB, ~p99) | 57 s | 1.9 s | 456 s | **0.4%** |
|
||||
|
||||
**PD-disagg 在 phase-isolation 这一维赢 100×–250×**。但**这不是 §3.2 该用的论证**，因为 §3.2 真正的 dominant failure 是 §4.1 的 D 池容量天花板（颠覆了上面的全部数学）。
|
||||
|
||||
**总结**：
|
||||
- D 侧 KV 容量天花板（§4.1）→ PD-disagg 在 agentic 上**结构性失败**。
|
||||
- MB1 + MB2 的合计 cost-benefit 在 phase isolation 维度上 PD-disagg 是赢的，**但这件事被容量天花板压倒**。
|
||||
- Paper §3.2 论证应该聚焦"D 池装不下"，MB1/MB2 数据用作 supporting context（per-request transfer charge 量化、phase isolation benefit 量化）而不是 main argument。
|
||||
|
||||
> ✅ **2026-05-30 更新 — 干净栈三轴 ablation 证实本节、并加 regime 细化。**
|
||||
> 本节的容量论点（D 池容量天花板 / decode 减半）在修复 `e13391e` 污染后的 clean stack
|
||||
> 上**得到确认**：concurrency 轴 N=64 时 PD 倾覆，**producer APC 从 71% 崩到 1.4%、TPS −30%**，
|
||||
> 而 colo 线性 scale（Fig 3）。**细化**：PD 并非"在 agentic 上一律失败"——它在
|
||||
> *低负载 / decode-heavy / 低复用* 区间**赢** colo；真正失败的是 agentic corner（高复用 +
|
||||
> 短输出 + 大上下文 + 高并发）——静态 P:D split 无法同时给出复用所需的 producer 容量
|
||||
> *和* decode 容量，而 colo 的弹性池两者兼得。
|
||||
> **另注**：旧 MB5 文档（`PD_DISAGG_RESULTS.md` §6）"session-affinity 救不了 PD / PD 复用=0%"
|
||||
> 的结论是 `e13391e`（producer 每次 KV 传输后 evict prefix）的**污染产物，已撤回**；
|
||||
> 干净栈上 session 路由的 producer APC 与 colo 持平（71–82%）。
|
||||
> 图：[`figs/mb5_pd_ablation/`](figs/mb5_pd_ablation/)。
|
||||
|
||||
## 5. EAR 设计的实证状态（§4）
|
||||
|
||||
| Pillar | 已实证 | 待实证 |
|
||||
|---|---|---|
|
||||
| **Affinity-default routing** (Pillar 1) | ✅ Current `unified` 算法 = LMetric + high-cache affinity；APC **79.4%**（达到 79.6% 上界 97%），TTFT p90 **7.3 s**，median worker p90 **10.3 s** | — |
|
||||
| **Hot-triggered session migration** (Pillar 2) | substrate 已通：`kv_both` connector 在 trace replay 上 net positive（TTFT p90 −18.6%，DR-fix 后 −36.6%），原 elastic_migration_v2 paper 的 "+45% kv_both penalty" obsolete | e2e 策略层（trigger 阈值 + target selection 在反馈环里）未直接验证 |
|
||||
|
||||
## 6. 已经能写的 paper 主张（按 confidence 排序）
|
||||
|
||||
1. **Agentic vs chatbot 在调度上是不同 regime**（dispatch coupling + sub-second tool-call mass）—— 实证完整
|
||||
2. **三类现有调度 baseline 各自的失败模式**（load-balance / static PD-disagg / pure sticky）—— 实证完整
|
||||
3. **Static PD-disagg 在 agentic 下失败的 dominant 根因是 D 侧 KV 容量**（不是 phase-isolation cost-benefit）—— 实证完整（`f4b` + colleague 4P+4D 数据）
|
||||
4. **Mooncake transfer cost 拓扑无关**（intra ≈ inter，~9.7 GB/s NIC 上限）—— 实证完整（MB2）
|
||||
5. **Phase isolation interference 在 chunked-prefill on 下仍然显著**（per-stream TPOT during prefill 15×–2000× baseline）—— 实证完整（MB1）。**注意**：这条数据本身不直接论证 "PD-disagg 失败"，因为算正确账后 PD-disagg 反而在这一维上赢；它的用途是给 §3.2 提供 phase-isolation benefit 上界的量化。
|
||||
6. **Affinity-default 调度（current unified）达到 APC 上界**，per-worker latency 也压倒 sticky —— 实证完整
|
||||
7. **Hot-triggered migration 修复 sticky 的 hot pin** —— design 完整、e2e 待验证
|
||||
|
||||
## 7. 待做
|
||||
|
||||
- **MB3-5**（end-to-end PD-disagg deployment）：D-pool runtime occupancy、cache reuse × PD interaction、PD ratio sweep。这些是 §5 完整实验矩阵的事
|
||||
- **EAR Pillar 2 migration e2e validation**（在 connector_tax DR-fix 之上重测）
|
||||
- **§5.4 wall-clock amplification sweep**（5 baseline × 3 runs，钉死 dispatch coupling 论证的实证 closure）
|
||||
- **Scale-out 验证**（dash1+dash2 = 16 GPU，等 dash0 + 3-node 可用时扩到 80 GPU）
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -48,8 +48,6 @@ def fig_b3_apc_vs_hotspot(comp: dict, upper: dict, out: Path) -> None:
|
||||
fig, ax = plt.subplots(figsize=(6, 4.5))
|
||||
for r in comp["rows"]:
|
||||
pol = r["policy"]
|
||||
if pol not in POLICY_ORDER:
|
||||
continue
|
||||
ax.scatter(r["apc_ratio"] * 100, r["hotspot_index_ttft_p90"],
|
||||
s=180, color=POLICY_COLOR.get(pol, "gray"), label=pol,
|
||||
edgecolors="black", linewidths=0.5)
|
||||
@@ -89,48 +87,6 @@ def fig_b3_latency_bars(comp: dict, out: Path) -> None:
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def fig_b3_latency_full_grid(results_dir: Path, out: Path) -> None:
|
||||
"""4 rows (mean / p50 / p90 / p99) × 3 cols (TTFT / TPOT / E2E) per policy.
|
||||
|
||||
Reads per-policy metrics.summary.json caches under raw_stats/, which
|
||||
expose mean alongside the percentiles (b3_policy_comparison.json does
|
||||
not record mean).
|
||||
"""
|
||||
raw_dir = results_dir / "raw_stats"
|
||||
pols = [p for p in POLICY_ORDER if (raw_dir / f"{p}.json").exists()]
|
||||
if not pols:
|
||||
return
|
||||
stats = {p: json.loads((raw_dir / f"{p}.json").read_text()) for p in pols}
|
||||
|
||||
rows = [("mean", "mean"), ("p50", "p50"), ("p90", "p90"), ("p99", "p99")]
|
||||
cols = [
|
||||
("TTFT (s)", "ttft", 1.0),
|
||||
("TPOT (ms)", "tpot", 1000.0),
|
||||
("E2E (s)", "e2e", 1.0),
|
||||
]
|
||||
fig, axes = plt.subplots(len(rows), len(cols), figsize=(11, 11), sharex=True)
|
||||
for i, (row_label, agg_key) in enumerate(rows):
|
||||
for j, (col_label, metric_key, scale) in enumerate(cols):
|
||||
ax = axes[i][j]
|
||||
vals = [stats[p][metric_key][agg_key] * scale for p in pols]
|
||||
ax.bar(pols, vals,
|
||||
color=[POLICY_COLOR.get(p, "gray") for p in pols],
|
||||
edgecolor="black", linewidth=0.5)
|
||||
for k, v in enumerate(vals):
|
||||
ax.text(k, v, f"{v:.1f}", ha="center", va="bottom", fontsize=8)
|
||||
if j == 0:
|
||||
ax.set_ylabel(row_label, fontsize=11)
|
||||
if i == 0:
|
||||
ax.set_title(col_label, fontsize=11)
|
||||
ax.grid(alpha=0.3, axis="y")
|
||||
ax.tick_params(axis="x", rotation=20, labelsize=9)
|
||||
ax.margins(y=0.18)
|
||||
fig.suptitle("B3 latencies per policy — mean / p50 / p90 / p99")
|
||||
fig.tight_layout()
|
||||
fig.savefig(out, dpi=120)
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def fig_b3_apc_vs_upper(comp: dict, upper: dict, out: Path) -> None:
|
||||
by = {r["policy"]: r for r in comp["rows"]}
|
||||
pols = [p for p in POLICY_ORDER if p in by]
|
||||
@@ -189,13 +145,7 @@ def fig_b3_failure_breakdown(comp: dict, out: Path) -> None:
|
||||
|
||||
|
||||
def fig_b3_per_worker_ttft(results_dir: Path, comp: dict, out: Path) -> None:
|
||||
"""Per-worker TTFT p90 grouped bars; title shows median + max worker p90.
|
||||
|
||||
We deliberately do NOT report a max/median 'hotspot index' here: it is a
|
||||
ratio and treats unified (most workers fast, one hot) as worse than
|
||||
sticky (all workers slow), which inverts the actual user-facing p90.
|
||||
"""
|
||||
import statistics
|
||||
"""Per-worker TTFT p90 grouped bars; reads each policy's hotspot_index.json."""
|
||||
by = {r["policy"]: r for r in comp["rows"]}
|
||||
pols = [p for p in POLICY_ORDER if p in by]
|
||||
fig, axes = plt.subplots(1, len(pols), figsize=(3 * len(pols), 4),
|
||||
@@ -216,12 +166,8 @@ def fig_b3_per_worker_ttft(results_dir: Path, comp: dict, out: Path) -> None:
|
||||
edgecolor="black", linewidth=0.5)
|
||||
for i, v in enumerate(vals):
|
||||
ax.text(i, v, f"{v:.1f}", ha="center", va="bottom", fontsize=8)
|
||||
median_v = statistics.median(vals)
|
||||
max_v = max(vals)
|
||||
ax.set_title(
|
||||
f"{pol}\nmedian {median_v:.1f}s · max {max_v:.1f}s",
|
||||
fontsize=10,
|
||||
)
|
||||
ax.set_title(f"{pol}\nhotspot={by[pol]['hotspot_index_ttft_p90']:.2f}",
|
||||
fontsize=10)
|
||||
ax.tick_params(axis="x", labelsize=8)
|
||||
ax.grid(alpha=0.3, axis="y")
|
||||
axes[0].set_ylabel("worker TTFT p90 (s)")
|
||||
@@ -308,67 +254,19 @@ def fig_reuse_decomposition(reuse: dict, out: Path) -> None:
|
||||
|
||||
|
||||
def fig_kv_footprint_cdf(kv: dict, out: Path) -> None:
|
||||
"""How many concurrent decodes fit per percentile, under three deployments.
|
||||
|
||||
KV pool assumption: 96 GiB H20 HBM split ~50% model params (Qwen3-Coder-
|
||||
30B-A3B in bf16 + headroom), ~10% runtime activations, leaving ~40% for
|
||||
the KV cache pool — i.e. ~38.4 GiB per instance.
|
||||
|
||||
For each request-size percentile, we report system-wide concurrent
|
||||
decode capacity = N_D × floor(KV_pool / req_size_MiB) under three 8-GPU
|
||||
deployments: all-combined, 4P+4D, 6P+2D. The point is that going from
|
||||
combined 8C to 4P+4D halves the system's decode population at the
|
||||
same per-request KV pressure.
|
||||
"""
|
||||
s = kv.get("kv_mib_per_request") or {}
|
||||
pct_keys = ["p50", "p90", "p95", "p99"]
|
||||
req_mib = [float(s.get(k, 0.0)) for k in pct_keys]
|
||||
req_gib = [v / 1024 for v in req_mib]
|
||||
|
||||
hbm_gib = 96.0
|
||||
kv_pool_frac = 0.40
|
||||
kv_pool_mib = hbm_gib * kv_pool_frac * 1024 # ≈ 39322 MiB per instance
|
||||
|
||||
deploys = [
|
||||
("Combined 8C", 8, "#2ca02c"),
|
||||
("PD-disagg 4P+4D", 4, "#ff7f0e"),
|
||||
("PD-disagg 6P+2D", 2, "#d62728"),
|
||||
]
|
||||
|
||||
import numpy as _np
|
||||
x = _np.arange(len(pct_keys))
|
||||
bar_w = 0.26
|
||||
|
||||
fig, ax = plt.subplots(figsize=(9, 5.2))
|
||||
for i, (label, n_d, color) in enumerate(deploys):
|
||||
per_inst = [int(kv_pool_mib // r) if r > 0 else 0 for r in req_mib]
|
||||
sys_cap = [n_d * pi for pi in per_inst]
|
||||
bars = ax.bar(x + (i - 1) * bar_w, sys_cap, bar_w,
|
||||
label=f"{label} (N_D={n_d})",
|
||||
color=color, edgecolor="black", linewidth=0.5)
|
||||
for j, (b, n) in enumerate(zip(bars, sys_cap)):
|
||||
ax.text(b.get_x() + b.get_width() / 2, n, str(n),
|
||||
ha="center", va="bottom", fontsize=9, color="#333")
|
||||
|
||||
# Annotate per-request KV size and per-instance fit just above the x-axis
|
||||
per_inst_combined = [int(kv_pool_mib // r) if r > 0 else 0 for r in req_mib]
|
||||
annot = [
|
||||
f"{pct}\n{rg:.1f} GiB / req\nfits {pi}/inst"
|
||||
for pct, rg, pi in zip(pct_keys, req_gib, per_inst_combined)
|
||||
]
|
||||
ax.set_xticks(x)
|
||||
ax.set_xticklabels(annot, fontsize=10)
|
||||
|
||||
ax.set_ylabel("System-wide concurrent decodes")
|
||||
ax.set_title(
|
||||
f"Per-instance KV pool ≈ {kv_pool_mib / 1024:.1f} GiB "
|
||||
f"(0.4 × H20 96 GiB; remaining 0.5 model + 0.1 activation)\n"
|
||||
f"PD-disagg halves the decode population at p90+ "
|
||||
f"(Qwen3-Coder-30B-A3B, 98304 B/token)"
|
||||
)
|
||||
ax.legend(loc="upper right")
|
||||
vals = [s.get(k) for k in ("p50", "p90", "p95", "p99")]
|
||||
labels = ["p50", "p90", "p95", "p99"]
|
||||
fig, ax = plt.subplots(figsize=(6, 3.5))
|
||||
ax.bar(labels, vals, color="#1f77b4", edgecolor="black", linewidth=0.5)
|
||||
for i, v in enumerate(vals):
|
||||
ax.text(i, v, f"{v:.0f} MiB", ha="center", va="bottom", fontsize=9)
|
||||
ax.axhline(95 * 1024, color="red", linestyle="--", alpha=0.5,
|
||||
label="H20 ~95 GiB usable")
|
||||
ax.set_ylabel("KV bytes per request (MiB)")
|
||||
ax.set_title("B1' Per-request KV footprint (Qwen3-Coder-30B-A3B, 98304 B/token)")
|
||||
ax.legend()
|
||||
ax.grid(alpha=0.3, axis="y")
|
||||
ax.margins(y=0.15)
|
||||
fig.tight_layout()
|
||||
fig.savefig(out, dpi=120)
|
||||
plt.close(fig)
|
||||
@@ -378,17 +276,9 @@ def main() -> None:
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--results-dir", type=Path, required=True)
|
||||
p.add_argument("--out-dir", type=Path, required=True)
|
||||
p.add_argument("--exclude-policies", default="",
|
||||
help="Comma-separated policies to drop from per-policy figures")
|
||||
args = p.parse_args()
|
||||
args.out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
excluded = {s.strip() for s in args.exclude_policies.split(",") if s.strip()}
|
||||
if excluded:
|
||||
global POLICY_ORDER
|
||||
POLICY_ORDER = [p for p in POLICY_ORDER if p not in excluded]
|
||||
print(f"excluding policies: {sorted(excluded)} -> kept {POLICY_ORDER}")
|
||||
|
||||
comp = _load(args.results_dir, "b3_policy_comparison.json")
|
||||
upper = _load(args.results_dir, "apc_upper_w600.json")
|
||||
b2 = _load(args.results_dir, "b2_sweep_summary.json")
|
||||
@@ -397,9 +287,6 @@ def main() -> None:
|
||||
|
||||
fig_b3_apc_vs_hotspot(comp, upper, args.out_dir / "fig_b3_apc_vs_hotspot.png")
|
||||
fig_b3_latency_bars(comp, args.out_dir / "fig_b3_latency_bars.png")
|
||||
fig_b3_latency_full_grid(
|
||||
args.results_dir, args.out_dir / "fig_b3_latency_full_grid.png"
|
||||
)
|
||||
fig_b3_apc_vs_upper(comp, upper, args.out_dir / "fig_b3_apc_vs_upper.png")
|
||||
fig_b3_failure_breakdown(comp, args.out_dir / "fig_b3_failure_breakdown.png")
|
||||
fig_b3_per_worker_ttft(args.results_dir, comp,
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
{
|
||||
"ttft": {
|
||||
"count": 1214.0,
|
||||
"mean": 5.111546324698484,
|
||||
"p50": 0.9387824369769078,
|
||||
"p90": 15.671339168207492,
|
||||
"p99": 53.56683189840049
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1214.0,
|
||||
"mean": 0.01757124870168204,
|
||||
"p50": 0.008854518407308914,
|
||||
"p90": 0.02122720699121469,
|
||||
"p99": 0.18280341184277568
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1214.0,
|
||||
"mean": 9.518126648903337,
|
||||
"p50": 2.754255389008904,
|
||||
"p90": 24.8209177934099,
|
||||
"p99": 80.59924928059091
|
||||
}
|
||||
}
|
||||
@@ -1,23 +0,0 @@
|
||||
{
|
||||
"ttft": {
|
||||
"count": 1214.0,
|
||||
"mean": 6.268620166597892,
|
||||
"p50": 1.2609447415161412,
|
||||
"p90": 20.197147866390882,
|
||||
"p99": 52.84285237012196
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1214.0,
|
||||
"mean": 0.02406975794215626,
|
||||
"p50": 0.009231464695980247,
|
||||
"p90": 0.026851662550158716,
|
||||
"p99": 0.3211630676943426
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1214.0,
|
||||
"mean": 11.702793988628443,
|
||||
"p50": 3.58568156149704,
|
||||
"p90": 33.459180271782685,
|
||||
"p99": 93.95083751494239
|
||||
}
|
||||
}
|
||||
@@ -1,23 +0,0 @@
|
||||
{
|
||||
"ttft": {
|
||||
"count": 1214.0,
|
||||
"mean": 5.55315460854824,
|
||||
"p50": 0.5415176274836995,
|
||||
"p90": 18.021296651283045,
|
||||
"p99": 74.09429564891524
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1214.0,
|
||||
"mean": 0.027834537397398284,
|
||||
"p50": 0.008952101894096181,
|
||||
"p90": 0.03641285916619554,
|
||||
"p99": 0.35152006935195085
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1214.0,
|
||||
"mean": 12.109200157184377,
|
||||
"p50": 2.081947358994512,
|
||||
"p90": 34.62592205510591,
|
||||
"p99": 139.68334607904353
|
||||
}
|
||||
}
|
||||
@@ -1,23 +0,0 @@
|
||||
{
|
||||
"ttft": {
|
||||
"count": 1213.0,
|
||||
"mean": 3.2790960856202394,
|
||||
"p50": 0.4997710260213353,
|
||||
"p90": 7.345769894809922,
|
||||
"p99": 42.34170345296613
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1213.0,
|
||||
"mean": 0.012493800538265787,
|
||||
"p50": 0.008079791456705824,
|
||||
"p90": 0.017110194704198407,
|
||||
"p99": 0.12655874612209597
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1213.0,
|
||||
"mean": 6.961301470549104,
|
||||
"p50": 1.7495028690318577,
|
||||
"p90": 18.033410895219994,
|
||||
"p99": 68.80023987947489
|
||||
}
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.1385939740576,
|
||||
"amplification": 1.0013425037685975,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.08144469193556772,
|
||||
"p50": 0.07862715201918036,
|
||||
"p90": 0.08015060934703797,
|
||||
"p99": 0.0875979653932154
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005001699398049616,
|
||||
"p50": 0.004988961030788246,
|
||||
"p90": 0.005045765990923557,
|
||||
"p99": 0.005062779263327164
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 0.3968209869372152,
|
||||
"p50": 0.393534954986535,
|
||||
"p90": 0.39730903680901974,
|
||||
"p99": 0.40925762055674536
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.1604231200181,
|
||||
"amplification": 1.0014153316291596,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.09946277569807849,
|
||||
"p50": 0.09600010397844017,
|
||||
"p90": 0.10452785079833121,
|
||||
"p99": 0.11205230774357905
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005007447102661814,
|
||||
"p50": 0.004987124730611131,
|
||||
"p90": 0.005003212126977151,
|
||||
"p99": 0.005478902989961502
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 0.415208436744531,
|
||||
"p50": 0.41056320699863136,
|
||||
"p90": 0.4200975856045261,
|
||||
"p99": 0.44871115096379066
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.2032543020323,
|
||||
"amplification": 1.0015582278907484,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.10561635944505095,
|
||||
"p50": 0.10468761203810573,
|
||||
"p90": 0.11257308297790587,
|
||||
"p99": 0.12065987563692024
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005328901365752947,
|
||||
"p50": 0.005144592110318915,
|
||||
"p90": 0.005990574603515958,
|
||||
"p99": 0.006688486758013448
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 0.4416300980896939,
|
||||
"p50": 0.42991435900330544,
|
||||
"p90": 0.4854830394731835,
|
||||
"p99": 0.5404117306252005
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"ttft_slo_s": 5.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 303.05708941399644,
|
||||
"amplification": 1.0110793839624128,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 1.674437926279444,
|
||||
"p50": 1.5353219069947954,
|
||||
"p90": 2.0787689138029237,
|
||||
"p99": 3.039117059087727
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.035498316425319934,
|
||||
"p50": 0.02951206674587743,
|
||||
"p90": 0.085249871320677,
|
||||
"p99": 0.15422643764865662
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 3.9111985703531236,
|
||||
"p50": 3.392241114997887,
|
||||
"p90": 7.760864628604043,
|
||||
"p99": 11.30427318874542
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1136,
|
||||
"completion_rate": 0.9734361610968295,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 866.9596620649972,
|
||||
"amplification": 2.8924089607535697,
|
||||
"ttft": {
|
||||
"count": 1136,
|
||||
"mean": 65.09021699308856,
|
||||
"p50": 66.22900710900285,
|
||||
"p90": 112.5535424454938,
|
||||
"p99": 124.55262411334482
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1136,
|
||||
"mean": 0.005710658520121912,
|
||||
"p50": 0.005725543936557461,
|
||||
"p90": 0.005750613698356098,
|
||||
"p99": 0.0058447879207267845
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1136,
|
||||
"mean": 65.4504681098121,
|
||||
"p50": 66.59053339700768,
|
||||
"p90": 112.9150809329949,
|
||||
"p99": 124.91415351489852
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 307.42712411200046,
|
||||
"amplification": 1.0256589867656205,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 3.6233640342625417,
|
||||
"p50": 3.255483777000336,
|
||||
"p90": 6.0935156565916255,
|
||||
"p99": 7.349482456580735
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.006360297526341433,
|
||||
"p50": 0.006324973206372104,
|
||||
"p90": 0.007198417158741947,
|
||||
"p99": 0.007942749238420567
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 4.024414356593621,
|
||||
"p50": 3.6796783979953034,
|
||||
"p90": 6.510242249601289,
|
||||
"p99": 7.7530036393977895
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 0.6563838903170522,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 766
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 0.0,
|
||||
"pd_advantage": 0.0,
|
||||
"n_slo": 0
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 0.20565552699228792,
|
||||
"pd_advantage": 0.3133159268929504,
|
||||
"n_slo": 240
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"ttft_slo_s": 5.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 0.7095115681233933,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 828
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 0.012853470437017995,
|
||||
"pd_advantage": 0.018115942028985508,
|
||||
"n_slo": 15
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 0.7746358183376179,
|
||||
"pd_advantage": 1.0917874396135265,
|
||||
"n_slo": 904
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.1927249849541,
|
||||
"amplification": 1.0015230992770423,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.1367582682356152,
|
||||
"p50": 0.1337889520218596,
|
||||
"p90": 0.13568343156948687,
|
||||
"p99": 0.1417779766954481
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005039774583944486,
|
||||
"p50": 0.005025791809968059,
|
||||
"p90": 0.005088228446667984,
|
||||
"p99": 0.0051380019550329985
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 0.45452996390628697,
|
||||
"p50": 0.45110413199290633,
|
||||
"p90": 0.4555703255813569,
|
||||
"p99": 0.4688122991356063
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.2166367470054,
|
||||
"amplification": 1.0016028753010615,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.1660798433322169,
|
||||
"p50": 0.16653224604669958,
|
||||
"p90": 0.17306958099361508,
|
||||
"p99": 0.1832630861806683
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005046875948133808,
|
||||
"p50": 0.0050270736812510425,
|
||||
"p90": 0.0050664691831029595,
|
||||
"p99": 0.005532995437730161
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 0.48433690918382055,
|
||||
"p50": 0.4845748710213229,
|
||||
"p90": 0.4916891472181305,
|
||||
"p99": 0.5235905526624989
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.2855537990108,
|
||||
"amplification": 1.001832800991369,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.17076528707394278,
|
||||
"p50": 0.16959826194215566,
|
||||
"p90": 0.17923957279417663,
|
||||
"p99": 0.18792458378709853
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005377765550956012,
|
||||
"p50": 0.005177637602808693,
|
||||
"p90": 0.006078562941697855,
|
||||
"p99": 0.007001848420689976
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 0.5098518045703324,
|
||||
"p50": 0.4961305959150195,
|
||||
"p90": 0.5569328068522736,
|
||||
"p99": 0.6314893902023319
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"ttft_slo_s": 5.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 1167,
|
||||
"n_success": 951,
|
||||
"completion_rate": 0.8149100257069408,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 567.1572082909988,
|
||||
"amplification": 1.8921879104611408,
|
||||
"ttft": {
|
||||
"count": 951,
|
||||
"mean": 156.08624654169503,
|
||||
"p50": 159.80228465000982,
|
||||
"p90": 263.44082942500245,
|
||||
"p99": 267.3189407904938
|
||||
},
|
||||
"tpot": {
|
||||
"count": 951,
|
||||
"mean": 0.5120599246480225,
|
||||
"p50": 0.5265210192539223,
|
||||
"p90": 0.5403215261268621,
|
||||
"p99": 0.5436686666190138
|
||||
},
|
||||
"e2e": {
|
||||
"count": 951,
|
||||
"mean": 188.3470215559188,
|
||||
"p50": 192.9057134770119,
|
||||
"p90": 294.68771972299146,
|
||||
"p99": 300.2205298934932
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 596,
|
||||
"completion_rate": 0.5107112253641817,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 789.8321366070013,
|
||||
"amplification": 2.63509093833936,
|
||||
"ttft": {
|
||||
"count": 596,
|
||||
"mean": 240.99040966441748,
|
||||
"p50": 259.1066791820049,
|
||||
"p90": 379.7789027979961,
|
||||
"p99": 381.22047349565617
|
||||
},
|
||||
"tpot": {
|
||||
"count": 596,
|
||||
"mean": 0.006412497327210905,
|
||||
"p50": 0.006444117785865692,
|
||||
"p90": 0.0064974606745333095,
|
||||
"p99": 0.006660929446938789
|
||||
},
|
||||
"e2e": {
|
||||
"count": 596,
|
||||
"mean": 241.39516468475196,
|
||||
"p50": 259.5132920899996,
|
||||
"p90": 380.1822811405,
|
||||
"p99": 381.620277955458
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 712,
|
||||
"completion_rate": 0.6101113967437874,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 880.7267912379903,
|
||||
"amplification": 2.938339780290167,
|
||||
"ttft": {
|
||||
"count": 712,
|
||||
"mean": 175.69983718378106,
|
||||
"p50": 172.43730882649834,
|
||||
"p90": 250.6456608123961,
|
||||
"p99": 594.014642834873
|
||||
},
|
||||
"tpot": {
|
||||
"count": 712,
|
||||
"mean": 0.007946936024875445,
|
||||
"p50": 0.007568808706382119,
|
||||
"p90": 0.010293392149296345,
|
||||
"p99": 0.011275891952114068
|
||||
},
|
||||
"e2e": {
|
||||
"count": 712,
|
||||
"mean": 176.20114515147316,
|
||||
"p50": 172.91078094949626,
|
||||
"p90": 251.09661155799986,
|
||||
"p99": 594.4257701966026
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 0.0,
|
||||
"pd_advantage": NaN,
|
||||
"n_slo": 0
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 0.0,
|
||||
"pd_advantage": NaN,
|
||||
"n_slo": 0
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 0.0,
|
||||
"pd_advantage": NaN,
|
||||
"n_slo": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"ttft_slo_s": 5.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 0.0,
|
||||
"pd_advantage": NaN,
|
||||
"n_slo": 0
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 0.0,
|
||||
"pd_advantage": NaN,
|
||||
"n_slo": 0
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 0.0,
|
||||
"pd_advantage": NaN,
|
||||
"n_slo": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.32634208898526,
|
||||
"amplification": 1.0019688816195438,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.2628433253752162,
|
||||
"p50": 0.2604120689211413,
|
||||
"p90": 0.2637787788407877,
|
||||
"p99": 0.2976114549371393
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005143964614959416,
|
||||
"p50": 0.005124552334318795,
|
||||
"p90": 0.005180289564166395,
|
||||
"p99": 0.0052467077378080115
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 0.5871811069348818,
|
||||
"p50": 0.5839827890740708,
|
||||
"p90": 0.5883447280153632,
|
||||
"p99": 0.6408826243295329
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.3759389320621,
|
||||
"amplification": 1.0021343499332585,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.30861241175426407,
|
||||
"p50": 0.3091614870354533,
|
||||
"p90": 0.3227507559815422,
|
||||
"p99": 0.3547600568598136
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005148936446188915,
|
||||
"p50": 0.0051316127147791645,
|
||||
"p90": 0.005184003428393413,
|
||||
"p99": 0.00547781713853871
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 0.6332931331915089,
|
||||
"p50": 0.6365591660141945,
|
||||
"p90": 0.6472898541716858,
|
||||
"p99": 0.7053035433730102
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.4041277950164,
|
||||
"amplification": 1.0022283955081221,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.31527804771431056,
|
||||
"p50": 0.31678270106203854,
|
||||
"p90": 0.3284846659982577,
|
||||
"p99": 0.3402980738645419
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.0055213440500518985,
|
||||
"p50": 0.005325577683776381,
|
||||
"p90": 0.006278165857056304,
|
||||
"p99": 0.0070092806818761975
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 0.6634175265073216,
|
||||
"p50": 0.6494167000055313,
|
||||
"p90": 0.7135140666039661,
|
||||
"p99": 0.7741130576422437
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"ttft_slo_s": 5.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.66455180699995,
|
||||
"amplification": 1.0030972395602922,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.584800394000121,
|
||||
"p50": 0.5817365200055065,
|
||||
"p90": 0.5870239008043427,
|
||||
"p99": 0.6088061455392736
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005585180932058873,
|
||||
"p50": 0.005308644777941813,
|
||||
"p90": 0.005355837698406836,
|
||||
"p99": 0.014590345752427337
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 0.9369382756572071,
|
||||
"p50": 0.9158835929993074,
|
||||
"p90": 0.9229304600012256,
|
||||
"p99": 1.5052119748995754
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.7607197400066,
|
||||
"amplification": 1.003418081467173,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.7293765576349805,
|
||||
"p50": 0.6887177099997643,
|
||||
"p90": 0.8911576298065484,
|
||||
"p99": 1.1879012519156076
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005308951736224922,
|
||||
"p50": 0.0053098901269203495,
|
||||
"p90": 0.005347346425488857,
|
||||
"p99": 0.005420954272433716
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 1.064155324243521,
|
||||
"p50": 1.0238807429996086,
|
||||
"p90": 1.22566919879755,
|
||||
"p99": 1.521622942700049
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 1167,
|
||||
"n_success": 1167,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.736197,
|
||||
"offered_qps": 3.8934236561358655,
|
||||
"wall_clock_s": 300.8507765819959,
|
||||
"amplification": 1.003718535142407,
|
||||
"ttft": {
|
||||
"count": 1167,
|
||||
"mean": 0.7028102769056583,
|
||||
"p50": 0.6950660040020011,
|
||||
"p90": 0.7394948218076024,
|
||||
"p99": 0.8658222487580485
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1167,
|
||||
"mean": 0.005782890183689697,
|
||||
"p50": 0.00558671395230617,
|
||||
"p90": 0.006629064533303286,
|
||||
"p99": 0.007366264003757681
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1167,
|
||||
"mean": 1.0674261210719522,
|
||||
"p50": 1.0612060459970962,
|
||||
"p90": 1.1425091800017981,
|
||||
"p99": 1.3045775910036173
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"ttft_slo_s": 5.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 1167
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,121 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 3552,
|
||||
"n_success": 1950,
|
||||
"completion_rate": 0.5489864864864865,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 348.4953830049999,
|
||||
"amplification": 1.1626534102882857,
|
||||
"ttft": {
|
||||
"count": 1950,
|
||||
"mean": 0.8245253885559363,
|
||||
"p50": 0.7988759850013594,
|
||||
"p90": 1.5384256363991882,
|
||||
"p99": 2.125257717882705
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1950,
|
||||
"mean": 0.07444567704430907,
|
||||
"p50": 0.08914986800879843,
|
||||
"p90": 0.09127071481447005,
|
||||
"p99": 0.09233926755616054
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1950,
|
||||
"mean": 77.04034660455785,
|
||||
"p50": 92.29079191399796,
|
||||
"p90": 94.47993849659906,
|
||||
"p99": 95.72745552870882
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 2754,
|
||||
"completion_rate": 0.7753378378378378,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 744.3694221920014,
|
||||
"amplification": 2.4833719166185753,
|
||||
"ttft": {
|
||||
"count": 2754,
|
||||
"mean": 4.5006646864741695,
|
||||
"p50": 2.329877773499902,
|
||||
"p90": 10.863291689799375,
|
||||
"p99": 21.572781211729307
|
||||
},
|
||||
"tpot": {
|
||||
"count": 2754,
|
||||
"mean": 0.046089308743682535,
|
||||
"p50": 0.04704797274047173,
|
||||
"p90": 0.04792202048768524,
|
||||
"p99": 0.059544689888886115
|
||||
},
|
||||
"e2e": {
|
||||
"count": 2754,
|
||||
"mean": 51.68415100813728,
|
||||
"p50": 50.91020956099965,
|
||||
"p90": 59.22212425199977,
|
||||
"p99": 68.42946432940865
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 1928,
|
||||
"completion_rate": 0.5427927927927928,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 821.7403777359868,
|
||||
"amplification": 2.7414975897474716,
|
||||
"ttft": {
|
||||
"count": 1928,
|
||||
"mean": 39.07585104131927,
|
||||
"p50": 42.6814165695032,
|
||||
"p90": 63.28579387369681,
|
||||
"p99": 73.84470144698193
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1928,
|
||||
"mean": 0.04064862157373329,
|
||||
"p50": 0.0398688508558163,
|
||||
"p90": 0.04277554483881431,
|
||||
"p99": 0.0695667276081441
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1928,
|
||||
"mean": 80.66499786808757,
|
||||
"p50": 84.05398454950046,
|
||||
"p90": 105.42814423799427,
|
||||
"p99": 113.52591980495636
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 0.10191441441441441,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 362
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 0.3502252252252252,
|
||||
"pd_advantage": 3.43646408839779,
|
||||
"n_slo": 1244
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 0.06447072072072071,
|
||||
"pd_advantage": 0.6325966850828729,
|
||||
"n_slo": 229
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,121 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3552,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 300.61600343801547,
|
||||
"amplification": 1.0029177964157683,
|
||||
"ttft": {
|
||||
"count": 3552,
|
||||
"mean": 0.139721406387908,
|
||||
"p50": 0.13759525847854093,
|
||||
"p90": 0.14188919204752892,
|
||||
"p99": 0.17052793529117438
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3552,
|
||||
"mean": 0.006496044329480822,
|
||||
"p50": 0.006430442302898453,
|
||||
"p90": 0.007889476366154117,
|
||||
"p99": 0.009282101713493095
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3552,
|
||||
"mean": 0.9651016481538636,
|
||||
"p50": 0.9548319909954444,
|
||||
"p90": 1.1440699087572284,
|
||||
"p99": 1.3324073514551853
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3552,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 300.7410873459885,
|
||||
"amplification": 1.0033351024670651,
|
||||
"ttft": {
|
||||
"count": 3552,
|
||||
"mean": 0.18467717330343286,
|
||||
"p50": 0.1806962049449794,
|
||||
"p90": 0.1929074571467936,
|
||||
"p99": 0.2592483600333798
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3552,
|
||||
"mean": 0.006980159784640828,
|
||||
"p50": 0.006945047342387123,
|
||||
"p90": 0.008164690015837552,
|
||||
"p99": 0.009163911323187739
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3552,
|
||||
"mean": 1.0715517728354285,
|
||||
"p50": 1.063076548918616,
|
||||
"p90": 1.2275727374362762,
|
||||
"p99": 1.4021264237130526
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3552,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 301.1243950970238,
|
||||
"amplification": 1.004613897210595,
|
||||
"ttft": {
|
||||
"count": 3552,
|
||||
"mean": 0.20078434001432569,
|
||||
"p50": 0.19958186394069344,
|
||||
"p90": 0.213167638995219,
|
||||
"p99": 0.23225504373782313
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3552,
|
||||
"mean": 0.010393430778126047,
|
||||
"p50": 0.010396533917478015,
|
||||
"p90": 0.012065167099743436,
|
||||
"p99": 0.01323438493101344
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3552,
|
||||
"mean": 1.5212831554822883,
|
||||
"p50": 1.5218885459471494,
|
||||
"p90": 1.7440477049094625,
|
||||
"p99": 1.9191367691196497
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 3552
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 3552
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 3552
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,121 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 3552,
|
||||
"n_success": 1050,
|
||||
"completion_rate": 0.2956081081081081,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 414.92026689198974,
|
||||
"amplification": 1.3842607013613617,
|
||||
"ttft": {
|
||||
"count": 1050,
|
||||
"mean": 0.8393797435154972,
|
||||
"p50": 0.8442945880015031,
|
||||
"p90": 1.4299255040969001,
|
||||
"p99": 2.1686189359705894
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1050,
|
||||
"mean": 0.08418485046407832,
|
||||
"p50": 0.08937374470713316,
|
||||
"p90": 0.09085719287396188,
|
||||
"p99": 0.09190563933350096
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1050,
|
||||
"mean": 173.22149556513375,
|
||||
"p50": 183.9983977669981,
|
||||
"p90": 187.03014375649946,
|
||||
"p99": 189.36978534828302
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 1320,
|
||||
"completion_rate": 0.3716216216216216,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 802.1317602040072,
|
||||
"amplification": 2.6760791447511205,
|
||||
"ttft": {
|
||||
"count": 1320,
|
||||
"mean": 19.92388417242484,
|
||||
"p50": 8.059029546493548,
|
||||
"p90": 64.71132232169766,
|
||||
"p99": 87.07121913193099
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1320,
|
||||
"mean": 0.05007250707090057,
|
||||
"p50": 0.0492711045202749,
|
||||
"p90": 0.07142825628519621,
|
||||
"p99": 0.08267229383702501
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1320,
|
||||
"mean": 122.44309290737698,
|
||||
"p50": 118.35192646600626,
|
||||
"p90": 167.89536100080443,
|
||||
"p99": 187.19021692074878
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 1041,
|
||||
"completion_rate": 0.29307432432432434,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 879.8981197069952,
|
||||
"amplification": 2.935523968100636,
|
||||
"ttft": {
|
||||
"count": 1041,
|
||||
"mean": 86.06142147925078,
|
||||
"p50": 92.45557322700915,
|
||||
"p90": 153.47880471601093,
|
||||
"p99": 159.16995789420034
|
||||
},
|
||||
"tpot": {
|
||||
"count": 1041,
|
||||
"mean": 0.04275836784091199,
|
||||
"p50": 0.038143618996089604,
|
||||
"p90": 0.06297858072350163,
|
||||
"p99": 0.07100282797010407
|
||||
},
|
||||
"e2e": {
|
||||
"count": 1041,
|
||||
"mean": 173.5925440399473,
|
||||
"p50": 199.81516771799943,
|
||||
"p90": 230.71809448600106,
|
||||
"p99": 234.9438766648003
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 0.009009009009009009,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 32
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 0.11486486486486487,
|
||||
"pd_advantage": 12.75,
|
||||
"n_slo": 408
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 0.04954954954954955,
|
||||
"pd_advantage": 5.5,
|
||||
"n_slo": 176
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,121 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3552,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 301.78380814404227,
|
||||
"amplification": 1.0068138369093473,
|
||||
"ttft": {
|
||||
"count": 3552,
|
||||
"mean": 0.14506312318632342,
|
||||
"p50": 0.14291562204016373,
|
||||
"p90": 0.1491016250802204,
|
||||
"p99": 0.1832571337954128
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3552,
|
||||
"mean": 0.0089611879488046,
|
||||
"p50": 0.008925844482420122,
|
||||
"p90": 0.010698060123752072,
|
||||
"p99": 0.011754058378410762
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3552,
|
||||
"mean": 2.430673889741114,
|
||||
"p50": 2.419925262918696,
|
||||
"p90": 2.874893436266575,
|
||||
"p99": 3.1480205075151746
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3551,
|
||||
"completion_rate": 0.9997184684684685,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 744.8868488909211,
|
||||
"amplification": 2.4850981601942745,
|
||||
"ttft": {
|
||||
"count": 3551,
|
||||
"mean": 0.20500909556101396,
|
||||
"p50": 0.20017725799698383,
|
||||
"p90": 0.21613375598099083,
|
||||
"p99": 0.29023518750909716
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3551,
|
||||
"mean": 0.010307210080394247,
|
||||
"p50": 0.010299421709450874,
|
||||
"p90": 0.011793581666607482,
|
||||
"p99": 0.012639003840969035
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3551,
|
||||
"mean": 2.8339171693501317,
|
||||
"p50": 2.8360355379991233,
|
||||
"p90": 3.2198793930001557,
|
||||
"p99": 3.438588996999897
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3552,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 303.4300506779691,
|
||||
"amplification": 1.0123060459587976,
|
||||
"ttft": {
|
||||
"count": 3552,
|
||||
"mean": 0.2333820717117756,
|
||||
"p50": 0.23283391550648957,
|
||||
"p90": 0.24394672318594532,
|
||||
"p99": 0.2734717815916518
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3552,
|
||||
"mean": 0.016453822599812207,
|
||||
"p50": 0.016576926972415737,
|
||||
"p90": 0.017214638463623238,
|
||||
"p99": 0.01769411424845092
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3552,
|
||||
"mean": 4.430368151474328,
|
||||
"p50": 4.463736370031256,
|
||||
"p90": 4.628522685484495,
|
||||
"p99": 4.75812664218829
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 3552
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 0.9997184684684685,
|
||||
"pd_advantage": 0.9997184684684685,
|
||||
"n_slo": 3551
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 3552
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,121 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 3552,
|
||||
"n_success": 545,
|
||||
"completion_rate": 0.15343468468468469,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 505.4139580530027,
|
||||
"amplification": 1.6861665622961597,
|
||||
"ttft": {
|
||||
"count": 545,
|
||||
"mean": 4.137136383750316,
|
||||
"p50": 0.5702517530007754,
|
||||
"p90": 1.6976309113961179,
|
||||
"p99": 56.1814190310361
|
||||
},
|
||||
"tpot": {
|
||||
"count": 545,
|
||||
"mean": 0.08390926873267023,
|
||||
"p50": 0.08933810225128375,
|
||||
"p90": 0.09449654152820693,
|
||||
"p99": 0.10591519025133134
|
||||
},
|
||||
"e2e": {
|
||||
"count": 545,
|
||||
"mean": 347.7765141811284,
|
||||
"p50": 366.3568219640001,
|
||||
"p90": 388.1241692415934,
|
||||
"p99": 435.2848098113155
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 771,
|
||||
"completion_rate": 0.2170608108108108,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 850.1328976760124,
|
||||
"amplification": 2.8362209684341706,
|
||||
"ttft": {
|
||||
"count": 771,
|
||||
"mean": 63.210636654403324,
|
||||
"p50": 1.1097561890055658,
|
||||
"p90": 179.64081536799495,
|
||||
"p99": 186.67013871119852
|
||||
},
|
||||
"tpot": {
|
||||
"count": 771,
|
||||
"mean": 0.05484690890217986,
|
||||
"p50": 0.04795774376874187,
|
||||
"p90": 0.08067800507863548,
|
||||
"p99": 0.09099416300415113
|
||||
},
|
||||
"e2e": {
|
||||
"count": 771,
|
||||
"mean": 287.81779562259663,
|
||||
"p50": 312.2160719559906,
|
||||
"p90": 372.0004520520015,
|
||||
"p99": 379.3279856524052
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 627,
|
||||
"completion_rate": 0.17652027027027026,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 867.8833550199925,
|
||||
"amplification": 2.8954402028102515,
|
||||
"ttft": {
|
||||
"count": 627,
|
||||
"mean": 179.58769048342904,
|
||||
"p50": 238.1998468660022,
|
||||
"p90": 378.29023678940143,
|
||||
"p99": 385.40577973942356
|
||||
},
|
||||
"tpot": {
|
||||
"count": 627,
|
||||
"mean": 0.04188420961205498,
|
||||
"p50": 0.03654626756630041,
|
||||
"p90": 0.06031132874202571,
|
||||
"p99": 0.06738955674930582
|
||||
},
|
||||
"e2e": {
|
||||
"count": 627,
|
||||
"mean": 351.1066520824709,
|
||||
"p50": 387.2650127700035,
|
||||
"p90": 507.9008203571953,
|
||||
"p99": 570.6463984230224
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 0.0,
|
||||
"pd_advantage": NaN,
|
||||
"n_slo": 0
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 0.05855855855855856,
|
||||
"pd_advantage": NaN,
|
||||
"n_slo": 208
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 0.036036036036036036,
|
||||
"pd_advantage": NaN,
|
||||
"n_slo": 128
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,121 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3552,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 304.97662343096454,
|
||||
"amplification": 1.0174657357946404,
|
||||
"ttft": {
|
||||
"count": 3552,
|
||||
"mean": 0.16242270423816507,
|
||||
"p50": 0.15836620499612764,
|
||||
"p90": 0.17363918052287775,
|
||||
"p99": 0.24847999344696287
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3552,
|
||||
"mean": 0.013691483182659388,
|
||||
"p50": 0.013865661168213087,
|
||||
"p90": 0.015242900696529327,
|
||||
"p99": 0.01626683903372128
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3552,
|
||||
"mean": 7.160272567887971,
|
||||
"p50": 7.247483663493767,
|
||||
"p90": 7.962273431208451,
|
||||
"p99": 8.480892732607899
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3551,
|
||||
"completion_rate": 0.9997184684684685,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 744.9637431279989,
|
||||
"amplification": 2.4853546954349977,
|
||||
"ttft": {
|
||||
"count": 3551,
|
||||
"mean": 0.2446379999283825,
|
||||
"p50": 0.24067969399038702,
|
||||
"p90": 0.2630795220611617,
|
||||
"p99": 0.3426029055262916
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3551,
|
||||
"mean": 0.016916919575073883,
|
||||
"p50": 0.016998299133030737,
|
||||
"p90": 0.01775875886689886,
|
||||
"p99": 0.018166751548973206
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3551,
|
||||
"mean": 8.89104466149889,
|
||||
"p50": 8.933106195996515,
|
||||
"p90": 9.333998591057025,
|
||||
"p99": 9.536390463996213
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3551,
|
||||
"completion_rate": 0.9997184684684685,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 312.45468625507783,
|
||||
"amplification": 1.0424141157985172,
|
||||
"ttft": {
|
||||
"count": 3551,
|
||||
"mean": 0.36231219612768273,
|
||||
"p50": 0.3462264990666881,
|
||||
"p90": 0.4059687410481274,
|
||||
"p99": 0.9837204645154998
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3551,
|
||||
"mean": 0.03268022218101953,
|
||||
"p50": 0.03333031399418835,
|
||||
"p90": 0.03557429400772957,
|
||||
"p99": 0.038558618127279086
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3551,
|
||||
"mean": 17.068018403084057,
|
||||
"p50": 17.392655145958997,
|
||||
"p90": 18.56302695896011,
|
||||
"p99": 20.159411454980727
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 3552
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 0.9997184684684685,
|
||||
"pd_advantage": 0.9997184684684685,
|
||||
"n_slo": 3551
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 0.9997184684684685,
|
||||
"pd_advantage": 0.9997184684684685,
|
||||
"n_slo": 3551
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,121 +0,0 @@
|
||||
{
|
||||
"baseline": "8C-proxy",
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"name": "8C-proxy",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3552,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 300.21845804993063,
|
||||
"amplification": 1.001591501940313,
|
||||
"ttft": {
|
||||
"count": 3552,
|
||||
"mean": 0.1369056966029782,
|
||||
"p50": 0.13516780693316832,
|
||||
"p90": 0.13835511771030723,
|
||||
"p99": 0.15317223175661632
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3552,
|
||||
"mean": 0.005430781936309906,
|
||||
"p50": 0.005039893150780468,
|
||||
"p90": 0.007063257358303028,
|
||||
"p99": 0.0077793481296283135
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3552,
|
||||
"mean": 0.4793568905444592,
|
||||
"p50": 0.45412574551301077,
|
||||
"p90": 0.5810393166844734,
|
||||
"p99": 0.6301419002050533
|
||||
}
|
||||
},
|
||||
"4P+4D": {
|
||||
"name": "4P+4D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3551,
|
||||
"completion_rate": 0.9997184684684685,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 735.1059510430787,
|
||||
"amplification": 2.452467041410379,
|
||||
"ttft": {
|
||||
"count": 3551,
|
||||
"mean": 0.17374673821534295,
|
||||
"p50": 0.17102365801110864,
|
||||
"p90": 0.18273873499128968,
|
||||
"p99": 0.24688138795318082
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3551,
|
||||
"mean": 0.005485491774206834,
|
||||
"p50": 0.0053864502226046865,
|
||||
"p90": 0.006091995366168992,
|
||||
"p99": 0.007109403222178419
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3551,
|
||||
"mean": 0.5196563635758822,
|
||||
"p50": 0.5100997349945828,
|
||||
"p90": 0.5655082209268585,
|
||||
"p99": 0.6639982180204242
|
||||
}
|
||||
},
|
||||
"6P+2D": {
|
||||
"name": "6P+2D",
|
||||
"n_offered": 3552,
|
||||
"n_success": 3552,
|
||||
"completion_rate": 1.0,
|
||||
"offered_window_s": 299.74141899999995,
|
||||
"offered_qps": 11.850214134070008,
|
||||
"wall_clock_s": 300.38101644301787,
|
||||
"amplification": 1.0021338307036503,
|
||||
"ttft": {
|
||||
"count": 3552,
|
||||
"mean": 0.18293367427152893,
|
||||
"p50": 0.1822461549891159,
|
||||
"p90": 0.1938482352765277,
|
||||
"p99": 0.21272844232735222
|
||||
},
|
||||
"tpot": {
|
||||
"count": 3552,
|
||||
"mean": 0.007192309629699456,
|
||||
"p50": 0.007143509595484902,
|
||||
"p90": 0.008732455453672816,
|
||||
"p99": 0.009842920153335268
|
||||
},
|
||||
"e2e": {
|
||||
"count": 3552,
|
||||
"mean": 0.636424056327808,
|
||||
"p50": 0.6324848984950222,
|
||||
"p90": 0.7393875011475757,
|
||||
"p99": 0.8261980937235056
|
||||
}
|
||||
}
|
||||
},
|
||||
"slo_grid": [
|
||||
{
|
||||
"ttft_slo_s": 2.0,
|
||||
"tpot_slo_s": 0.05,
|
||||
"arms": {
|
||||
"8C-proxy": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 3552
|
||||
},
|
||||
"4P+4D": {
|
||||
"attainment": 0.9997184684684685,
|
||||
"pd_advantage": 0.9997184684684685,
|
||||
"n_slo": 3551
|
||||
},
|
||||
"6P+2D": {
|
||||
"attainment": 1.0,
|
||||
"pd_advantage": 1.0,
|
||||
"n_slo": 3552
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,131 +0,0 @@
|
||||
# LPWL vs 4 baselines — parameter-free routing for agentic workloads
|
||||
|
||||
Date: 2026-05-29. Hardware: dash1, 8×H20, Qwen3-Coder-30B-A3B, TP=1,
|
||||
max_model_len=200000, fresh vLLM per arm (cold APC), `--policy` via
|
||||
`scripts/b3_isolated_policy.sh`. Analyzer: `scripts/bench_report.py`.
|
||||
|
||||
## Motivation
|
||||
|
||||
unified+A+B carries too many knobs (`overload_factor`, `lmetric_decode_weight`,
|
||||
the 0.5 `cache_ratio` gate). Goal: a policy derived from the agentic *pattern*
|
||||
with no tuned constants, that does not overfit.
|
||||
|
||||
## LPWL (Least-Prefill-Work-Left)
|
||||
|
||||
`scripts/cache_aware_proxy.py:pick_instance_leastwork`, `--policy leastwork`:
|
||||
|
||||
```
|
||||
score_i = pending_prefill_tokens_i + max(0, input_len − cache_hit_i) → argmin
|
||||
```
|
||||
|
||||
Tie-break: fewest `num_requests`, then round-robin. **Zero hyperparameters.**
|
||||
|
||||
Why this shape (straight from the workload):
|
||||
- Decode is cheap (I/O ~217×) ⇒ the only load worth modeling is outstanding
|
||||
*prefill* token-work. No decode weight; dropping LMetric's `×num_requests`
|
||||
also makes an idle-but-decoding host score `input` (its true marginal cost),
|
||||
not 0 — fixing the empty-batch degeneracy for free.
|
||||
- Cache-awareness *is* the affinity mechanism: a returning session's owner has
|
||||
`new_uncached ≈ 0`, so it sticks unless its prefill backlog exceeds the cache
|
||||
saving (`input`). The stick-vs-spill crossover is computed from real
|
||||
token-work — no `overload_factor`, no `cache_ratio` gate.
|
||||
- Session skew degrades gracefully: a heavy session inflates its owner's
|
||||
`pending_prefill`, auto-diverting *other* sessions while the heavy one stays
|
||||
put (no cold re-prefill).
|
||||
|
||||
## Results — 600s trace (`w600_r0.0015_st30_first600s.jsonl`, 807 reqs)
|
||||
|
||||
This is the colder regime (theoretical APC ceiling ≈ 70% vs 80% for full w600).
|
||||
|
||||
| policy | knobs | TTFT mean | TTFT p90 | E2E mean | E2E p90 | E2E p99 | TPOT p90 | APC | req-bal |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| **LPWL** | **0** | **3398** | **7983** | **8116** | **19014** | 87024 | 26 | 0.648 | **1.55×** |
|
||||
| unified+A+B | 3 | 3876 | 11562 | 8199 | 22569 | **74266** | **25** | 0.661 | 1.56× |
|
||||
| unified default | 2 | 5066 | 16389 | 10481 | 28427 | 96361 | 34 | 0.689 | 2.28× |
|
||||
| LMetric | 0 | 4809 | 14037 | 10051 | 26726 | 97442 | 32 | 0.507 | 2.11× |
|
||||
| sticky | 0 | 5758 | 20356 | 10815 | 34734 | 82732 | 28 | **0.696** | 3.86× |
|
||||
|
||||
(latencies ms; req-bal = max:min per-worker request count; this batch predates
|
||||
the GPU-capture harness change so per-worker GPU util reads N/A.)
|
||||
|
||||
### Findings
|
||||
|
||||
1. **LPWL is overall best with zero knobs:** TTFT mean −12% / p90 −31%, E2E
|
||||
mean ~tie / p90 −16% vs the tuned unified+A+B; best request balance; TPOT
|
||||
tied-best. Only loss is E2E p99 (+17%) from heavy-class decode concentration.
|
||||
2. **The baselines bracket the problem and explain why LPWL works:** sticky has
|
||||
the highest APC (0.696) but worst latency (hot-pin, 3.86× imbalance); LMetric
|
||||
has the worst APC (0.507) because `×num_requests` swallows the cache signal.
|
||||
LPWL drops exactly that factor, so locality re-emerges (APC 0.648, beside the
|
||||
explicit-affinity policies) while balance stays tight — the sweet spot, no
|
||||
gate, no tuning.
|
||||
3. **Anti-overfit, demonstrated:** unified+A+B was tuned (of=1.3, lmw=0.01) on
|
||||
the *full* w600; on the colder 600s regime the parameter-free policy beats it
|
||||
by 31% TTFT p90. The tuning did not transfer; LPWL did.
|
||||
|
||||
### Per-class TTFT (ms, mean / p50 / p90 / p99) — LPWL dominates except the floor
|
||||
|
||||
| class | LPWL p90 | A+B p90 | LPWL p99 | A+B p99 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| WARM<5k | 319 | 324 | 1032 | 2092 |
|
||||
| MED5-20k | 1618 | 1952 | 3013 | 33189 |
|
||||
| HEAVY20-50k | 4851 | 6198 | 14599 | 29044 |
|
||||
| HEAVY+>50k | 28942 | 33777 | 52651 | 50778 |
|
||||
|
||||
LPWL's only weak class is the workload-inherent HEAVY+>50k floor (≈tied across
|
||||
all policies). Elsewhere it avoids the mid-class tails the unified gate creates
|
||||
when it pins a mid request behind a 50k-token turn on a barely-warm owner.
|
||||
|
||||
## Full-w600 cross-check (1214 reqs, `outputs/lpwl_vs_ab_live/`)
|
||||
|
||||
On the warmer full trace, LPWL vs unified+A+B is a wash: LPWL wins TTFT p90
|
||||
(−14%) but loses TPOT (+38%) and per-worker balance. Combined claim across both
|
||||
regimes: **LPWL ∈ [tied, clearly-better] vs a tuned baseline, at zero knobs.**
|
||||
|
||||
## Ablation: derived-κ decode term (`leastwork_kappa`) — NET-NEGATIVE
|
||||
|
||||
Tested the proposed knob-free fix for LPWL's E2E-p99: `--policy leastwork_kappa`,
|
||||
`score = (pending_prefill + new_uncached) × (1 + κ·ongoing_decode_tokens)`, with
|
||||
κ = 2.5e-6 *derived* from hardware (KV ~100 KB/tok ÷ HBM 4 TB/s ÷ TPOT 10 ms on
|
||||
H20+Qwen3-30B-A3B), not trace-tuned. Same 600s trace, fresh vLLM, cold APC.
|
||||
|
||||
| metric | leastwork | leastwork_kappa | Δ |
|
||||
|---|---:|---:|---:|
|
||||
| TTFT p90 | 7983 | 9390 | +18% (worse) |
|
||||
| TTFT p99 | 44891 | 42370 | −6% |
|
||||
| E2E p90 | 19014 | 21674 | +14% (worse) |
|
||||
| E2E p99 | 87024 | 90155 | +4% (did NOT fix) |
|
||||
| APC | 0.648 | 0.647 | tie |
|
||||
| req-balance | 1.55× | 1.97× | worse |
|
||||
|
||||
**Verdict: decode-awareness is the wrong lever for agentic.** The κ term is
|
||||
correct physics aimed at a negligible effect (decode is cheap, output p50≈80),
|
||||
so it mostly bounces heavy requests off their cache-owner → cold re-prefill
|
||||
elsewhere → new hotspots (balance degrades 1.55×→1.97×). It does NOT fix E2E-p99
|
||||
because that tail is the **structural HEAVY+>50k floor** (per-class p99 ≈51–52k
|
||||
for *all* policies), not decode interference — i.e. not routing-fixable. This is
|
||||
a negative result that *justifies* LPWL's omission of any decode term. The policy
|
||||
is kept in-tree as a documented ablation; do not revive without a decode-heavy
|
||||
regime. (First run on the GPU-capturing harness: per-worker GPU util mean 42–83%,
|
||||
1.95× spread — it even shows the κ-induced imbalance.)
|
||||
|
||||
## Caveats / open work
|
||||
|
||||
- n=1 per arm. The 600s −31% TTFT p90 is corroborated by mean/p50/per-class, but
|
||||
repeat to bound run-to-run noise (no 3× repeats yet, by request — quick single
|
||||
set first).
|
||||
- E2E-p99 deep tail is the one consistent LPWL weak spot (heavy-session decode
|
||||
concentration). Proposed knob-free fix: add `+ κ·ongoing_decode` with
|
||||
`κ = measured(TPOT/token) / prefill_throughput` (a derived hardware ratio, not
|
||||
a tuned scalar). Not yet implemented.
|
||||
|
||||
## Repro
|
||||
|
||||
```bash
|
||||
# 5-policy, 600s trace (≈18 min/arm, ~90 min total)
|
||||
OUTROOT=.../outputs/policy5_600s \
|
||||
bash microbench/connector_tax/cache_sweep/run_5policy_600s.sh
|
||||
# unified report
|
||||
.venv/bin/python scripts/bench_report.py --root .../outputs/policy5_600s \
|
||||
leastwork unified_ab unified_def lmetric sticky
|
||||
```
|
||||
@@ -1,193 +0,0 @@
|
||||
# MB1 — Prefill–Decode Interference (chunked-prefill on, vLLM 0.18.1 default)
|
||||
|
||||
Persistent record of the phase-interference microbench used to put a
|
||||
quantitative upper bound on **what PD-disaggregation can buy** under the
|
||||
chunked-prefill-on baseline. Re-runs append a dated section at the
|
||||
bottom; the **Summary** block is what gets cited.
|
||||
|
||||
---
|
||||
|
||||
## Summary (latest)
|
||||
|
||||
| Headline | Value |
|
||||
|---|---|
|
||||
| Baseline single-stream TPOT (D=1, idle GPU) | **4.8 ms** |
|
||||
| Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** |
|
||||
| Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** |
|
||||
| Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** |
|
||||
|
||||
**What MB1 actually measures**:
|
||||
|
||||
> During a prefill burst, every ongoing decode stream is essentially
|
||||
> halted (per-stream effective TPOT is 15×–2000× baseline, scaling with
|
||||
> prefill size). The **total decode time lost per prefill event is
|
||||
> `D × T_prefill`** (D concurrent decodes each lose ~T_prefill of useful
|
||||
> work). For the trace mean (P ≈ 33k tokens, T_prefill ≈ 4.5 s) at D=8
|
||||
> that's **~36 seconds of decode-equivalent work lost per request**.
|
||||
> This is the **upper bound on what PD-disaggregation's phase isolation
|
||||
> could recover** on the decode side.
|
||||
|
||||
**⚠ Correction (2026-05-27)**: an earlier version of this README framed
|
||||
the §3.2 PD-disagg argument as "phase-isolation benefit is capped at
|
||||
the decode duration of the new request (~50–200 ms), so MB2 transfer
|
||||
cost dominates". That framing was wrong. The correct accounting is
|
||||
benefit-per-prefill-event = D × T_prefill (aggregate decode time saved
|
||||
across all stalled streams), which is **much larger than per-request
|
||||
transfer cost**. The actual reason static PD-disagg fails in agentic
|
||||
is **D-side KV pool capacity** (`figs/f4b_pdsep_kv_wall.png`), not a
|
||||
cost-vs-benefit imbalance on phase isolation. See `RESULTS_SUMMARY.md`
|
||||
section 4 for the corrected framing.
|
||||
|
||||
## Setup
|
||||
|
||||
| Component | Value |
|
||||
|---|---|
|
||||
| Host | dash1, H20 96 GiB, driver 570.133.20 |
|
||||
| Venv | `/home/admin/cpfs/wjh/agentic-kv-fresh/.venv` |
|
||||
| vLLM | 0.18.1 official wheel (chunked-prefill default-on, V1 engine) |
|
||||
| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct` |
|
||||
| Launch flags | `--tensor-parallel-size 1 --enable-prefix-caching --gpu-memory-utilization 0.9 --max-model-len 200000 --max-num-batched-tokens 8192` |
|
||||
| kv_connector | **none** (this measures pure single-GPU phase interference; PD-disagg cost lives in MB2) |
|
||||
|
||||
## Method
|
||||
|
||||
Adapted from `microbench/interference/driver.py`:
|
||||
|
||||
1. Start D streaming decode requests on `/v1/chat/completions` with a
|
||||
long max_tokens cap. Discard the first 32 tokens as warmup.
|
||||
2. After 1 s, inject one prefill-only request with `max_tokens=1` and
|
||||
an input of `P` synthetic tokens (uuid-seeded for zero prefix-cache
|
||||
reuse). Measure the prefill's TTFT.
|
||||
3. Bin the *during-prefill* tokens from each decode stream by whether
|
||||
their wall-clock falls inside `[prefill_inject_ts, prefill_inject_ts +
|
||||
prefill_ttft]`. Report inter-token p50 / p90.
|
||||
4. Bin a baseline run (D streams, no prefill injection) the same way.
|
||||
|
||||
We additionally compute the **effective per-stream TPOT during the
|
||||
prefill burst** as the single most informative summary:
|
||||
|
||||
```
|
||||
eff_TPOT_during = prefill_ttft_ms / (num_tokens_during_prefill / D)
|
||||
```
|
||||
|
||||
This is the average rate at which each decode stream produces tokens
|
||||
while a prefill is in flight. Compared to baseline TPOT it gives the
|
||||
real per-stream throughput penalty (chunked-prefill p50 looks deceptively
|
||||
fine because most decode-token intervals during the burst are at normal
|
||||
speed; p90 sees the stall but is itself noisy; the effective TPOT is
|
||||
the cleanest "average over the whole burst window" number).
|
||||
|
||||
## Results — 2026-05-27, dash1 GPU 0, chunk_tokens=8192
|
||||
|
||||
3 D × 5 P × 3 reps. Aggregated by `analyze_mb1.py`.
|
||||
|
||||
| D | P (tok) | base TPOT (ms) | prefill_ttft (ms) | per-stream tokens during | effective TPOT during (ms) | penalty | max PD-disagg benefit per stream (ms) |
|
||||
|--:|--:|--:|--:|--:|--:|--:|--:|
|
||||
| 1 | 2 048 | 4.79 | 163 | 4.0 | 41 | 8× | 144 |
|
||||
| 1 | 8 192 | 4.78 | 584 | 5.0 | 117 | 24× | 560 |
|
||||
| 1 | 32 768 | 4.78 | 4 515 | 5.0 | 903 | 189× | 4 491 |
|
||||
| 1 | 65 536 | 4.78 | 15 568 | 5.3 | 2 919 | 610× | 15 542 |
|
||||
| 1 | 131 072 | 4.78 | 56 765 | 5.7 | 10 017 | 2 094× | 56 738 |
|
||||
| 4 | 2 048 | 5.62 | 138 | 3.9 | 36 | 6× | 117 |
|
||||
| 4 | 8 192 | 6.08 | 574 | 4.5 | 128 | 21× | 547 |
|
||||
| 4 | 32 768 | 6.09 | 4 529 | 11.9 | 381 | 63× | 4 457 |
|
||||
| 4 | 65 536 | 5.85 | 15 587 | 19.8 | 789 | 135× | 15 471 |
|
||||
| 4 | 131 072 | 6.27 | 56 697 | 37.4 | 1 517 | 242× | 56 463 |
|
||||
| 8 | 2 048 | 7.71 | 143 | 4.5 | 32 | 4× | 109 |
|
||||
| 8 | 8 192 | 7.69 | 583 | 5.1 | 114 | 15× | 544 |
|
||||
| 8 | 32 768 | 7.42 | 4 520 | 11.7 | 387 | 52× | 4 434 |
|
||||
| 8 | 65 536 | 7.67 | 15 615 | 20.6 | 757 | 99× | 15 457 |
|
||||
| 8 | 131 072 | 7.74 | 56 991 | 40.2 | 1 419 | 183× | 56 680 |
|
||||
|
||||
**Reading the table**:
|
||||
|
||||
- *Baseline TPOT* grows mildly with D (4.8 ms → 7.7 ms as D goes 1 → 8).
|
||||
Multi-stream decoding has small but nonzero contention even without
|
||||
prefill.
|
||||
- *Effective TPOT during* grows mostly with P: a single 8k prefill stalls
|
||||
decode for ~580 ms regardless of D, so each stream emits only a handful
|
||||
of tokens during that 580 ms window — effective per-stream TPOT
|
||||
collapses to 100–130 ms. Larger prefill = more chunks = larger stall.
|
||||
- *Penalty* is the eff/baseline ratio. Above 50× for P ≥ 32k. Above
|
||||
500× for D=1 at P ≥ 65k.
|
||||
- *Max PD-disagg benefit per stream* = `prefill_ttft − per_stream_tokens
|
||||
× baseline_TPOT` ≈ `prefill_ttft` (since interference essentially
|
||||
halts decode). This is the entire prefill duration's worth of decode
|
||||
time that could in principle be recovered.
|
||||
|
||||
**Connecting to the §3.2 PD-disagg argument** (corrected):
|
||||
|
||||
PD-disagg's promised phase-isolation benefit is **per prefill event**,
|
||||
not per request. When a new prefill arrives, it stalls every concurrent
|
||||
decode stream on the same GPU. The aggregate decode time lost across
|
||||
those D streams is `D × T_prefill`. PD-disagg moving prefill off-decode-GPU
|
||||
recovers all of it.
|
||||
|
||||
Plugging numbers per prefill event:
|
||||
|
||||
| Prefill size | T_prefill | PD-disagg cost (MB2 T_transfer) | PD-disagg benefit (D=8 × T_prefill) | Ratio |
|
||||
|---:|---:|---:|---:|---:|
|
||||
| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s | 0.7 % |
|
||||
| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s | 0.9 % |
|
||||
| 125k tok (~p99) | 57 s | 1.9 s | 456 s | 0.4 % |
|
||||
|
||||
On the **phase-isolation axis alone**, PD-disagg wins by 100×–250×.
|
||||
The reason static PD-disagg nonetheless **fails in agentic** is a
|
||||
*different* failure mode: the D-side KV pool cannot fit p90+ requests
|
||||
(p99 = 11.5 GiB; D-instance pool ≈ 38 GiB; 4P+4D halves system-wide
|
||||
decode capacity → TTFT p50 62×, success rate 99.5% → 52% in colleague's
|
||||
4P+4D experiment). The structural problem is **capacity** (see
|
||||
`figs/f4b_pdsep_kv_wall.png`), not transfer-cost vs phase-isolation
|
||||
trade-off.
|
||||
|
||||
## Reproduction
|
||||
|
||||
```bash
|
||||
# vllm pair-free single-instance launch
|
||||
ssh dash1 'GPU=0 PORT=8000 CHUNK_TOKENS=8192 \
|
||||
bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh start'
|
||||
|
||||
# sweep
|
||||
ssh dash1 'source /home/admin/cpfs/wjh/agentic-kv-fresh/.venv/bin/activate && \
|
||||
python /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_driver.py \
|
||||
--host 127.0.0.1 --port 8000 \
|
||||
--model /home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
|
||||
--decode-batch-sizes 1,4,8 --prefill-tokens 2048,8192,32768,65536,131072 \
|
||||
--reps 3 --output-dir /home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results'
|
||||
|
||||
# pull + analyze
|
||||
scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/summary.csv \
|
||||
analysis/mb1/summary.csv
|
||||
.venv/bin/python microbench/fresh_setup/analyze_mb1.py \
|
||||
--summary analysis/mb1/summary.csv --out analysis/mb1/breakdown.json
|
||||
.venv/bin/python microbench/fresh_setup/plot_mb1.py \
|
||||
--mb1 analysis/mb1/breakdown.json \
|
||||
--mb2-intra analysis/mb2/intra_kvboth_breakdown.json \
|
||||
--mb2-inter analysis/mb2/inter_kvboth_breakdown.json
|
||||
|
||||
# teardown
|
||||
ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop'
|
||||
```
|
||||
|
||||
## Open questions / next runs
|
||||
|
||||
- **Chunk size sensitivity**: this run uses `--max-num-batched-tokens
|
||||
8192`. Sarathi-Serve goes smaller (e.g. 1024) and recovers more
|
||||
decode interleaving inside each prefill burst. Worth running
|
||||
chunk_tokens ∈ {1024, 2048, 4096, 16384} to map the chunk-size axis.
|
||||
- **Higher D**: 12, 16 streams to see whether the penalty saturates or
|
||||
keeps shrinking per-stream.
|
||||
- **Cross-validate effective_TPOT_during with token-time-series plot**:
|
||||
raw per-token timestamps could reveal whether the stall is a few big
|
||||
spikes or many small ones (currently inferred from p50/p90 spread).
|
||||
|
||||
## Run log
|
||||
|
||||
### 2026-05-27 — dash1 GPU 0, chunk_tokens=8192
|
||||
|
||||
3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on
|
||||
dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`.
|
||||
Figure: `figs/mb1_interference.png`. The figure
|
||||
`figs/pd_cost_vs_benefit.png` from the original commit `029821c` was
|
||||
based on the wrong "benefit ≤ decode duration" accounting; **deleted in
|
||||
the correction commit**.
|
||||
@@ -1,199 +0,0 @@
|
||||
{
|
||||
"summary": [
|
||||
{
|
||||
"decode_batch_size": 1,
|
||||
"new_prefill_tokens": 2048,
|
||||
"baseline_tpot_ms": 4.79,
|
||||
"during_tpot_p50_ms_raw": 35.43,
|
||||
"during_tpot_p90_ms_raw": 79.91,
|
||||
"prefill_ttft_ms": 163.3,
|
||||
"num_tokens_during_prefill_total": 4.0,
|
||||
"per_stream_tokens_during": 4.0,
|
||||
"effective_tpot_during_ms": 40.8,
|
||||
"interference_penalty_x": 8.5,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 144.2
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 1,
|
||||
"new_prefill_tokens": 8192,
|
||||
"baseline_tpot_ms": 4.78,
|
||||
"during_tpot_p50_ms_raw": 6.56,
|
||||
"during_tpot_p90_ms_raw": 328.57,
|
||||
"prefill_ttft_ms": 583.9,
|
||||
"num_tokens_during_prefill_total": 5.0,
|
||||
"per_stream_tokens_during": 5.0,
|
||||
"effective_tpot_during_ms": 116.8,
|
||||
"interference_penalty_x": 24.4,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 560.0
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 1,
|
||||
"new_prefill_tokens": 32768,
|
||||
"baseline_tpot_ms": 4.78,
|
||||
"during_tpot_p50_ms_raw": 4.75,
|
||||
"during_tpot_p90_ms_raw": 4.9,
|
||||
"prefill_ttft_ms": 4515.3,
|
||||
"num_tokens_during_prefill_total": 5.0,
|
||||
"per_stream_tokens_during": 5.0,
|
||||
"effective_tpot_during_ms": 903.1,
|
||||
"interference_penalty_x": 188.8,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 4491.4
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 1,
|
||||
"new_prefill_tokens": 65536,
|
||||
"baseline_tpot_ms": 4.78,
|
||||
"during_tpot_p50_ms_raw": 4.69,
|
||||
"during_tpot_p90_ms_raw": 4.97,
|
||||
"prefill_ttft_ms": 15567.6,
|
||||
"num_tokens_during_prefill_total": 5.3,
|
||||
"per_stream_tokens_during": 5.33,
|
||||
"effective_tpot_during_ms": 2918.9,
|
||||
"interference_penalty_x": 610.2,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 15542.0
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 1,
|
||||
"new_prefill_tokens": 131072,
|
||||
"baseline_tpot_ms": 4.78,
|
||||
"during_tpot_p50_ms_raw": 4.71,
|
||||
"during_tpot_p90_ms_raw": 4.9,
|
||||
"prefill_ttft_ms": 56765.2,
|
||||
"num_tokens_during_prefill_total": 5.7,
|
||||
"per_stream_tokens_during": 5.67,
|
||||
"effective_tpot_during_ms": 10017.4,
|
||||
"interference_penalty_x": 2094.5,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 56738.1
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 4,
|
||||
"new_prefill_tokens": 2048,
|
||||
"baseline_tpot_ms": 5.62,
|
||||
"during_tpot_p50_ms_raw": 22.18,
|
||||
"during_tpot_p90_ms_raw": 84.85,
|
||||
"prefill_ttft_ms": 138.3,
|
||||
"num_tokens_during_prefill_total": 15.5,
|
||||
"per_stream_tokens_during": 3.88,
|
||||
"effective_tpot_during_ms": 35.7,
|
||||
"interference_penalty_x": 6.3,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 116.6
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 4,
|
||||
"new_prefill_tokens": 8192,
|
||||
"baseline_tpot_ms": 6.08,
|
||||
"during_tpot_p50_ms_raw": 8.45,
|
||||
"during_tpot_p90_ms_raw": 515.39,
|
||||
"prefill_ttft_ms": 574.1,
|
||||
"num_tokens_during_prefill_total": 18.0,
|
||||
"per_stream_tokens_during": 4.5,
|
||||
"effective_tpot_during_ms": 127.6,
|
||||
"interference_penalty_x": 21.0,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 546.8
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 4,
|
||||
"new_prefill_tokens": 32768,
|
||||
"baseline_tpot_ms": 6.09,
|
||||
"during_tpot_p50_ms_raw": 9.83,
|
||||
"during_tpot_p90_ms_raw": 1314.87,
|
||||
"prefill_ttft_ms": 4529.1,
|
||||
"num_tokens_during_prefill_total": 47.5,
|
||||
"per_stream_tokens_during": 11.88,
|
||||
"effective_tpot_during_ms": 381.4,
|
||||
"interference_penalty_x": 62.7,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 4456.9
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 4,
|
||||
"new_prefill_tokens": 65536,
|
||||
"baseline_tpot_ms": 5.85,
|
||||
"during_tpot_p50_ms_raw": 6.41,
|
||||
"during_tpot_p90_ms_raw": 2077.47,
|
||||
"prefill_ttft_ms": 15586.5,
|
||||
"num_tokens_during_prefill_total": 79.0,
|
||||
"per_stream_tokens_during": 19.75,
|
||||
"effective_tpot_during_ms": 789.2,
|
||||
"interference_penalty_x": 135.0,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 15471.0
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 4,
|
||||
"new_prefill_tokens": 131072,
|
||||
"baseline_tpot_ms": 6.27,
|
||||
"during_tpot_p50_ms_raw": 6.3,
|
||||
"during_tpot_p90_ms_raw": 4405.18,
|
||||
"prefill_ttft_ms": 56697.1,
|
||||
"num_tokens_during_prefill_total": 149.5,
|
||||
"per_stream_tokens_during": 37.38,
|
||||
"effective_tpot_during_ms": 1517.0,
|
||||
"interference_penalty_x": 241.8,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 56462.6
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 8,
|
||||
"new_prefill_tokens": 2048,
|
||||
"baseline_tpot_ms": 7.71,
|
||||
"during_tpot_p50_ms_raw": 8.38,
|
||||
"during_tpot_p90_ms_raw": 98.98,
|
||||
"prefill_ttft_ms": 143.1,
|
||||
"num_tokens_during_prefill_total": 35.7,
|
||||
"per_stream_tokens_during": 4.46,
|
||||
"effective_tpot_during_ms": 32.1,
|
||||
"interference_penalty_x": 4.2,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 108.8
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 8,
|
||||
"new_prefill_tokens": 8192,
|
||||
"baseline_tpot_ms": 7.69,
|
||||
"during_tpot_p50_ms_raw": 9.34,
|
||||
"during_tpot_p90_ms_raw": 519.29,
|
||||
"prefill_ttft_ms": 583.3,
|
||||
"num_tokens_during_prefill_total": 41.0,
|
||||
"per_stream_tokens_during": 5.12,
|
||||
"effective_tpot_during_ms": 113.8,
|
||||
"interference_penalty_x": 14.8,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 543.9
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 8,
|
||||
"new_prefill_tokens": 32768,
|
||||
"baseline_tpot_ms": 7.42,
|
||||
"during_tpot_p50_ms_raw": 11.61,
|
||||
"during_tpot_p90_ms_raw": 1315.48,
|
||||
"prefill_ttft_ms": 4520.3,
|
||||
"num_tokens_during_prefill_total": 93.3,
|
||||
"per_stream_tokens_during": 11.67,
|
||||
"effective_tpot_during_ms": 387.5,
|
||||
"interference_penalty_x": 52.2,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 4433.7
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 8,
|
||||
"new_prefill_tokens": 65536,
|
||||
"baseline_tpot_ms": 7.67,
|
||||
"during_tpot_p50_ms_raw": 19.09,
|
||||
"during_tpot_p90_ms_raw": 2471.4,
|
||||
"prefill_ttft_ms": 15615.5,
|
||||
"num_tokens_during_prefill_total": 165.0,
|
||||
"per_stream_tokens_during": 20.62,
|
||||
"effective_tpot_during_ms": 757.1,
|
||||
"interference_penalty_x": 98.8,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 15457.4
|
||||
},
|
||||
{
|
||||
"decode_batch_size": 8,
|
||||
"new_prefill_tokens": 131072,
|
||||
"baseline_tpot_ms": 7.74,
|
||||
"during_tpot_p50_ms_raw": 11.51,
|
||||
"during_tpot_p90_ms_raw": 4895.27,
|
||||
"prefill_ttft_ms": 56991.4,
|
||||
"num_tokens_during_prefill_total": 321.3,
|
||||
"per_stream_tokens_during": 40.17,
|
||||
"effective_tpot_during_ms": 1418.9,
|
||||
"interference_penalty_x": 183.3,
|
||||
"max_pd_disagg_benefit_ms_per_stream": 56680.4
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,46 +0,0 @@
|
||||
chunk_size,decode_batch_size,new_prefill_tokens,repetition,tpot_baseline_p50_ms,tpot_baseline_p90_ms,tpot_during_prefill_p50_ms,tpot_during_prefill_p90_ms,tpot_after_prefill_p50_ms,prefill_ttft_ms,num_tokens_during_prefill,tpot_penalty_p50_ms,tpot_penalty_ratio
|
||||
8192,1,131072,0,4.777565016411245,4.900234832894057,4.701301048044115,4.948397364933044,0.0,56719.25117995124,7,-0.07626396836712956,0.9840370632099913
|
||||
8192,1,131072,1,4.779465030878782,4.883405601140112,4.707481013610959,4.85471700085327,0.0,56696.089847013354,5,-0.07198401726782322,0.9849388965495606
|
||||
8192,1,131072,2,4.790953011251986,4.880544205661863,4.728371975943446,4.907831805758178,0.0,56880.19039196661,5,-0.06258103530853987,0.9869376645603573
|
||||
8192,1,2048,0,4.77885699365288,4.894876398611814,41.434570477576926,88.97331730695441,0.0,183.2046649651602,4,36.655713483924046,8.670393471202205
|
||||
8192,1,2048,1,4.788161953911185,4.949774022679776,41.68213551747613,83.5143867880106,0.0,175.55483896285295,4,36.89397356356494,8.705247633369687
|
||||
8192,1,2048,2,4.7893429873511195,4.874200583435595,23.186982492916286,67.25202781381086,0.0,131.23180496040732,4,18.397639505565166,4.841370215946989
|
||||
8192,1,32768,0,4.789774015080184,4.870833398308605,4.738486022688448,4.886626999359578,0.0,4500.839321000967,5,-0.051287992391735315,0.9892921895207875
|
||||
8192,1,32768,1,4.776834975928068,4.891659819986671,4.729953012429178,4.9245511763729155,0.0,4496.073378017172,5,-0.0468819634988904,0.9901855593221991
|
||||
8192,1,32768,2,4.784431017469615,4.866032593417913,4.782894975505769,4.8977664206177,0.0,4549.013931944501,5,-0.0015360419638454914,0.9996789499193871
|
||||
8192,1,65536,0,4.778854956384748,4.9255444086156785,4.633405013009906,4.895579582080245,0.0,15530.37424501963,5,-0.1454499433748424,0.9695638506080803
|
||||
8192,1,65536,1,4.784283053595573,4.8808404128067195,4.754905996378511,4.985795798711479,0.0,15584.887631004676,5,-0.02937705721706152,0.99385967408534
|
||||
8192,1,65536,2,4.787993966601789,4.9004736240021884,4.6836750116199255,5.0271204963792115,0.0,15587.390075030271,6,-0.1043189549818635,0.9782123879625725
|
||||
8192,1,8192,0,4.785028984770179,4.878618801012635,7.490115996915847,324.06569679733366,0.0,573.2795029762201,5,2.7050870121456683,1.565323014919123
|
||||
8192,1,8192,1,4.778591974172741,4.899543372448534,5.9131429879926145,336.8099076091312,0.0,606.6823820001446,5,1.1345510138198733,1.237423705550061
|
||||
8192,1,8192,2,4.78826800826937,4.90188361145556,6.276679981965572,324.8370993998833,0.0,571.7499859747477,5,1.488411973696202,1.310845585736994
|
||||
8192,4,131072,0,6.113810988608748,6.309205386787653,0.0,0.0,0.0,56702.702289039735,0,-6.113810988608748,0.0
|
||||
8192,4,131072,1,6.630807969486341,7.086459483252838,6.2820459716022015,4400.500871409893,0.0,56807.70832300186,150,-0.3487619978841394,0.9474027902045915
|
||||
8192,4,131072,2,6.073819473385811,6.344516028184444,6.326125003397465,4409.856556192978,0.0,56580.784838995896,149,0.2523055300116539,1.0415398467335428
|
||||
8192,4,2048,0,5.402160517405719,5.543816485442221,6.210724503034726,84.62208869168535,6.125201500253752,140.3041940066032,18,0.8085639856290072,1.1496741873966574
|
||||
8192,4,2048,1,6.067108013667166,6.381415005307645,0.0,0.0,0.0,140.06177097326145,0,-6.067108013667166,0.0
|
||||
8192,4,2048,2,5.400336522143334,5.536347016459331,38.15686801681295,85.07051098858938,5.25214200024493,134.67552902875468,13,32.756531494669616,7.065646346363043
|
||||
8192,4,32768,0,6.115561525803059,6.369604001520202,7.216634490760043,1314.6978712815326,5.17624247004278,4522.433568025008,50,1.101072964956984,1.1800444587649532
|
||||
8192,4,32768,1,6.070095987524837,6.3612310332246125,0.0,0.0,0.0,4508.074064040557,0,-6.070095987524837,0.0
|
||||
8192,4,32768,2,6.0734800063073635,6.312666402664036,12.442811043001711,1315.0411327951588,4.754714027512819,4556.892123946454,45,6.369331036694348,2.0487119460473635
|
||||
8192,4,65536,0,5.406292999396101,5.540905491216108,0.0,0.0,0.0,15581.590663990937,0,-5.406292999396101,0.0
|
||||
8192,4,65536,1,6.076910009142011,6.315114628523588,0.0,0.0,0.0,15574.196094006766,0,-6.076910009142011,0.0
|
||||
8192,4,65536,2,6.060379033442587,6.384042033459991,6.411670008674264,2077.4700703914277,4.8022730043157935,15603.720718005206,79,0.3512909752316773,1.0579651822589267
|
||||
8192,4,8192,0,6.110575021011755,6.416070973500609,8.451583969872445,515.3855616226792,5.358011490898207,574.6672929963097,18,2.34100894886069,1.3831077993169092
|
||||
8192,4,8192,1,6.051429023500532,6.398122606333345,0.0,0.0,0.0,573.6081749782898,0,-6.051429023500532,0.0
|
||||
8192,4,8192,2,6.064729997888207,6.366449000779539,0.0,0.0,0.0,574.1707819979638,0,-6.064729997888207,0.0
|
||||
8192,8,131072,0,7.737616979284212,7.99839201499708,10.740376019384712,4742.438135773409,7.792441989295185,57010.66731195897,335,3.0027590401005,1.388072845701685
|
||||
8192,8,131072,1,7.744895527139306,8.013638522243127,8.647068490972742,5123.228083999129,7.672236970392987,56970.40947602363,310,0.9021729638334364,1.116486137310966
|
||||
8192,8,131072,2,7.740180502878502,8.016240986762568,15.140031988266855,4820.136589207682,7.68946303287521,56993.02393599646,319,7.3998514853883535,1.9560308680962177
|
||||
8192,8,2048,0,7.741285488009453,8.022559515666217,8.103576023131609,124.87094267853536,7.6825070136692375,141.97922096354887,30,0.36229053512215614,1.046799789993963
|
||||
8192,8,2048,1,7.728310010861605,8.021069981623441,8.17067950265482,84.82906777062453,7.745136506855488,144.1582590341568,38,0.4423694917932153,1.0572401328584768
|
||||
8192,8,2048,2,7.662211020942777,8.034424972720444,8.87883099494502,87.23540699575096,7.592331967316568,143.27958395006135,39,1.216619974002242,1.1587818412566437
|
||||
8192,8,32768,0,7.295333489309996,7.422819995554164,11.429400008637458,1315.43214758276,7.8034960315562785,4523.641717038117,94,4.134066519327462,1.5666727265292526
|
||||
8192,8,32768,1,7.278127042809501,7.490781514206901,12.640403030673042,1315.491412486881,7.821676495950669,4519.993302994408,90,5.362275987863541,1.736765922925357
|
||||
8192,8,32768,2,7.684049021918327,8.047712198458612,10.752685484476388,1315.5166705255397,7.80402502277866,4517.200137954205,96,3.068636462558061,1.3993514947399404
|
||||
8192,8,65536,0,7.708174001891166,8.017168991500512,26.662671996746212,2496.8427699001018,7.768569514155388,15603.601168957539,160,18.954497994855046,3.459012729889679
|
||||
8192,8,65536,1,7.594842027174309,7.9874323040712625,13.054963492322713,2459.1690181812737,7.54699349636212,15620.474929979537,174,5.460121465148404,1.7189249553331216
|
||||
8192,8,65536,2,7.693717983784154,7.933055714238435,17.5579380011186,2458.176895044744,7.808708498487249,15622.32490995666,161,9.864220017334446,2.2821135422594123
|
||||
8192,8,8192,0,7.636573514901102,7.904737605713308,10.151655005756766,514.8188057704829,7.7977380133233964,575.7745200535282,37,2.515081490855664,1.3293468577167538
|
||||
8192,8,8192,1,7.687711506150663,7.965393498307094,9.002390026580542,524.0793236298487,7.753994490485638,592.1044679707848,45,1.3146785204298794,1.1710103870804793
|
||||
8192,8,8192,2,7.756220467854291,8.035426988499239,8.864110975991935,518.9726910321042,7.770269992761314,581.98908099439,41,1.1078905081376433,1.1428389655411813
|
||||
|
@@ -1,51 +0,0 @@
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 50331648, "duration_s": 0.06580113701056689, "t_start_unix": 1779885615.6732209, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885615.7390358}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 50331648, "duration_s": 0.0052392969955690205, "t_start_unix": 1779885616.0322638, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885616.0375087}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 201326592, "duration_s": 0.02050818904535845, "t_start_unix": 1779885616.2556505, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885616.2761638}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 201326592, "duration_s": 0.02001398801803589, "t_start_unix": 1779885616.4400308, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885616.46005}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 805306368, "duration_s": 0.08249958901433274, "t_start_unix": 1779885617.072654, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885617.1551628}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 805306368, "duration_s": 0.08082435996038839, "t_start_unix": 1779885617.7853239, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885617.866155}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.00040365802124142647, "t_start_unix": 1779885642.3123364, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.3127441}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.000374739000108093, "t_start_unix": 1779885642.3945863, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.3949661}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 50331648, "duration_s": 0.005158354004379362, "t_start_unix": 1779885642.4766958, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.481858}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 50331648, "duration_s": 0.00513053999748081, "t_start_unix": 1779885642.5614145, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.5665495}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 50331648, "duration_s": 0.0051341859507374465, "t_start_unix": 1779885642.6461189, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.6512585}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 100663296, "duration_s": 0.010125375993084162, "t_start_unix": 1779885642.7582293, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.76836}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 100663296, "duration_s": 0.010367848037276417, "t_start_unix": 1779885642.878179, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885642.8885527}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 100663296, "duration_s": 0.010208865976892412, "t_start_unix": 1779885642.997267, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.0074801}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 100663296, "duration_s": 0.01015280099818483, "t_start_unix": 1779885643.1138487, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.124007}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 100663296, "duration_s": 0.010063701018225402, "t_start_unix": 1779885643.2299926, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.2400591}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.00040220399387180805, "t_start_unix": 1779885643.4041483, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.4045541}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.0003751559997908771, "t_start_unix": 1779885643.569847, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.5702271}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 201326592, "duration_s": 0.020166583999525756, "t_start_unix": 1779885643.7347023, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.7548745}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 201326592, "duration_s": 0.019989027990959585, "t_start_unix": 1779885643.9190295, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885643.939023}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 201326592, "duration_s": 0.02000429102918133, "t_start_unix": 1779885644.1027336, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885644.122742}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 402653184, "duration_s": 0.040499016002286226, "t_start_unix": 1779885644.419112, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885644.4596438}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 402653184, "duration_s": 0.04015034798067063, "t_start_unix": 1779885644.761118, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885644.8012745}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 402653184, "duration_s": 0.04006708099041134, "t_start_unix": 1779885645.1008255, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885645.1408994}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 402653184, "duration_s": 0.04012463602703065, "t_start_unix": 1779885645.440819, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885645.4809544}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 402653184, "duration_s": 0.04009692999534309, "t_start_unix": 1779885645.7793777, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885645.8194828}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.00046314700739458203, "t_start_unix": 1779885646.445756, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885646.4462252}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1572864, "duration_s": 0.0004179630195721984, "t_start_unix": 1779885647.0757725, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885647.0761962}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 805306368, "duration_s": 0.08170936699025333, "t_start_unix": 1779885647.7075222, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885647.7892444}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 805306368, "duration_s": 0.08086105203256011, "t_start_unix": 1779885648.422354, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885648.503224}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 805306368, "duration_s": 0.08071460900828242, "t_start_unix": 1779885649.1382625, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885649.2189863}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1610612736, "duration_s": 0.1638482889975421, "t_start_unix": 1779885650.8273866, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885650.991251}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1610612736, "duration_s": 0.16476035403320566, "t_start_unix": 1779885652.5951493, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885652.7599196}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1610612736, "duration_s": 0.1617715489701368, "t_start_unix": 1779885654.3595936, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885654.5214472}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1610612736, "duration_s": 0.1616577100357972, "t_start_unix": 1779885656.125777, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885656.2874432}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 1610612736, "duration_s": 0.1617818950326182, "t_start_unix": 1779885657.8867118, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885658.0485108}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 3221225472, "duration_s": 0.30771408596774563, "t_start_unix": 1779885662.69878, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885663.0065253}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 3221225472, "duration_s": 0.3096057590446435, "t_start_unix": 1779885667.6915898, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885668.0012283}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 3221225472, "duration_s": 0.3080696280230768, "t_start_unix": 1779885672.684307, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885672.9923992}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 3221225472, "duration_s": 0.5257709489669651, "t_start_unix": 1779885677.7157974, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885678.2415998}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 3221225472, "duration_s": 0.5268570999614894, "t_start_unix": 1779885682.9683614, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885683.4952588}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 6442450944, "duration_s": 1.901308581000194, "t_start_unix": 1779885698.9342654, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885700.835605}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 6442450944, "duration_s": 1.9211939970264211, "t_start_unix": 1779885716.3570645, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885718.2782962}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 6442450944, "duration_s": 1.7335722800344229, "t_start_unix": 1779885733.838071, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885735.571683}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 6442450944, "duration_s": 0.9299940629862249, "t_start_unix": 1779885751.0288215, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885751.9588547}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 6442450944, "duration_s": 0.6535220990190282, "t_start_unix": 1779885767.3456392, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885767.9991918}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 12884901888, "duration_s": 1.2830275790183805, "t_start_unix": 1779885822.9884846, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885824.2715507}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 12884901888, "duration_s": 5.053741328010801, "t_start_unix": 1779885879.5277712, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885884.5815506}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 12884901888, "duration_s": 9.158571614010725, "t_start_unix": 1779885939.9274101, "ret": 0, "tp_rank": 0, "t_log_unix": 1779885949.086019}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 12884901888, "duration_s": 2.8184301540022716, "t_start_unix": 1779886004.3031003, "ret": 0, "tp_rank": 0, "t_log_unix": 1779886007.121565}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.133:16878", "total_bytes": 12884901888, "duration_s": 1.4530277770245448, "t_start_unix": 1779886062.159227, "ret": 0, "tp_rank": 0, "t_log_unix": 1779886063.6122832}
|
||||
@@ -1,51 +0,0 @@
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 50331648, "duration_s": 0.023202952987048775, "t_start_unix": 1779879143.174031, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879143.1972432}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 50331648, "duration_s": 0.005375694017857313, "t_start_unix": 1779879143.2982283, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879143.3036084}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 201326592, "duration_s": 0.021170366962905973, "t_start_unix": 1779879143.5159554, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879143.5371296}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 201326592, "duration_s": 0.020726953051052988, "t_start_unix": 1779879143.6974514, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879143.7181835}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 805306368, "duration_s": 0.08536655298667029, "t_start_unix": 1779879144.3294952, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879144.4148676}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 805306368, "duration_s": 0.08367906499188393, "t_start_unix": 1779879145.0419943, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879145.125678}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.0004059679922647774, "t_start_unix": 1779879221.7078288, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879221.7082384}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.000346789020113647, "t_start_unix": 1779879221.7838593, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879221.7842083}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 50331648, "duration_s": 0.005353622022084892, "t_start_unix": 1779879221.8607252, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879221.8660822}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 50331648, "duration_s": 0.005279594974126667, "t_start_unix": 1779879221.9432015, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879221.9484842}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 50331648, "duration_s": 0.0053006180096417665, "t_start_unix": 1779879222.0243337, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.0296378}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 100663296, "duration_s": 0.010396577999927104, "t_start_unix": 1779879222.130936, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.141335}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 100663296, "duration_s": 0.010438029014039785, "t_start_unix": 1779879222.2442062, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.2546473}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 100663296, "duration_s": 0.010436972021125257, "t_start_unix": 1779879222.3581295, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.3685696}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 100663296, "duration_s": 0.010396371013484895, "t_start_unix": 1779879222.4725878, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.4829867}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 100663296, "duration_s": 0.010352785000577569, "t_start_unix": 1779879222.5837166, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.5940716}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.00034007197245955467, "t_start_unix": 1779879222.7521152, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.752458}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.00041691696969792247, "t_start_unix": 1779879222.9143836, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879222.914805}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 201326592, "duration_s": 0.020633380976505578, "t_start_unix": 1779879223.0778644, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879223.098502}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 201326592, "duration_s": 0.020639199996367097, "t_start_unix": 1779879223.2603853, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879223.2810278}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 201326592, "duration_s": 0.020575353992171586, "t_start_unix": 1779879223.4418828, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879223.4624615}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 402653184, "duration_s": 0.041439525957684964, "t_start_unix": 1779879223.7544343, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879223.7958782}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 402653184, "duration_s": 0.04152030003024265, "t_start_unix": 1779879224.0914912, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879224.133016}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 402653184, "duration_s": 0.04148670402355492, "t_start_unix": 1779879224.4262393, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879224.4677298}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 402653184, "duration_s": 0.04146742797456682, "t_start_unix": 1779879224.7617002, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879224.8031723}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 402653184, "duration_s": 0.04143296502297744, "t_start_unix": 1779879225.0978234, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879225.1392617}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.0003991159610450268, "t_start_unix": 1779879225.760789, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879225.7611918}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1572864, "duration_s": 0.00041423802031204104, "t_start_unix": 1779879226.3864496, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879226.3868673}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 805306368, "duration_s": 0.08309489500243217, "t_start_unix": 1779879227.0107942, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879227.0938945}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 805306368, "duration_s": 0.08372796402545646, "t_start_unix": 1779879227.7207224, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879227.8044555}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 805306368, "duration_s": 0.08398396399570629, "t_start_unix": 1779879228.4314566, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879228.5154452}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1610612736, "duration_s": 0.16950496198842302, "t_start_unix": 1779879230.1334376, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879230.3029544}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1610612736, "duration_s": 0.16713789198547602, "t_start_unix": 1779879231.8981037, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879232.0652575}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1610612736, "duration_s": 0.16713115200400352, "t_start_unix": 1779879233.6608078, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879233.827945}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1610612736, "duration_s": 0.16709016199456528, "t_start_unix": 1779879235.419875, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879235.5869706}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 1610612736, "duration_s": 0.166486973001156, "t_start_unix": 1779879237.1821773, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879237.34867}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 3221225472, "duration_s": 0.31926770601421595, "t_start_unix": 1779879241.9880297, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879242.3073065}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 3221225472, "duration_s": 0.3197040680097416, "t_start_unix": 1779879246.9779432, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879247.2976692}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 3221225472, "duration_s": 0.32088329299585894, "t_start_unix": 1779879251.9643052, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879252.285209}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 3221225472, "duration_s": 0.5439103110111319, "t_start_unix": 1779879256.9989722, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879257.5428913}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 3221225472, "duration_s": 0.5193864739849232, "t_start_unix": 1779879262.2562187, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879262.7756212}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 6442450944, "duration_s": 1.9844180009677075, "t_start_unix": 1779879278.199048, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879280.1834915}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 6442450944, "duration_s": 2.1099297259934247, "t_start_unix": 1779879295.6967168, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879297.8066647}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 6442450944, "duration_s": 1.8950715209939517, "t_start_unix": 1779879313.3236735, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879315.2187643}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 6442450944, "duration_s": 0.9277855920372531, "t_start_unix": 1779879330.6715357, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879331.599329}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 6442450944, "duration_s": 0.6652462020283565, "t_start_unix": 1779879346.9950044, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879347.6602724}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 12884901888, "duration_s": 1.3330365709844045, "t_start_unix": 1779879402.7169023, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879404.04997}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 12884901888, "duration_s": 5.839069904992357, "t_start_unix": 1779879459.0566247, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879464.8957155}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 12884901888, "duration_s": 9.862486142024864, "t_start_unix": 1779879519.9567635, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879529.8192694}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 12884901888, "duration_s": 2.8350498770014383, "t_start_unix": 1779879584.9780834, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879587.813154}
|
||||
{"event": "send_blocks", "remote_session": "172.27.123.142:16428", "total_bytes": 12884901888, "duration_s": 1.485496642999351, "t_start_unix": 1779879642.639775, "ret": 0, "tp_rank": 0, "t_log_unix": 1779879644.1252885}
|
||||
@@ -1,102 +0,0 @@
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ad00672f263a6643-0-9479211a"], "t_start_unix": 1779879143.1678784, "tp_rank": 0, "t_log_unix": 1779879143.167884}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ad00672f263a6643-0-9479211a"], "duration_s": 0.03333390498301014, "t_start_unix": 1779879143.1678784, "tp_rank": 0, "t_log_unix": 1779879143.201217}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ace77e2b02f9f141-0-b3c061bc"], "t_start_unix": 1779879143.2968972, "tp_rank": 0, "t_log_unix": 1779879143.2969005}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ace77e2b02f9f141-0-b3c061bc"], "duration_s": 0.007019245007541031, "t_start_unix": 1779879143.2968972, "tp_rank": 0, "t_log_unix": 1779879143.3039184}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4a2366879c68ded-0-8ac4098e"], "t_start_unix": 1779879143.5146625, "tp_rank": 0, "t_log_unix": 1779879143.5146651}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4a2366879c68ded-0-8ac4098e"], "duration_s": 0.02278437599306926, "t_start_unix": 1779879143.5146625, "tp_rank": 0, "t_log_unix": 1779879143.537448}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8690cafcace0d5e2-0-b89f33d2"], "t_start_unix": 1779879143.6958342, "tp_rank": 0, "t_log_unix": 1779879143.6958375}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8690cafcace0d5e2-0-b89f33d2"], "duration_s": 0.022794076008722186, "t_start_unix": 1779879143.6958342, "tp_rank": 0, "t_log_unix": 1779879143.71863}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b087e2ec4cfa8eb7-0-b908f425"], "t_start_unix": 1779879144.3279662, "tp_rank": 0, "t_log_unix": 1779879144.3279696}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b087e2ec4cfa8eb7-0-b908f425"], "duration_s": 0.08753501297906041, "t_start_unix": 1779879144.3279662, "tp_rank": 0, "t_log_unix": 1779879144.415505}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a115d16ff5575e08-0-9fa81984"], "t_start_unix": 1779879145.040141, "tp_rank": 0, "t_log_unix": 1779879145.0401456}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a115d16ff5575e08-0-9fa81984"], "duration_s": 0.0860149699728936, "t_start_unix": 1779879145.040141, "tp_rank": 0, "t_log_unix": 1779879145.1261594}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9e585ed083951df5-0-b03f812b"], "t_start_unix": 1779879221.7062025, "tp_rank": 0, "t_log_unix": 1779879221.7062056}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9e585ed083951df5-0-b03f812b"], "duration_s": 0.002459956973325461, "t_start_unix": 1779879221.7062025, "tp_rank": 0, "t_log_unix": 1779879221.7086644}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9271d403c044eadd-0-9c3c4639"], "t_start_unix": 1779879221.7826598, "tp_rank": 0, "t_log_unix": 1779879221.782662}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9271d403c044eadd-0-9c3c4639"], "duration_s": 0.0020201010047458112, "t_start_unix": 1779879221.7826598, "tp_rank": 0, "t_log_unix": 1779879221.7846813}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82a580cefd3e2440-0-a383c3c4"], "t_start_unix": 1779879221.859549, "tp_rank": 0, "t_log_unix": 1779879221.8595514}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82a580cefd3e2440-0-a383c3c4"], "duration_s": 0.006836243963334709, "t_start_unix": 1779879221.859549, "tp_rank": 0, "t_log_unix": 1779879221.8663864}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a31cb4bc9e7f63d2-0-8f48aacd"], "t_start_unix": 1779879221.9419758, "tp_rank": 0, "t_log_unix": 1779879221.9419782}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a31cb4bc9e7f63d2-0-8f48aacd"], "duration_s": 0.00694335694424808, "t_start_unix": 1779879221.9419758, "tp_rank": 0, "t_log_unix": 1779879221.9489205}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a9dfc1a5b425d994-0-a0930098"], "t_start_unix": 1779879222.0232244, "tp_rank": 0, "t_log_unix": 1779879222.0232272}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a9dfc1a5b425d994-0-a0930098"], "duration_s": 0.006697195000015199, "t_start_unix": 1779879222.0232244, "tp_rank": 0, "t_log_unix": 1779879222.0299227}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9712857755af2efc-0-90b2dc9b"], "t_start_unix": 1779879222.1297998, "tp_rank": 0, "t_log_unix": 1779879222.1298025}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9712857755af2efc-0-90b2dc9b"], "duration_s": 0.01183948403922841, "t_start_unix": 1779879222.1297998, "tp_rank": 0, "t_log_unix": 1779879222.1416407}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4f0a10dee65acbe-0-a3c132fc"], "t_start_unix": 1779879222.243023, "tp_rank": 0, "t_log_unix": 1779879222.243025}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4f0a10dee65acbe-0-a3c132fc"], "duration_s": 0.01214482297655195, "t_start_unix": 1779879222.243023, "tp_rank": 0, "t_log_unix": 1779879222.2551687}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4c514b80b52a3f2-0-bcd24f8e"], "t_start_unix": 1779879222.3569698, "tp_rank": 0, "t_log_unix": 1779879222.356972}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4c514b80b52a3f2-0-bcd24f8e"], "duration_s": 0.011961110983975232, "t_start_unix": 1779879222.3569698, "tp_rank": 0, "t_log_unix": 1779879222.368932}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ac7118d8090d181c-0-8af4adf0"], "t_start_unix": 1779879222.4715128, "tp_rank": 0, "t_log_unix": 1779879222.4715152}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ac7118d8090d181c-0-8af4adf0"], "duration_s": 0.011788576026447117, "t_start_unix": 1779879222.4715128, "tp_rank": 0, "t_log_unix": 1779879222.4833028}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-85291bcb93aaf638-0-868db1a8"], "t_start_unix": 1779879222.5826046, "tp_rank": 0, "t_log_unix": 1779879222.5826073}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-85291bcb93aaf638-0-868db1a8"], "duration_s": 0.0118055299972184, "t_start_unix": 1779879222.5826046, "tp_rank": 0, "t_log_unix": 1779879222.5944116}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a448cf2e059ba0c9-0-a1360796"], "t_start_unix": 1779879222.750828, "tp_rank": 0, "t_log_unix": 1779879222.7508304}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a448cf2e059ba0c9-0-a1360796"], "duration_s": 0.0021119200391694903, "t_start_unix": 1779879222.750828, "tp_rank": 0, "t_log_unix": 1779879222.7529414}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b486fd9e945a4658-0-8bb561cd"], "t_start_unix": 1779879222.913044, "tp_rank": 0, "t_log_unix": 1779879222.9130466}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b486fd9e945a4658-0-8bb561cd"], "duration_s": 0.0022232600022107363, "t_start_unix": 1779879222.913044, "tp_rank": 0, "t_log_unix": 1779879222.9152684}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82da2bfe65f276c6-0-88d9a9a2"], "t_start_unix": 1779879223.0765986, "tp_rank": 0, "t_log_unix": 1779879223.0766027}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82da2bfe65f276c6-0-88d9a9a2"], "duration_s": 0.022250515001360327, "t_start_unix": 1779879223.0765986, "tp_rank": 0, "t_log_unix": 1779879223.0988505}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-93bd777652eba5f3-0-9ec3d058"], "t_start_unix": 1779879223.2591784, "tp_rank": 0, "t_log_unix": 1779879223.2591808}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-93bd777652eba5f3-0-9ec3d058"], "duration_s": 0.022157608007546514, "t_start_unix": 1779879223.2591784, "tp_rank": 0, "t_log_unix": 1779879223.2813375}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81f950480a3cabf9-0-bbf8584f"], "t_start_unix": 1779879223.4402068, "tp_rank": 0, "t_log_unix": 1779879223.440209}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81f950480a3cabf9-0-bbf8584f"], "duration_s": 0.022589912987314165, "t_start_unix": 1779879223.4402068, "tp_rank": 0, "t_log_unix": 1779879223.4627984}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b109ed06b5882659-0-8d14993c"], "t_start_unix": 1779879223.7529812, "tp_rank": 0, "t_log_unix": 1779879223.752984}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b109ed06b5882659-0-8d14993c"], "duration_s": 0.043345845013391227, "t_start_unix": 1779879223.7529812, "tp_rank": 0, "t_log_unix": 1779879223.796329}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8a57776c81d64b2c-0-ace8fb2b"], "t_start_unix": 1779879224.0899644, "tp_rank": 0, "t_log_unix": 1779879224.0899673}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8a57776c81d64b2c-0-ace8fb2b"], "duration_s": 0.04341953102266416, "t_start_unix": 1779879224.0899644, "tp_rank": 0, "t_log_unix": 1779879224.1333857}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9b1a5dce18758450-0-b17b3649"], "t_start_unix": 1779879224.424807, "tp_rank": 0, "t_log_unix": 1779879224.42481}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9b1a5dce18758450-0-b17b3649"], "duration_s": 0.04336977802449837, "t_start_unix": 1779879224.424807, "tp_rank": 0, "t_log_unix": 1779879224.4681823}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8c7d412b85f43ed7-0-9dea4add"], "t_start_unix": 1779879224.7599711, "tp_rank": 0, "t_log_unix": 1779879224.7599735}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8c7d412b85f43ed7-0-9dea4add"], "duration_s": 0.043769759009592235, "t_start_unix": 1779879224.7599711, "tp_rank": 0, "t_log_unix": 1779879224.8037443}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8860308db3f010a5-0-ad51eb46"], "t_start_unix": 1779879225.0962389, "tp_rank": 0, "t_log_unix": 1779879225.0962446}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8860308db3f010a5-0-ad51eb46"], "duration_s": 0.043612666020635515, "t_start_unix": 1779879225.0962389, "tp_rank": 0, "t_log_unix": 1779879225.1398532}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-86cca1a2b9427801-0-ba41ade7"], "t_start_unix": 1779879225.7592747, "tp_rank": 0, "t_log_unix": 1779879225.759278}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-86cca1a2b9427801-0-ba41ade7"], "duration_s": 0.002386144013144076, "t_start_unix": 1779879225.7592747, "tp_rank": 0, "t_log_unix": 1779879225.7616625}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a208c6d804293be7-0-94d265ab"], "t_start_unix": 1779879226.384918, "tp_rank": 0, "t_log_unix": 1779879226.384921}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a208c6d804293be7-0-94d265ab"], "duration_s": 0.0023903060355223715, "t_start_unix": 1779879226.384918, "tp_rank": 0, "t_log_unix": 1779879226.3873098}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b53bea2317cc1211-0-8fcad8a8"], "t_start_unix": 1779879227.0092332, "tp_rank": 0, "t_log_unix": 1779879227.0092363}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b53bea2317cc1211-0-8fcad8a8"], "duration_s": 0.08524628396844491, "t_start_unix": 1779879227.0092332, "tp_rank": 0, "t_log_unix": 1779879227.094482}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9daf909593bbdf03-0-8fd7d50e"], "t_start_unix": 1779879227.7190688, "tp_rank": 0, "t_log_unix": 1779879227.719072}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9daf909593bbdf03-0-8fd7d50e"], "duration_s": 0.08596085698809475, "t_start_unix": 1779879227.7190688, "tp_rank": 0, "t_log_unix": 1779879227.8050315}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9ef40f3b6d736128-0-8e8e1c30"], "t_start_unix": 1779879228.4297745, "tp_rank": 0, "t_log_unix": 1779879228.4297774}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9ef40f3b6d736128-0-8e8e1c30"], "duration_s": 0.0860762019874528, "t_start_unix": 1779879228.4297745, "tp_rank": 0, "t_log_unix": 1779879228.5158527}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-851e5d7e3e83d7ea-0-a66a5e0b"], "t_start_unix": 1779879230.131392, "tp_rank": 0, "t_log_unix": 1779879230.1313956}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-851e5d7e3e83d7ea-0-a66a5e0b"], "duration_s": 0.1721468890318647, "t_start_unix": 1779879230.131392, "tp_rank": 0, "t_log_unix": 1779879230.3035412}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9be12af6a9ccccf5-0-af1230c7"], "t_start_unix": 1779879231.896075, "tp_rank": 0, "t_log_unix": 1779879231.896078}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9be12af6a9ccccf5-0-af1230c7"], "duration_s": 0.16974544001277536, "t_start_unix": 1779879231.896075, "tp_rank": 0, "t_log_unix": 1779879232.0658224}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b61b9b237366297b-0-9832f0e3"], "t_start_unix": 1779879233.6589305, "tp_rank": 0, "t_log_unix": 1779879233.6589334}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b61b9b237366297b-0-9832f0e3"], "duration_s": 0.16975757898762822, "t_start_unix": 1779879233.6589305, "tp_rank": 0, "t_log_unix": 1779879233.8286898}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bae0d0efe47ece8f-0-affbc685"], "t_start_unix": 1779879235.4181106, "tp_rank": 0, "t_log_unix": 1779879235.418114}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bae0d0efe47ece8f-0-affbc685"], "duration_s": 0.1695251659839414, "t_start_unix": 1779879235.4181106, "tp_rank": 0, "t_log_unix": 1779879235.587638}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a34bc73c9cd2efc1-0-90d647fc"], "t_start_unix": 1779879237.1803744, "tp_rank": 0, "t_log_unix": 1779879237.1803775}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a34bc73c9cd2efc1-0-90d647fc"], "duration_s": 0.16962904302636161, "t_start_unix": 1779879237.1803744, "tp_rank": 0, "t_log_unix": 1779879237.3500054}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-89a36c12ee6b0ff3-0-9fddbc0f"], "t_start_unix": 1779879241.9859307, "tp_rank": 0, "t_log_unix": 1779879241.9859338}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-89a36c12ee6b0ff3-0-9fddbc0f"], "duration_s": 0.32203804596792907, "t_start_unix": 1779879241.9859307, "tp_rank": 0, "t_log_unix": 1779879242.3079708}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8d65512eb7e3c36c-0-8b23597c"], "t_start_unix": 1779879246.9755645, "tp_rank": 0, "t_log_unix": 1779879246.9755676}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8d65512eb7e3c36c-0-8b23597c"], "duration_s": 0.3227974839974195, "t_start_unix": 1779879246.9755645, "tp_rank": 0, "t_log_unix": 1779879247.2983644}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a13c271ecbbca78b-0-b76a0370"], "t_start_unix": 1779879251.9618897, "tp_rank": 0, "t_log_unix": 1779879251.961893}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a13c271ecbbca78b-0-b76a0370"], "duration_s": 0.3240378479822539, "t_start_unix": 1779879251.9618897, "tp_rank": 0, "t_log_unix": 1779879252.2859304}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bada04ec8c556aca-0-a263d637"], "t_start_unix": 1779879256.9512377, "tp_rank": 0, "t_log_unix": 1779879256.9512408}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bada04ec8c556aca-0-a263d637"], "duration_s": 0.5924434679909609, "t_start_unix": 1779879256.9512377, "tp_rank": 0, "t_log_unix": 1779879257.5436878}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9641a077022e6123-0-8c3c0975"], "t_start_unix": 1779879262.2127163, "tp_rank": 0, "t_log_unix": 1779879262.2127194}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9641a077022e6123-0-8c3c0975"], "duration_s": 0.5644763479940593, "t_start_unix": 1779879262.2127163, "tp_rank": 0, "t_log_unix": 1779879262.777195}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bb3a4e5084af8c3a-0-bdfa0931"], "t_start_unix": 1779879278.1063075, "tp_rank": 0, "t_log_unix": 1779879278.106311}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bb3a4e5084af8c3a-0-bdfa0931"], "duration_s": 2.0784930550144054, "t_start_unix": 1779879278.1063075, "tp_rank": 0, "t_log_unix": 1779879280.1848085}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-91b951f85c93a71b-0-8396bee5"], "t_start_unix": 1779879295.600993, "tp_rank": 0, "t_log_unix": 1779879295.6009963}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-91b951f85c93a71b-0-8396bee5"], "duration_s": 2.2067435560165904, "t_start_unix": 1779879295.600993, "tp_rank": 0, "t_log_unix": 1779879297.8077443}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81d236ecb6aadadf-0-ac184d51"], "t_start_unix": 1779879313.2315958, "tp_rank": 0, "t_log_unix": 1779879313.2315989}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81d236ecb6aadadf-0-ac184d51"], "duration_s": 1.9879729640088044, "t_start_unix": 1779879313.2315958, "tp_rank": 0, "t_log_unix": 1779879315.219571}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4c76c62b44c4295-0-b007a6ed"], "t_start_unix": 1779879330.6154163, "tp_rank": 0, "t_log_unix": 1779879330.6154196}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4c76c62b44c4295-0-b007a6ed"], "duration_s": 0.9849357060156763, "t_start_unix": 1779879330.6154163, "tp_rank": 0, "t_log_unix": 1779879331.6003594}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a06d4b774a8af9a5-0-980e9d23"], "t_start_unix": 1779879346.990221, "tp_rank": 0, "t_log_unix": 1779879346.9902246}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a06d4b774a8af9a5-0-980e9d23"], "duration_s": 0.6725030990201049, "t_start_unix": 1779879346.990221, "tp_rank": 0, "t_log_unix": 1779879347.6627269}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bf0d435e06e3349f-0-8507c933"], "t_start_unix": 1779879402.7123013, "tp_rank": 0, "t_log_unix": 1779879402.7123044}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bf0d435e06e3349f-0-8507c933"], "duration_s": 1.3384539679973386, "t_start_unix": 1779879402.7123013, "tp_rank": 0, "t_log_unix": 1779879404.0507588}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f87ae0fb0c7eec8-0-a8a1daea"], "t_start_unix": 1779879458.9232886, "tp_rank": 0, "t_log_unix": 1779879458.9232917}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f87ae0fb0c7eec8-0-a8a1daea"], "duration_s": 5.973284716019407, "t_start_unix": 1779879458.9232886, "tp_rank": 0, "t_log_unix": 1779879464.896582}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a62e48e40e6c6ad7-0-acca9741"], "t_start_unix": 1779879519.7647448, "tp_rank": 0, "t_log_unix": 1779879519.7647479}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a62e48e40e6c6ad7-0-acca9741"], "duration_s": 10.056511385017075, "t_start_unix": 1779879519.7647448, "tp_rank": 0, "t_log_unix": 1779879529.8212643}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-824479d53bab40e4-0-af951a11"], "t_start_unix": 1779879584.888362, "tp_rank": 0, "t_log_unix": 1779879584.8883653}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-824479d53bab40e4-0-af951a11"], "duration_s": 2.925714804965537, "t_start_unix": 1779879584.888362, "tp_rank": 0, "t_log_unix": 1779879587.814085}
|
||||
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f06f19c981c0b3f-0-b3afb370"], "t_start_unix": 1779879642.6076336, "tp_rank": 0, "t_log_unix": 1779879642.6076367}
|
||||
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f06f19c981c0b3f-0-b3afb370"], "duration_s": 1.5183607729850337, "t_start_unix": 1779879642.6076336, "tp_rank": 0, "t_log_unix": 1779879644.1259985}
|
||||
@@ -1,314 +0,0 @@
|
||||
# MB2 — Mooncake KV Transfer Cost (vanilla vLLM 0.18.1)
|
||||
|
||||
Persistent record of the per-stage KV transfer microbench used in §3.2 of
|
||||
the EAR paper. Re-runs append a dated section at the bottom; the
|
||||
**Summary** block at the top is what gets cited in the paper.
|
||||
|
||||
---
|
||||
|
||||
## Summary (latest)
|
||||
|
||||
| Path | Steady-state BW | Agentic-tail p99 transfer (11.5 GiB KV) |
|
||||
|---|---|---|
|
||||
| **intra-node** (dash1 GPU 0↔1) | **~9.7 GB/s** (96 MiB – 3 GiB) | p50 **1.9 s** · min **1.5 s** · max **10 s** |
|
||||
| **inter-node** (dash1 GPU0 → dash2 GPU0, 200 Gbps RoCE) | **~10.0 GB/s** (essentially identical) | p50 **1.7 s** · min **1.3 s** · max **9.2 s** |
|
||||
|
||||
**Cross-cutting finding** (2026-05-27): **Mooncake transfer cost is
|
||||
topology-independent** on this hardware. Intra-node and inter-node curves
|
||||
are statistically indistinguishable (see `figs/mb2_transfer_time_compare.png`,
|
||||
`figs/mb2_transfer_bw_compare.png`). Mechanism: Mooncake's
|
||||
`batch_transfer_sync_write` always goes through the RDMA NIC, including
|
||||
the intra-node case (RDMA loopback). The 200 Gbps NIC, not NVLink, is
|
||||
the bottleneck. **Implication for §3.2**: PD-disaggregation does not
|
||||
get cheaper by co-locating P and D on the same node — the ~9.7 GB/s
|
||||
ceiling applies regardless. Halving the transfer cost cannot be bought
|
||||
back by topology.
|
||||
|
||||
**What MB2 actually measures**: the **per-request charge** that
|
||||
PD-disagg pays for every routed request — `T_transfer ≈ KV_size / 9.7
|
||||
GB/s`. For agentic this is **8 ms (192 MiB / trace lower) – 1.9 s
|
||||
(11.5 GiB / p99)**.
|
||||
|
||||
**⚠ Correction (2026-05-27)**: an earlier version of this README
|
||||
framed §3.2 as "transfer cost (1.5–10 s) >> decode duration (50–200 ms),
|
||||
so PD-disagg loses on cost-vs-benefit." That accounting was wrong:
|
||||
PD-disagg's phase-isolation benefit is **per-prefill-event** and equals
|
||||
`D × T_prefill` (aggregate across stalled decode streams), not the
|
||||
single-request decode duration. With trace-mean `T_prefill = 4.5 s` and
|
||||
D = 8, the benefit is ~36 s — far larger than the ~0.32 s transfer
|
||||
cost. PD-disagg's phase-isolation axis is a *win*, not a loss.
|
||||
|
||||
The actual reason static PD-disagg fails in agentic is **D-side KV
|
||||
capacity** (`figs/f4b_pdsep_kv_wall.png`), not a cost-vs-benefit
|
||||
imbalance. See `RESULTS_SUMMARY.md` section 4 for the corrected
|
||||
framing. MB2 still serves as the source of the per-request transfer
|
||||
cost number used in that analysis.
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
| Component | Value |
|
||||
|---|---|
|
||||
| Host | `dash1` (`ds-6348bee4-1-...-rwkv2`), 8× NVIDIA H20 96 GiB, driver 570.133.20 |
|
||||
| Venv | `/home/admin/cpfs/wjh/agentic-kv-fresh/.venv` (shared via cpfs from any dash host) |
|
||||
| vLLM | 0.18.1 official wheel |
|
||||
| mooncake-transfer-engine | 0.3.11.post1 (`pip install mooncake-transfer-engine`) |
|
||||
| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct` |
|
||||
| Per-token KV | 98304 B |
|
||||
| kv_role | `kv_both` on both instances (see *Known limitations* re kv_producer/kv_consumer) |
|
||||
| Per-instance config | `--tensor-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 200000 --enable-prefix-caching` |
|
||||
|
||||
## Method
|
||||
|
||||
3-step black-box bench:
|
||||
|
||||
1. `do_remote_decode` to A (producer) with a client-generated `transfer_id`.
|
||||
`max_tokens=1`; A computes prefill and parks the KV for later pull.
|
||||
2. `do_remote_prefill` to B (consumer) with the same `transfer_id` plus
|
||||
`remote_engine_id` (from A's `/query` on bootstrap port) and
|
||||
`remote_bootstrap_addr` (`http://127.0.0.1:8998`). **This step
|
||||
triggers the actual KV transfer; it is the measured step.**
|
||||
3. Plain `completion` on B (`--skip-verify` off): expect `cached_tokens
|
||||
≈ prompt_len`, confirming the KV landed on B.
|
||||
|
||||
Per-stage breakdown is obtained by instrumenting the vLLM-shipped
|
||||
`MooncakeConnector` (NOT the mooncake-package's `mooncake_connector_v1`,
|
||||
which vLLM 0.18.1 does not load) at two sites:
|
||||
|
||||
- **`_send_blocks`** (P-side, line 980): emits `send_blocks` event with
|
||||
`total_bytes`, `duration_s`, `t_start_unix`. The `duration_s` is the
|
||||
wall-time of a single `batch_transfer_sync_write` call — **this is
|
||||
what we call `pure_transfer`**.
|
||||
- **`receive_kv_from_single_worker`** (D-side, line 1139, async):
|
||||
emits `receive_kv_enter` at function start and `receive_kv_finish`
|
||||
on FINISH-status response. The wall-time between them is
|
||||
**`rx_total`** (= ZMQ round-trip + setup + pure_transfer + ack).
|
||||
|
||||
Pairing across A's and B's logs is by **time window**: each B
|
||||
(enter, finish) pair is matched to the A send_blocks whose
|
||||
`t_start_unix` falls in `[rx_t_start, rx_t_end]`. With single-request
|
||||
benchmarks this is unambiguous.
|
||||
|
||||
Scripts:
|
||||
- `microbench/fresh_setup/start_vllm_pair.sh` — bring up pair + apply/revert patch
|
||||
- `microbench/fresh_setup/instrument_mooncake.py` — apply/revert MB2 patches
|
||||
- `microbench/fresh_setup/mb2_kv_transfer.py` — client (3-step bench loop)
|
||||
- `microbench/fresh_setup/analyze_mb2.py` — pair A/B events into per-size table
|
||||
- `microbench/fresh_setup/plot_mb2.py` — log-log time + bandwidth curves
|
||||
|
||||
## Results — intra-node (2026-05-27, dash1 GPU 0+1, kv_both)
|
||||
|
||||
Raw events: `A_intra_kvboth.jsonl`, `B_intra_kvboth.jsonl`.
|
||||
Joined + aggregated: `intra_kvboth_breakdown.json`.
|
||||
Figures: `figs/mb2_transfer_time_intra.png`, `figs/mb2_transfer_bw_intra.png`.
|
||||
|
||||
| input_tokens | KV (MiB) | n | pure_ms p50 | pure_ms max | rx_total_ms | overhead_ms | BW p50 (GB/s) | BW max (GB/s) |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 512 | 48 | 5 | 5.3 | 5.6 | 12.2 | 3.3 | 9.40 | 9.53 |
|
||||
| 1024 | 96 | 5 | 10.4 | 10.5 | 11.9 | 1.5 | 9.68 | 9.72 |
|
||||
| 2048 | 192 | 5 | 20.6 | 21.0 | 22.5 | 1.8 | 9.75 | 9.78 |
|
||||
| 4096 | 384 | 5 | 41.5 | 41.7 | 43.5 | 2.0 | 9.71 | 9.72 |
|
||||
| 8192 | 768 | 5 | 83.7 | 84.4 | 86.2 | 2.2 | 9.62 | 9.69 |
|
||||
| 16384 | 1536 | 5 | 167.1 | 167.7 | 170.2 | 2.7 | 9.64 | 9.67 |
|
||||
| 32768 | 3072 | 5 | 320.9 | 322.1 | 425.2 | 20.5 | 10.04 | 10.09 |
|
||||
| 65536 | 6144 | 5 | **1895.1** | 2375.2 | 1586.1 | 69.6 | 3.40 | 9.68 |
|
||||
| 131072 | 12288 | 5 | **2835.1** | 8923.6 | 4362.5 | 91.4 | 4.54 | 9.67 |
|
||||
|
||||
**Three regimes** in the data:
|
||||
|
||||
1. **<= 3 GiB** — linear in size, bandwidth ≈ **9.7 GB/s steady**.
|
||||
2. **6 GiB ± a bit** — onset of variance: max bandwidth still 9.7 GB/s,
|
||||
but p50 collapses to ~3.4 GB/s. Some runs achieve full speed; others
|
||||
take 2–3 × longer.
|
||||
3. **12 GiB** — wide spread (min 1.5 s, max 10 s for the same 11.5 GiB
|
||||
transfer). This is the agentic-p99 size region.
|
||||
|
||||
The bandwidth ceiling of ~10 GB/s is well below H20's NVLink p2p
|
||||
(claimed ~900 GB/s in IB) — likely the transfer is **PCIe-staged
|
||||
through host memory** rather than NVLink direct. To confirm we would
|
||||
need `nvidia-smi topo -m` and `mooncake_transfer_engine_topology_dump`
|
||||
analysis; not done yet.
|
||||
|
||||
## Known limitations of this measurement
|
||||
|
||||
- **kv_both, not strict PD-disagg.** vLLM 0.18.1 with
|
||||
`kv_role=kv_consumer` raises `AttributeError: 'MooncakeConnectorWorker'
|
||||
object has no attribute 'bootstrap_server'` (the attribute is only
|
||||
assigned inside `if not self.is_kv_consumer`). The transfer mechanics
|
||||
are identical — same `batch_transfer_sync_write` — so the cost
|
||||
measurement is comparable. The role gate only affects which request
|
||||
types each instance *accepts*. §5.2 strict PD-disagg baseline will
|
||||
need either to fix that bug or front the pair with a role-aware proxy.
|
||||
- **Single in-flight request.** All measurements here are serial.
|
||||
Real PD-disagg will have many concurrent transfers; bandwidth
|
||||
contention is not characterized.
|
||||
- **Intra-node only.** Inter-node RDMA path will be slower; not yet
|
||||
measured.
|
||||
- **Sanity preamble events.** The raw logs include 6 events from
|
||||
earlier sanity runs in addition to the 45-event sweep. `analyze_mb2.py`
|
||||
treats them as additional samples (same sizes); the per-size
|
||||
aggregates use all of them.
|
||||
|
||||
## Implications for §3.2 PD-disagg argument
|
||||
|
||||
For each PD-disagg-routed request, transfer wall-time is:
|
||||
|
||||
```
|
||||
T_transfer(KV_size) ≈ KV_size / 9.7 GB/s for KV_size ≤ 3 GiB
|
||||
≈ 0.3 – 10 s for KV_size in [3, 12] GiB
|
||||
```
|
||||
|
||||
This is the **per-request transfer charge** of PD-disagg. It's a
|
||||
real cost, but in the context of phase-isolation accounting it is
|
||||
*small* compared to the benefit:
|
||||
|
||||
| Prefill | T_prefill (MB1) | T_transfer (MB2) | Phase-isolation benefit at D=8 = D × T_prefill |
|
||||
|---:|---:|---:|---:|
|
||||
| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s |
|
||||
| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s |
|
||||
| 125k tok (~p99) | 57 s | 1.9 s | 456 s |
|
||||
|
||||
On the phase-isolation axis alone, PD-disagg recovers two orders of
|
||||
magnitude more decode time than it pays in transfer. **It is NOT this
|
||||
axis that defeats static PD-disagg in agentic** — see colleague's
|
||||
4P+4D experiment (TTFT p50 62×, success rate 99.5% → 52%) which is
|
||||
driven by **D-side KV-pool overflow** on long-context requests
|
||||
(`figs/f4b_pdsep_kv_wall.png`), not by transfer latency.
|
||||
|
||||
What MB2 contributes to the paper is therefore:
|
||||
- The **per-request transfer cost number** (used as the cost input
|
||||
to the cost-benefit accounting above).
|
||||
- The empirical observation that **Mooncake's transfer cost is
|
||||
topology-independent** — intra-node and inter-node both go through
|
||||
the RDMA NIC and hit the same 9.7 GB/s ceiling. PD-disagg's
|
||||
transfer cost does not get cheaper by co-locating P and D.
|
||||
|
||||
The dominant §3.2 failure mode of static PD-disagg in agentic is
|
||||
**capacity**, not transfer cost. MB3 / MB4 / MB5 will quantify the
|
||||
remaining axes (D-pool occupancy, cache reuse degradation under PD
|
||||
routing, static-partition mismatch).
|
||||
|
||||
## Open questions / next runs
|
||||
|
||||
- **Inter-node RDMA**: dash1 ↔ dash2. Expected lower bandwidth (~5–15
|
||||
GB/s); want to see if the 6 GiB-onset variance moves.
|
||||
- **Bandwidth ceiling investigation**: is the 9.7 GB/s ceiling PCIe (so
|
||||
the connector is not using NVLink direct) or some internal limit? If
|
||||
PCIe, can it be lifted with NVLink-direct mooncake config?
|
||||
- **Variance at 6+ GiB**: investigate. Maybe related to chunking
|
||||
inside `batch_transfer_sync_write`, or GPU memory pressure when KV
|
||||
approaches HBM ceiling.
|
||||
- **Concurrent transfers**: measure aggregate bandwidth when N
|
||||
simultaneous transfers happen. PD-disagg in practice does this.
|
||||
- **Strict kv_producer/kv_consumer**: patch the bootstrap_server bug
|
||||
or use a proxy; verify transfer time is unchanged.
|
||||
|
||||
## Reproduction
|
||||
|
||||
```bash
|
||||
# On dash machine with cpfs mount + ssh access:
|
||||
bash microbench/fresh_setup/install.sh # once (idempotent)
|
||||
bash microbench/fresh_setup/deploy.sh dash1 # push scripts to cpfs
|
||||
|
||||
# bring up pair (intra-node)
|
||||
ssh dash1 'GPU_A=0 GPU_B=1 bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/start_vllm_pair.sh start'
|
||||
|
||||
# sweep
|
||||
ssh dash1 'source /home/admin/cpfs/wjh/agentic-kv-fresh/.venv/bin/activate && \
|
||||
python /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb2_kv_transfer.py \
|
||||
--sizes 512,1024,2048,4096,8192,16384,32768,65536,131072 \
|
||||
--repeats 5 --label intra-kvboth \
|
||||
--out /home/admin/cpfs/wjh/agentic-kv-fresh/mb2_results/intra_kvboth.json'
|
||||
|
||||
# pull logs
|
||||
scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb2_transfer_logs/A/.efc_*_mb2_transfer_pid*.jsonl \
|
||||
analysis/mb2/A_intra_kvboth.jsonl
|
||||
scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb2_transfer_logs/B/.efc_*_mb2_transfer_pid*.jsonl \
|
||||
analysis/mb2/B_intra_kvboth.jsonl
|
||||
|
||||
# analyze
|
||||
.venv/bin/python microbench/fresh_setup/analyze_mb2.py \
|
||||
--a-log analysis/mb2/A_intra_kvboth.jsonl \
|
||||
--b-log analysis/mb2/B_intra_kvboth.jsonl \
|
||||
--out analysis/mb2/intra_kvboth_breakdown.json
|
||||
|
||||
.venv/bin/python microbench/fresh_setup/plot_mb2.py \
|
||||
--breakdown analysis/mb2/intra_kvboth_breakdown.json \
|
||||
--out-time figs/mb2_transfer_time_intra.png \
|
||||
--out-bw figs/mb2_transfer_bw_intra.png
|
||||
|
||||
# tear down
|
||||
ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/start_vllm_pair.sh stop'
|
||||
```
|
||||
|
||||
## Run log
|
||||
|
||||
### 2026-05-27 — intra-node, kv_both, dash1 GPU 0+1
|
||||
|
||||
Sweep: `512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072` tokens
|
||||
× 5 repeats. Sanity preamble of `512, 2048, 8192` × 2 included in the
|
||||
raw logs (counted as additional samples for those sizes).
|
||||
|
||||
Result table above. **9.7 GB/s steady-state up to 3 GiB**, variance
|
||||
opens at 6 GiB, p99 agentic-tail transfer 1.5 – 10 s.
|
||||
|
||||
Committed as `de164e5`.
|
||||
|
||||
### 2026-05-27 — inter-node, kv_both, dash1 GPU 0 → dash2 GPU 0
|
||||
|
||||
Same sweep config. 200 Gbps RoCE between hosts (RTT ~0.2 ms ping).
|
||||
Producer A on dash1 GPU 0, consumer B on dash2 GPU 0.
|
||||
remote_bootstrap_addr=`http://172.27.123.142:8998` (dash1's internal IP).
|
||||
|
||||
Raw events: `A_inter_kvboth.jsonl` (45 send_blocks + 6 sanity).
|
||||
B's receive_kv events are **missing** for this run — the
|
||||
`MB2_LOG_DIR` env var did not propagate from the start-script through
|
||||
vLLM's EngineCore subprocess on dash2 (visible via
|
||||
`cat /proc/$ENGINE_PID/environ` shows empty for dash2 but contains
|
||||
MB2_LOG_DIR for dash1 — bookmark for future investigation, likely
|
||||
spawn-vs-fork difference in vLLM's multiproc executor across hosts).
|
||||
Pure-transfer numbers below come from A's send_blocks alone; full
|
||||
rx_total breakdown not available for this run.
|
||||
|
||||
Per-size pure-transfer (analyzed by `analyze_mb2_send_only.py`):
|
||||
|
||||
| input_tokens | KV (MiB) | n | pure_ms p50 | min | max | BW p50 (GB/s) | BW max |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 512 | 48 | 5 | 5.2 | 5.1 | 65.8 | 9.76 | 9.81 |
|
||||
| 1024 | 96 | 5 | 10.2 | 10.1 | 10.4 | 9.91 | 10.00 |
|
||||
| 2048 | 192 | 5 | 20.0 | 20.0 | 20.5 | 10.06 | 10.07 |
|
||||
| 4096 | 384 | 5 | 40.1 | 40.1 | 40.5 | 10.04 | 10.05 |
|
||||
| 8192 | 768 | 5 | 80.9 | 80.7 | 82.5 | 9.96 | 9.98 |
|
||||
| 16384 | 1536 | 5 | 161.8 | 161.7 | 164.8 | 9.96 | 9.96 |
|
||||
| 32768 | 3072 | 5 | 309.6 | 307.7 | 526.9 | 10.40 | 10.47 |
|
||||
| 65536 | 6144 | 5 | 1733.6 | 653.5 | 1921.2 | 3.72 | 9.86 |
|
||||
| 131072 | 12288 | 5 | 2818.4 | 1283.0 | 9158.6 | 4.57 | 10.04 |
|
||||
|
||||
Side-by-side comparison with the 2026-05-27 intra-node run:
|
||||
|
||||
| Size | intra p50 ms | inter p50 ms | gap | intra GB/s | inter GB/s |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| 512 | 5.3 | 5.2 | −2% | 9.40 | 9.76 |
|
||||
| 1024 | 10.4 | 10.2 | −2% | 9.68 | 9.91 |
|
||||
| 2048 | 20.6 | 20.0 | −3% | 9.75 | 10.06 |
|
||||
| 4096 | 41.5 | 40.1 | −3% | 9.71 | 10.04 |
|
||||
| 8192 | 83.7 | 80.9 | −3% | 9.62 | 9.96 |
|
||||
| 16384 | 167.1 | 161.8 | −3% | 9.64 | 9.96 |
|
||||
| 32768 | 320.9 | 309.6 | −3% | 10.04 | 10.40 |
|
||||
| 65536 | 1895.1 | 1733.6 | −9% | 3.40 | 3.72 |
|
||||
|131072 | 2835.1 | 2818.4 | −1% | 4.54 | 4.57 |
|
||||
|
||||
The two paths produce essentially the same numbers — **mooncake intra-
|
||||
node is not using NVLink**, it's going through RDMA-loopback on the
|
||||
local NIC and gets the same ~10 GB/s ceiling as cross-node RDMA. The
|
||||
6+ GiB variance regime is also identical between paths.
|
||||
|
||||
Figures: `figs/mb2_transfer_time_inter.png`, `figs/mb2_transfer_bw_inter.png`,
|
||||
`figs/mb2_transfer_time_compare.png` (overlay), `figs/mb2_transfer_bw_compare.png`.
|
||||
|
||||
This collapses the §3.2 narrative to a single number: **PD-disagg
|
||||
across this cluster costs ~9.7–10 GB/s of transfer bandwidth no matter
|
||||
how you place P and D** (within-node or across-node). For p99 agentic
|
||||
KV (11.5 GiB), that's 1.3–10 s of transfer; for 6 GiB it's 0.7–2 s.
|
||||
Decode is 50–200 ms. So PD-disagg's cost dominates regardless of layout.
|
||||
@@ -1,112 +0,0 @@
|
||||
{
|
||||
"summary": [
|
||||
{
|
||||
"input_tokens": 512,
|
||||
"kv_mib": 48.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 17.29,
|
||||
"pure_transfer_ms_p50": 5.16,
|
||||
"pure_transfer_ms_min": 5.13,
|
||||
"pure_transfer_ms_max": 65.8,
|
||||
"throughput_gbps_mean": 7.95,
|
||||
"throughput_gbps_p50": 9.76,
|
||||
"throughput_gbps_max": 9.81
|
||||
},
|
||||
{
|
||||
"input_tokens": 1024,
|
||||
"kv_mib": 96.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 10.18,
|
||||
"pure_transfer_ms_p50": 10.15,
|
||||
"pure_transfer_ms_min": 10.06,
|
||||
"pure_transfer_ms_max": 10.37,
|
||||
"throughput_gbps_mean": 9.89,
|
||||
"throughput_gbps_p50": 9.91,
|
||||
"throughput_gbps_max": 10.0
|
||||
},
|
||||
{
|
||||
"input_tokens": 2048,
|
||||
"kv_mib": 192.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 20.14,
|
||||
"pure_transfer_ms_p50": 20.01,
|
||||
"pure_transfer_ms_min": 19.99,
|
||||
"pure_transfer_ms_max": 20.51,
|
||||
"throughput_gbps_mean": 10.0,
|
||||
"throughput_gbps_p50": 10.06,
|
||||
"throughput_gbps_max": 10.07
|
||||
},
|
||||
{
|
||||
"input_tokens": 4096,
|
||||
"kv_mib": 384.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 40.19,
|
||||
"pure_transfer_ms_p50": 40.12,
|
||||
"pure_transfer_ms_min": 40.07,
|
||||
"pure_transfer_ms_max": 40.5,
|
||||
"throughput_gbps_mean": 10.02,
|
||||
"throughput_gbps_p50": 10.04,
|
||||
"throughput_gbps_max": 10.05
|
||||
},
|
||||
{
|
||||
"input_tokens": 8192,
|
||||
"kv_mib": 768.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 81.32,
|
||||
"pure_transfer_ms_p50": 80.86,
|
||||
"pure_transfer_ms_min": 80.71,
|
||||
"pure_transfer_ms_max": 82.5,
|
||||
"throughput_gbps_mean": 9.9,
|
||||
"throughput_gbps_p50": 9.96,
|
||||
"throughput_gbps_max": 9.98
|
||||
},
|
||||
{
|
||||
"input_tokens": 16384,
|
||||
"kv_mib": 1536.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 162.76,
|
||||
"pure_transfer_ms_p50": 161.78,
|
||||
"pure_transfer_ms_min": 161.66,
|
||||
"pure_transfer_ms_max": 164.76,
|
||||
"throughput_gbps_mean": 9.9,
|
||||
"throughput_gbps_p50": 9.96,
|
||||
"throughput_gbps_max": 9.96
|
||||
},
|
||||
{
|
||||
"input_tokens": 32768,
|
||||
"kv_mib": 3072.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 395.6,
|
||||
"pure_transfer_ms_p50": 309.61,
|
||||
"pure_transfer_ms_min": 307.71,
|
||||
"pure_transfer_ms_max": 526.86,
|
||||
"throughput_gbps_mean": 8.71,
|
||||
"throughput_gbps_p50": 10.4,
|
||||
"throughput_gbps_max": 10.47
|
||||
},
|
||||
{
|
||||
"input_tokens": 65536,
|
||||
"kv_mib": 6144.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 1427.92,
|
||||
"pure_transfer_ms_p50": 1733.57,
|
||||
"pure_transfer_ms_min": 653.52,
|
||||
"pure_transfer_ms_max": 1921.19,
|
||||
"throughput_gbps_mean": 5.45,
|
||||
"throughput_gbps_p50": 3.72,
|
||||
"throughput_gbps_max": 9.86
|
||||
},
|
||||
{
|
||||
"input_tokens": 131072,
|
||||
"kv_mib": 12288.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 3953.36,
|
||||
"pure_transfer_ms_p50": 2818.43,
|
||||
"pure_transfer_ms_min": 1283.03,
|
||||
"pure_transfer_ms_max": 9158.57,
|
||||
"throughput_gbps_mean": 5.49,
|
||||
"throughput_gbps_p50": 4.57,
|
||||
"throughput_gbps_max": 10.04
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,679 +0,0 @@
|
||||
{
|
||||
"model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct",
|
||||
"kv_bytes_per_token": 98304,
|
||||
"src_host": "172.27.123.142",
|
||||
"src_port": 8000,
|
||||
"dst_host": "172.27.123.133",
|
||||
"dst_port": 8001,
|
||||
"config_label": "inter-kvboth",
|
||||
"raw": [
|
||||
{
|
||||
"input_tokens": 512,
|
||||
"session": "677ca02530774bae990d549d0591d336",
|
||||
"t_step1_client_unix": 1779885642.2549865,
|
||||
"t_step2_client_unix": 1779885642.3058398,
|
||||
"t_step2_end_unix": 1779885642.3249059,
|
||||
"t_prefill_s": 0.05082751903682947,
|
||||
"t_transfer_s": 0.019030610972549766,
|
||||
"t_followup_s": 0.01642513304250315,
|
||||
"cached_followup": 496,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 512,
|
||||
"session": "6758b8079675495a9a13a73db1199a71",
|
||||
"t_step1_client_unix": 1779885642.3416288,
|
||||
"t_step2_client_unix": 1779885642.3896646,
|
||||
"t_step2_end_unix": 1779885642.40659,
|
||||
"t_prefill_s": 0.04800783301470801,
|
||||
"t_transfer_s": 0.016899197013117373,
|
||||
"t_followup_s": 0.017115428985562176,
|
||||
"cached_followup": 496,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 512,
|
||||
"session": "e9da485a88cc40eb927e18c37298d815",
|
||||
"t_step1_client_unix": 1779885642.4239926,
|
||||
"t_step2_client_unix": 1779885642.4724562,
|
||||
"t_step2_end_unix": 1779885642.492528,
|
||||
"t_prefill_s": 0.04843967000488192,
|
||||
"t_transfer_s": 0.02004897501319647,
|
||||
"t_followup_s": 0.016289777995552868,
|
||||
"cached_followup": 496,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 512,
|
||||
"session": "a70db16a01b04409b04ef30e94082720",
|
||||
"t_step1_client_unix": 1779885642.509085,
|
||||
"t_step2_client_unix": 1779885642.5566719,
|
||||
"t_step2_end_unix": 1779885642.577205,
|
||||
"t_prefill_s": 0.047563806001562625,
|
||||
"t_transfer_s": 0.020512140006758273,
|
||||
"t_followup_s": 0.016482150997035205,
|
||||
"cached_followup": 496,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 512,
|
||||
"session": "d13cddd850904d4591b15c112ab32a5f",
|
||||
"t_step1_client_unix": 1779885642.5939884,
|
||||
"t_step2_client_unix": 1779885642.6417143,
|
||||
"t_step2_end_unix": 1779885642.6617038,
|
||||
"t_prefill_s": 0.04769411502638832,
|
||||
"t_transfer_s": 0.019964047998655587,
|
||||
"t_followup_s": 0.01518680399749428,
|
||||
"cached_followup": 496,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 1024,
|
||||
"session": "64d9b1335bd04698ab4f681fe4fe1e40",
|
||||
"t_step1_client_unix": 1779885642.6773622,
|
||||
"t_step2_client_unix": 1779885642.7512894,
|
||||
"t_step2_end_unix": 1779885642.7811818,
|
||||
"t_prefill_s": 0.07390057999873534,
|
||||
"t_transfer_s": 0.029867444012779742,
|
||||
"t_followup_s": 0.017206061049364507,
|
||||
"cached_followup": 1008,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 1024,
|
||||
"session": "0fc46cc27a844482b9eab2fef48a56d2",
|
||||
"t_step1_client_unix": 1779885642.7988534,
|
||||
"t_step2_client_unix": 1779885642.8733544,
|
||||
"t_step2_end_unix": 1779885642.9005985,
|
||||
"t_prefill_s": 0.074467666039709,
|
||||
"t_transfer_s": 0.027220223972108215,
|
||||
"t_followup_s": 0.0172900699544698,
|
||||
"cached_followup": 1008,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 1024,
|
||||
"session": "dac7cceb94c0465abce6bff73119de06",
|
||||
"t_step1_client_unix": 1779885642.9183617,
|
||||
"t_step2_client_unix": 1779885642.9924595,
|
||||
"t_step2_end_unix": 1779885643.0189831,
|
||||
"t_prefill_s": 0.07407327799592167,
|
||||
"t_transfer_s": 0.026499781000893563,
|
||||
"t_followup_s": 0.01622009096900001,
|
||||
"cached_followup": 1008,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 1024,
|
||||
"session": "4137a7ebecf148db8aa61249ef0d0100",
|
||||
"t_step1_client_unix": 1779885643.0356703,
|
||||
"t_step2_client_unix": 1779885643.109194,
|
||||
"t_step2_end_unix": 1779885643.135519,
|
||||
"t_prefill_s": 0.07349911500932649,
|
||||
"t_transfer_s": 0.02630024799145758,
|
||||
"t_followup_s": 0.01637468097032979,
|
||||
"cached_followup": 1008,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 1024,
|
||||
"session": "e28432915bf64c4891dc5157d0bebe41",
|
||||
"t_step1_client_unix": 1779885643.1523507,
|
||||
"t_step2_client_unix": 1779885643.2252986,
|
||||
"t_step2_end_unix": 1779885643.2520146,
|
||||
"t_prefill_s": 0.07292484800564125,
|
||||
"t_transfer_s": 0.026693789986893535,
|
||||
"t_followup_s": 0.016969401971437037,
|
||||
"cached_followup": 1008,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 2048,
|
||||
"session": "717a42513a2c48358ebfbb73ca07c3fd",
|
||||
"t_step1_client_unix": 1779885643.269825,
|
||||
"t_step2_client_unix": 1779885643.3983366,
|
||||
"t_step2_end_unix": 1779885643.417481,
|
||||
"t_prefill_s": 0.12848620803561062,
|
||||
"t_transfer_s": 0.019112227018922567,
|
||||
"t_followup_s": 0.017094306997023523,
|
||||
"cached_followup": 2032,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 2048,
|
||||
"session": "f2d1287f0e564a60a0d8476bb917ec9a",
|
||||
"t_step1_client_unix": 1779885643.4353812,
|
||||
"t_step2_client_unix": 1779885643.5640473,
|
||||
"t_step2_end_unix": 1779885643.5819182,
|
||||
"t_prefill_s": 0.12862860300811008,
|
||||
"t_transfer_s": 0.01783871802035719,
|
||||
"t_followup_s": 0.016443474043626338,
|
||||
"cached_followup": 2032,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 2048,
|
||||
"session": "cff5a22ae9c341d8acc455188ecc63b5",
|
||||
"t_step1_client_unix": 1779885643.5992126,
|
||||
"t_step2_client_unix": 1779885643.7292078,
|
||||
"t_step2_end_unix": 1779885643.766638,
|
||||
"t_prefill_s": 0.12995763396611437,
|
||||
"t_transfer_s": 0.037401642999611795,
|
||||
"t_followup_s": 0.018064863979816437,
|
||||
"cached_followup": 2032,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 2048,
|
||||
"session": "cc86ca16270b493fab68af1e889007af",
|
||||
"t_step1_client_unix": 1779885643.7855458,
|
||||
"t_step2_client_unix": 1779885643.9140093,
|
||||
"t_step2_end_unix": 1779885643.9510815,
|
||||
"t_prefill_s": 0.12843843002337962,
|
||||
"t_transfer_s": 0.03704743104754016,
|
||||
"t_followup_s": 0.017415895010344684,
|
||||
"cached_followup": 2032,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 2048,
|
||||
"session": "85f348460077474ab5c7f6db23005f48",
|
||||
"t_step1_client_unix": 1779885643.9692879,
|
||||
"t_step2_client_unix": 1779885644.0971973,
|
||||
"t_step2_end_unix": 1779885644.1345108,
|
||||
"t_prefill_s": 0.12788102397462353,
|
||||
"t_transfer_s": 0.037287415005266666,
|
||||
"t_followup_s": 0.01738263398874551,
|
||||
"cached_followup": 2032,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 4096,
|
||||
"session": "f715284b17af4bc29ccb689f21ac61b7",
|
||||
"t_step1_client_unix": 1779885644.1534681,
|
||||
"t_step2_client_unix": 1779885644.4121828,
|
||||
"t_step2_end_unix": 1779885644.4760094,
|
||||
"t_prefill_s": 0.25868210894986987,
|
||||
"t_transfer_s": 0.06378706597024575,
|
||||
"t_followup_s": 0.020318155991844833,
|
||||
"cached_followup": 4080,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 4096,
|
||||
"session": "b99bf61a90d54373a57af6a38d67581d",
|
||||
"t_step1_client_unix": 1779885644.497922,
|
||||
"t_step2_client_unix": 1779885644.7539463,
|
||||
"t_step2_end_unix": 1779885644.8172574,
|
||||
"t_prefill_s": 0.255985789000988,
|
||||
"t_transfer_s": 0.06327042501652613,
|
||||
"t_followup_s": 0.021031456999480724,
|
||||
"cached_followup": 4080,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 4096,
|
||||
"session": "69925f5b50f74c48800999908730d9ec",
|
||||
"t_step1_client_unix": 1779885644.839903,
|
||||
"t_step2_client_unix": 1779885645.0941288,
|
||||
"t_step2_end_unix": 1779885645.1562943,
|
||||
"t_prefill_s": 0.2541897820192389,
|
||||
"t_transfer_s": 0.06213490100344643,
|
||||
"t_followup_s": 0.021715508948545903,
|
||||
"cached_followup": 4080,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 4096,
|
||||
"session": "312743112bc84c0699a91e3110a63f03",
|
||||
"t_step1_client_unix": 1779885645.1795704,
|
||||
"t_step2_client_unix": 1779885645.4336376,
|
||||
"t_step2_end_unix": 1779885645.4955726,
|
||||
"t_prefill_s": 0.2540306959999725,
|
||||
"t_transfer_s": 0.06190510099986568,
|
||||
"t_followup_s": 0.020819986995775253,
|
||||
"cached_followup": 4080,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 4096,
|
||||
"session": "e5975bd25e1b4aa586cbfe9b68370de2",
|
||||
"t_step1_client_unix": 1779885645.5179617,
|
||||
"t_step2_client_unix": 1779885645.7724028,
|
||||
"t_step2_end_unix": 1779885645.8355522,
|
||||
"t_prefill_s": 0.2544059020001441,
|
||||
"t_transfer_s": 0.06311210000421852,
|
||||
"t_followup_s": 0.02235932502662763,
|
||||
"cached_followup": 4080,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 8192,
|
||||
"session": "f8989efa9261470e81c2891945b460a5",
|
||||
"t_step1_client_unix": 1779885645.8609917,
|
||||
"t_step2_client_unix": 1779885646.4364054,
|
||||
"t_step2_end_unix": 1779885646.4636223,
|
||||
"t_prefill_s": 0.575372110994067,
|
||||
"t_transfer_s": 0.02718952501891181,
|
||||
"t_followup_s": 0.024976509041152894,
|
||||
"cached_followup": 8176,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 8192,
|
||||
"session": "3a8632e8562c4cbcaa32b57ba23bbf68",
|
||||
"t_step1_client_unix": 1779885646.4916308,
|
||||
"t_step2_client_unix": 1779885647.0670328,
|
||||
"t_step2_end_unix": 1779885647.0934644,
|
||||
"t_prefill_s": 0.5753633370040916,
|
||||
"t_transfer_s": 0.026399013004265726,
|
||||
"t_followup_s": 0.027285586984362453,
|
||||
"cached_followup": 8176,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 8192,
|
||||
"session": "4d2cfc02defd492d89e90f652032f630",
|
||||
"t_step1_client_unix": 1779885647.1237462,
|
||||
"t_step2_client_unix": 1779885647.6985745,
|
||||
"t_step2_end_unix": 1779885647.8096595,
|
||||
"t_prefill_s": 0.574782071984373,
|
||||
"t_transfer_s": 0.11104170099133626,
|
||||
"t_followup_s": 0.02618999598780647,
|
||||
"cached_followup": 8176,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 8192,
|
||||
"session": "3485a775304947ada42d61cfd39b7520",
|
||||
"t_step1_client_unix": 1779885647.8389955,
|
||||
"t_step2_client_unix": 1779885648.4133987,
|
||||
"t_step2_end_unix": 1779885648.5235367,
|
||||
"t_prefill_s": 0.574362020008266,
|
||||
"t_transfer_s": 0.11010575300315395,
|
||||
"t_followup_s": 0.028998883964959532,
|
||||
"cached_followup": 8176,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 8192,
|
||||
"session": "6f87692fb5884094b7b141116ba739f8",
|
||||
"t_step1_client_unix": 1779885648.5556417,
|
||||
"t_step2_client_unix": 1779885649.1294918,
|
||||
"t_step2_end_unix": 1779885649.2386518,
|
||||
"t_prefill_s": 0.5738097460125573,
|
||||
"t_transfer_s": 0.1091307660099119,
|
||||
"t_followup_s": 0.025612210971303284,
|
||||
"cached_followup": 8176,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 16384,
|
||||
"session": "c4e007ebdcf244cf95a8ccb3ab59ea0f",
|
||||
"t_step1_client_unix": 1779885649.270192,
|
||||
"t_step2_client_unix": 1779885650.7972357,
|
||||
"t_step2_end_unix": 1779885651.0219116,
|
||||
"t_prefill_s": 1.5270042870542966,
|
||||
"t_transfer_s": 0.22463428904302418,
|
||||
"t_followup_s": 0.03729638300137594,
|
||||
"cached_followup": 16368,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 16384,
|
||||
"session": "6f34ab6672e14f41b7780323b30ce21e",
|
||||
"t_step1_client_unix": 1779885651.0651574,
|
||||
"t_step2_client_unix": 1779885652.5826244,
|
||||
"t_step2_end_unix": 1779885652.7871268,
|
||||
"t_prefill_s": 1.5174251759890467,
|
||||
"t_transfer_s": 0.2044643560075201,
|
||||
"t_followup_s": 0.03547715302556753,
|
||||
"cached_followup": 16368,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 16384,
|
||||
"session": "ca02bd5aa32544f2b1b09096e5f63c8b",
|
||||
"t_step1_client_unix": 1779885652.8284712,
|
||||
"t_step2_client_unix": 1779885654.3465412,
|
||||
"t_step2_end_unix": 1779885654.5522587,
|
||||
"t_prefill_s": 1.5180304029490799,
|
||||
"t_transfer_s": 0.20567574101733044,
|
||||
"t_followup_s": 0.03611186001216993,
|
||||
"cached_followup": 16368,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 16384,
|
||||
"session": "59047ab966574f72ab12693ad29b090f",
|
||||
"t_step1_client_unix": 1779885654.5942106,
|
||||
"t_step2_client_unix": 1779885656.1123874,
|
||||
"t_step2_end_unix": 1779885656.3158703,
|
||||
"t_prefill_s": 1.5181243289844133,
|
||||
"t_transfer_s": 0.20344976399792358,
|
||||
"t_followup_s": 0.035486461012624204,
|
||||
"cached_followup": 16368,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 16384,
|
||||
"session": "9a6e2228fd40414c8fd2b78bf399552c",
|
||||
"t_step1_client_unix": 1779885656.357184,
|
||||
"t_step2_client_unix": 1779885657.8739264,
|
||||
"t_step2_end_unix": 1779885658.079835,
|
||||
"t_prefill_s": 1.5166968539706431,
|
||||
"t_transfer_s": 0.20585927896900102,
|
||||
"t_followup_s": 0.03941280301660299,
|
||||
"cached_followup": 16368,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 32768,
|
||||
"session": "5cf9fb856b674c0a9237c06980287cb3",
|
||||
"t_step1_client_unix": 1779885658.1312225,
|
||||
"t_step2_client_unix": 1779885662.6780548,
|
||||
"t_step2_end_unix": 1779885663.0500648,
|
||||
"t_prefill_s": 4.54679255298106,
|
||||
"t_transfer_s": 0.3719712099991739,
|
||||
"t_followup_s": 0.05820785299874842,
|
||||
"cached_followup": 32752,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 32768,
|
||||
"session": "524f192c92834453bc48052bc6516126",
|
||||
"t_step1_client_unix": 1779885663.1199641,
|
||||
"t_step2_client_unix": 1779885667.6679316,
|
||||
"t_step2_end_unix": 1779885668.047607,
|
||||
"t_prefill_s": 4.547927976993378,
|
||||
"t_transfer_s": 0.3796397229889408,
|
||||
"t_followup_s": 0.058314190013334155,
|
||||
"cached_followup": 32752,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 32768,
|
||||
"session": "fb7aa2c99bf24febb83536c54e30a36d",
|
||||
"t_step1_client_unix": 1779885668.117419,
|
||||
"t_step2_client_unix": 1779885672.6644971,
|
||||
"t_step2_end_unix": 1779885673.0345452,
|
||||
"t_prefill_s": 4.547034470015205,
|
||||
"t_transfer_s": 0.3700092720100656,
|
||||
"t_followup_s": 0.05658108199713752,
|
||||
"cached_followup": 32752,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 32768,
|
||||
"session": "85612444c3224b65afb09d65a829b5cc",
|
||||
"t_step1_client_unix": 1779885673.1025856,
|
||||
"t_step2_client_unix": 1779885677.6487892,
|
||||
"t_step2_end_unix": 1779885678.2856307,
|
||||
"t_prefill_s": 4.546165039995685,
|
||||
"t_transfer_s": 0.6368027949938551,
|
||||
"t_followup_s": 0.05732736398931593,
|
||||
"cached_followup": 32752,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 32768,
|
||||
"session": "fe51fbacc4d544b3a500b099f852b19b",
|
||||
"t_step1_client_unix": 1779885678.3544915,
|
||||
"t_step2_client_unix": 1779885682.9011996,
|
||||
"t_step2_end_unix": 1779885683.542081,
|
||||
"t_prefill_s": 4.546669405011926,
|
||||
"t_transfer_s": 0.6408427829737775,
|
||||
"t_followup_s": 0.057976288022473454,
|
||||
"cached_followup": 32752,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 65536,
|
||||
"session": "9628b04d4d9e4a9fb26decccef84d32f",
|
||||
"t_step1_client_unix": 1779885683.6233714,
|
||||
"t_step2_client_unix": 1779885698.8039112,
|
||||
"t_step2_end_unix": 1779885700.9075115,
|
||||
"t_prefill_s": 15.18050062697148,
|
||||
"t_transfer_s": 2.103562032978516,
|
||||
"t_followup_s": 0.0921451430185698,
|
||||
"cached_followup": 65520,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 65536,
|
||||
"session": "b8bd7328d2274613b473e932db11081c",
|
||||
"t_step1_client_unix": 1779885701.0228417,
|
||||
"t_step2_client_unix": 1779885716.2263126,
|
||||
"t_step2_end_unix": 1779885718.3809934,
|
||||
"t_prefill_s": 15.200471161981113,
|
||||
"t_transfer_s": 2.1520405350020155,
|
||||
"t_followup_s": 0.11590708000585437,
|
||||
"cached_followup": 65520,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 65536,
|
||||
"session": "8ca1dfe708c945e59c194ee59b7de87d",
|
||||
"t_step1_client_unix": 1779885718.5226529,
|
||||
"t_step2_client_unix": 1779885733.706992,
|
||||
"t_step2_end_unix": 1779885735.6409419,
|
||||
"t_prefill_s": 15.184306180977728,
|
||||
"t_transfer_s": 1.9339170460007153,
|
||||
"t_followup_s": 0.09697375196265057,
|
||||
"cached_followup": 65520,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 65536,
|
||||
"session": "aa7555475c87427ab1a1d95277545655",
|
||||
"t_step1_client_unix": 1779885735.7609437,
|
||||
"t_step2_client_unix": 1779885750.9334345,
|
||||
"t_step2_end_unix": 1779885752.0260077,
|
||||
"t_prefill_s": 15.172441756993067,
|
||||
"t_transfer_s": 1.09253996796906,
|
||||
"t_followup_s": 0.08802600798662752,
|
||||
"cached_followup": 65520,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 65536,
|
||||
"session": "dece39eeadd645c7a8e563763f93c35f",
|
||||
"t_step1_client_unix": 1779885752.1370633,
|
||||
"t_step2_client_unix": 1779885767.3073487,
|
||||
"t_step2_end_unix": 1779885768.0632908,
|
||||
"t_prefill_s": 15.17024433100596,
|
||||
"t_transfer_s": 0.7559080219944008,
|
||||
"t_followup_s": 0.08695380098652095,
|
||||
"cached_followup": 65520,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 131072,
|
||||
"session": "98a339dad05a463e8ac1425413221d27",
|
||||
"t_step1_client_unix": 1779885768.196416,
|
||||
"t_step2_client_unix": 1779885822.9193504,
|
||||
"t_step2_end_unix": 1779885824.3877244,
|
||||
"t_prefill_s": 54.72287795698503,
|
||||
"t_transfer_s": 1.4683381259674206,
|
||||
"t_followup_s": 0.16380483901593834,
|
||||
"cached_followup": 131056,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 131072,
|
||||
"session": "4ce2151ad3d4493397bb77c7a62871e7",
|
||||
"t_step1_client_unix": 1779885824.5981278,
|
||||
"t_step2_client_unix": 1779885879.3268123,
|
||||
"t_step2_end_unix": 1779885884.7123976,
|
||||
"t_prefill_s": 54.72864995297277,
|
||||
"t_transfer_s": 5.38555136800278,
|
||||
"t_followup_s": 0.17249851900851354,
|
||||
"cached_followup": 131056,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 131072,
|
||||
"session": "6a214438dcf74f36bbefaaed3583c04c",
|
||||
"t_step1_client_unix": 1779885884.9482107,
|
||||
"t_step2_client_unix": 1779885939.6747935,
|
||||
"t_step2_end_unix": 1779885949.2077906,
|
||||
"t_prefill_s": 54.72654917498585,
|
||||
"t_transfer_s": 9.532963149016723,
|
||||
"t_followup_s": 0.1658564520184882,
|
||||
"cached_followup": 131056,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 131072,
|
||||
"session": "4cd33910256e43768a2f475650d7714c",
|
||||
"t_step1_client_unix": 1779885949.4215908,
|
||||
"t_step2_client_unix": 1779886004.145101,
|
||||
"t_step2_end_unix": 1779886007.2428164,
|
||||
"t_prefill_s": 54.72347703698324,
|
||||
"t_transfer_s": 3.0976800689822994,
|
||||
"t_followup_s": 0.16629640298197046,
|
||||
"cached_followup": 131056,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
},
|
||||
{
|
||||
"input_tokens": 131072,
|
||||
"session": "7e3626253b5d4fe29b3cec1e3598bb39",
|
||||
"t_step1_client_unix": 1779886007.4549844,
|
||||
"t_step2_client_unix": 1779886062.0564816,
|
||||
"t_step2_end_unix": 1779886063.7279255,
|
||||
"t_prefill_s": 54.60145856201416,
|
||||
"t_transfer_s": 1.6714086790452711,
|
||||
"t_followup_s": 0.1628041280200705,
|
||||
"cached_followup": 131056,
|
||||
"pull_completion_tokens": 1,
|
||||
"ok": true
|
||||
}
|
||||
],
|
||||
"summary": [
|
||||
{
|
||||
"input_tokens": 512,
|
||||
"n_ok": 5,
|
||||
"transfer_s_mean": 0.019290994200855494,
|
||||
"transfer_s_p50": 0.019964047998655587,
|
||||
"transfer_s_p90": 0.020512140006758273,
|
||||
"transfer_s_min": 0.016899197013117373,
|
||||
"transfer_s_max": 0.020512140006758273
|
||||
},
|
||||
{
|
||||
"input_tokens": 1024,
|
||||
"n_ok": 5,
|
||||
"transfer_s_mean": 0.027316297392826528,
|
||||
"transfer_s_p50": 0.026693789986893535,
|
||||
"transfer_s_p90": 0.029867444012779742,
|
||||
"transfer_s_min": 0.02630024799145758,
|
||||
"transfer_s_max": 0.029867444012779742
|
||||
},
|
||||
{
|
||||
"input_tokens": 2048,
|
||||
"n_ok": 5,
|
||||
"transfer_s_mean": 0.029737486818339674,
|
||||
"transfer_s_p50": 0.03704743104754016,
|
||||
"transfer_s_p90": 0.037401642999611795,
|
||||
"transfer_s_min": 0.01783871802035719,
|
||||
"transfer_s_max": 0.037401642999611795
|
||||
},
|
||||
{
|
||||
"input_tokens": 4096,
|
||||
"n_ok": 5,
|
||||
"transfer_s_mean": 0.0628419185988605,
|
||||
"transfer_s_p50": 0.06311210000421852,
|
||||
"transfer_s_p90": 0.06378706597024575,
|
||||
"transfer_s_min": 0.06190510099986568,
|
||||
"transfer_s_max": 0.06378706597024575
|
||||
},
|
||||
{
|
||||
"input_tokens": 8192,
|
||||
"n_ok": 5,
|
||||
"transfer_s_mean": 0.07677335160551593,
|
||||
"transfer_s_p50": 0.1091307660099119,
|
||||
"transfer_s_p90": 0.11104170099133626,
|
||||
"transfer_s_min": 0.026399013004265726,
|
||||
"transfer_s_max": 0.11104170099133626
|
||||
},
|
||||
{
|
||||
"input_tokens": 16384,
|
||||
"n_ok": 5,
|
||||
"transfer_s_mean": 0.20881668580695986,
|
||||
"transfer_s_p50": 0.20567574101733044,
|
||||
"transfer_s_p90": 0.22463428904302418,
|
||||
"transfer_s_min": 0.20344976399792358,
|
||||
"transfer_s_max": 0.22463428904302418
|
||||
},
|
||||
{
|
||||
"input_tokens": 32768,
|
||||
"n_ok": 5,
|
||||
"transfer_s_mean": 0.4798531565931626,
|
||||
"transfer_s_p50": 0.3796397229889408,
|
||||
"transfer_s_p90": 0.6408427829737775,
|
||||
"transfer_s_min": 0.3700092720100656,
|
||||
"transfer_s_max": 0.6408427829737775
|
||||
},
|
||||
{
|
||||
"input_tokens": 65536,
|
||||
"n_ok": 5,
|
||||
"transfer_s_mean": 1.6075935207889416,
|
||||
"transfer_s_p50": 1.9339170460007153,
|
||||
"transfer_s_p90": 2.1520405350020155,
|
||||
"transfer_s_min": 0.7559080219944008,
|
||||
"transfer_s_max": 2.1520405350020155
|
||||
},
|
||||
{
|
||||
"input_tokens": 131072,
|
||||
"n_ok": 5,
|
||||
"transfer_s_mean": 4.2311882782028984,
|
||||
"transfer_s_p50": 3.0976800689822994,
|
||||
"transfer_s_p90": 9.532963149016723,
|
||||
"transfer_s_min": 1.4683381259674206,
|
||||
"transfer_s_max": 9.532963149016723
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,758 +0,0 @@
|
||||
{
|
||||
"rows": [
|
||||
{
|
||||
"input_tokens_est": 512,
|
||||
"total_bytes": 50331648,
|
||||
"pure_transfer_s": 0.023202952987048775,
|
||||
"rx_total_s": 0.03333390498301014,
|
||||
"rx_overhead_s": 0.010130951995961368,
|
||||
"rx_t_start_unix": 1779879143.1678784,
|
||||
"send_t_start_unix": 1779879143.174031,
|
||||
"req_ids": [
|
||||
"cmpl-ad00672f263a6643-0-9479211a"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 512,
|
||||
"total_bytes": 50331648,
|
||||
"pure_transfer_s": 0.005375694017857313,
|
||||
"rx_total_s": 0.007019245007541031,
|
||||
"rx_overhead_s": 0.0016435509896837175,
|
||||
"rx_t_start_unix": 1779879143.2968972,
|
||||
"send_t_start_unix": 1779879143.2982283,
|
||||
"req_ids": [
|
||||
"cmpl-ace77e2b02f9f141-0-b3c061bc"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 2048,
|
||||
"total_bytes": 201326592,
|
||||
"pure_transfer_s": 0.021170366962905973,
|
||||
"rx_total_s": 0.02278437599306926,
|
||||
"rx_overhead_s": 0.0016140090301632881,
|
||||
"rx_t_start_unix": 1779879143.5146625,
|
||||
"send_t_start_unix": 1779879143.5159554,
|
||||
"req_ids": [
|
||||
"cmpl-a4a2366879c68ded-0-8ac4098e"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 2048,
|
||||
"total_bytes": 201326592,
|
||||
"pure_transfer_s": 0.020726953051052988,
|
||||
"rx_total_s": 0.022794076008722186,
|
||||
"rx_overhead_s": 0.0020671229576691985,
|
||||
"rx_t_start_unix": 1779879143.6958342,
|
||||
"send_t_start_unix": 1779879143.6974514,
|
||||
"req_ids": [
|
||||
"cmpl-8690cafcace0d5e2-0-b89f33d2"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 8192,
|
||||
"total_bytes": 805306368,
|
||||
"pure_transfer_s": 0.08536655298667029,
|
||||
"rx_total_s": 0.08753501297906041,
|
||||
"rx_overhead_s": 0.002168459992390126,
|
||||
"rx_t_start_unix": 1779879144.3279662,
|
||||
"send_t_start_unix": 1779879144.3294952,
|
||||
"req_ids": [
|
||||
"cmpl-b087e2ec4cfa8eb7-0-b908f425"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 8192,
|
||||
"total_bytes": 805306368,
|
||||
"pure_transfer_s": 0.08367906499188393,
|
||||
"rx_total_s": 0.0860149699728936,
|
||||
"rx_overhead_s": 0.002335904981009662,
|
||||
"rx_t_start_unix": 1779879145.040141,
|
||||
"send_t_start_unix": 1779879145.0419943,
|
||||
"req_ids": [
|
||||
"cmpl-a115d16ff5575e08-0-9fa81984"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16,
|
||||
"total_bytes": 1572864,
|
||||
"pure_transfer_s": 0.0004059679922647774,
|
||||
"rx_total_s": 0.002459956973325461,
|
||||
"rx_overhead_s": 0.0020539889810606837,
|
||||
"rx_t_start_unix": 1779879221.7062025,
|
||||
"send_t_start_unix": 1779879221.7078288,
|
||||
"req_ids": [
|
||||
"cmpl-9e585ed083951df5-0-b03f812b"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16,
|
||||
"total_bytes": 1572864,
|
||||
"pure_transfer_s": 0.000346789020113647,
|
||||
"rx_total_s": 0.0020201010047458112,
|
||||
"rx_overhead_s": 0.0016733119846321642,
|
||||
"rx_t_start_unix": 1779879221.7826598,
|
||||
"send_t_start_unix": 1779879221.7838593,
|
||||
"req_ids": [
|
||||
"cmpl-9271d403c044eadd-0-9c3c4639"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 512,
|
||||
"total_bytes": 50331648,
|
||||
"pure_transfer_s": 0.005353622022084892,
|
||||
"rx_total_s": 0.006836243963334709,
|
||||
"rx_overhead_s": 0.0014826219412498176,
|
||||
"rx_t_start_unix": 1779879221.859549,
|
||||
"send_t_start_unix": 1779879221.8607252,
|
||||
"req_ids": [
|
||||
"cmpl-82a580cefd3e2440-0-a383c3c4"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 512,
|
||||
"total_bytes": 50331648,
|
||||
"pure_transfer_s": 0.005279594974126667,
|
||||
"rx_total_s": 0.00694335694424808,
|
||||
"rx_overhead_s": 0.0016637619701214135,
|
||||
"rx_t_start_unix": 1779879221.9419758,
|
||||
"send_t_start_unix": 1779879221.9432015,
|
||||
"req_ids": [
|
||||
"cmpl-a31cb4bc9e7f63d2-0-8f48aacd"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 512,
|
||||
"total_bytes": 50331648,
|
||||
"pure_transfer_s": 0.0053006180096417665,
|
||||
"rx_total_s": 0.006697195000015199,
|
||||
"rx_overhead_s": 0.0013965769903734326,
|
||||
"rx_t_start_unix": 1779879222.0232244,
|
||||
"send_t_start_unix": 1779879222.0243337,
|
||||
"req_ids": [
|
||||
"cmpl-a9dfc1a5b425d994-0-a0930098"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 1024,
|
||||
"total_bytes": 100663296,
|
||||
"pure_transfer_s": 0.010396577999927104,
|
||||
"rx_total_s": 0.01183948403922841,
|
||||
"rx_overhead_s": 0.001442906039301306,
|
||||
"rx_t_start_unix": 1779879222.1297998,
|
||||
"send_t_start_unix": 1779879222.130936,
|
||||
"req_ids": [
|
||||
"cmpl-9712857755af2efc-0-90b2dc9b"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 1024,
|
||||
"total_bytes": 100663296,
|
||||
"pure_transfer_s": 0.010438029014039785,
|
||||
"rx_total_s": 0.01214482297655195,
|
||||
"rx_overhead_s": 0.0017067939625121653,
|
||||
"rx_t_start_unix": 1779879222.243023,
|
||||
"send_t_start_unix": 1779879222.2442062,
|
||||
"req_ids": [
|
||||
"cmpl-b4f0a10dee65acbe-0-a3c132fc"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 1024,
|
||||
"total_bytes": 100663296,
|
||||
"pure_transfer_s": 0.010436972021125257,
|
||||
"rx_total_s": 0.011961110983975232,
|
||||
"rx_overhead_s": 0.0015241389628499746,
|
||||
"rx_t_start_unix": 1779879222.3569698,
|
||||
"send_t_start_unix": 1779879222.3581295,
|
||||
"req_ids": [
|
||||
"cmpl-b4c514b80b52a3f2-0-bcd24f8e"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 1024,
|
||||
"total_bytes": 100663296,
|
||||
"pure_transfer_s": 0.010396371013484895,
|
||||
"rx_total_s": 0.011788576026447117,
|
||||
"rx_overhead_s": 0.001392205012962222,
|
||||
"rx_t_start_unix": 1779879222.4715128,
|
||||
"send_t_start_unix": 1779879222.4725878,
|
||||
"req_ids": [
|
||||
"cmpl-ac7118d8090d181c-0-8af4adf0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 1024,
|
||||
"total_bytes": 100663296,
|
||||
"pure_transfer_s": 0.010352785000577569,
|
||||
"rx_total_s": 0.0118055299972184,
|
||||
"rx_overhead_s": 0.0014527449966408312,
|
||||
"rx_t_start_unix": 1779879222.5826046,
|
||||
"send_t_start_unix": 1779879222.5837166,
|
||||
"req_ids": [
|
||||
"cmpl-85291bcb93aaf638-0-868db1a8"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16,
|
||||
"total_bytes": 1572864,
|
||||
"pure_transfer_s": 0.00034007197245955467,
|
||||
"rx_total_s": 0.0021119200391694903,
|
||||
"rx_overhead_s": 0.0017718480667099357,
|
||||
"rx_t_start_unix": 1779879222.750828,
|
||||
"send_t_start_unix": 1779879222.7521152,
|
||||
"req_ids": [
|
||||
"cmpl-a448cf2e059ba0c9-0-a1360796"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16,
|
||||
"total_bytes": 1572864,
|
||||
"pure_transfer_s": 0.00041691696969792247,
|
||||
"rx_total_s": 0.0022232600022107363,
|
||||
"rx_overhead_s": 0.0018063430325128138,
|
||||
"rx_t_start_unix": 1779879222.913044,
|
||||
"send_t_start_unix": 1779879222.9143836,
|
||||
"req_ids": [
|
||||
"cmpl-b486fd9e945a4658-0-8bb561cd"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 2048,
|
||||
"total_bytes": 201326592,
|
||||
"pure_transfer_s": 0.020633380976505578,
|
||||
"rx_total_s": 0.022250515001360327,
|
||||
"rx_overhead_s": 0.0016171340248547494,
|
||||
"rx_t_start_unix": 1779879223.0765986,
|
||||
"send_t_start_unix": 1779879223.0778644,
|
||||
"req_ids": [
|
||||
"cmpl-82da2bfe65f276c6-0-88d9a9a2"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 2048,
|
||||
"total_bytes": 201326592,
|
||||
"pure_transfer_s": 0.020639199996367097,
|
||||
"rx_total_s": 0.022157608007546514,
|
||||
"rx_overhead_s": 0.0015184080111794174,
|
||||
"rx_t_start_unix": 1779879223.2591784,
|
||||
"send_t_start_unix": 1779879223.2603853,
|
||||
"req_ids": [
|
||||
"cmpl-93bd777652eba5f3-0-9ec3d058"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 2048,
|
||||
"total_bytes": 201326592,
|
||||
"pure_transfer_s": 0.020575353992171586,
|
||||
"rx_total_s": 0.022589912987314165,
|
||||
"rx_overhead_s": 0.002014558995142579,
|
||||
"rx_t_start_unix": 1779879223.4402068,
|
||||
"send_t_start_unix": 1779879223.4418828,
|
||||
"req_ids": [
|
||||
"cmpl-81f950480a3cabf9-0-bbf8584f"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 4096,
|
||||
"total_bytes": 402653184,
|
||||
"pure_transfer_s": 0.041439525957684964,
|
||||
"rx_total_s": 0.043345845013391227,
|
||||
"rx_overhead_s": 0.0019063190557062626,
|
||||
"rx_t_start_unix": 1779879223.7529812,
|
||||
"send_t_start_unix": 1779879223.7544343,
|
||||
"req_ids": [
|
||||
"cmpl-b109ed06b5882659-0-8d14993c"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 4096,
|
||||
"total_bytes": 402653184,
|
||||
"pure_transfer_s": 0.04152030003024265,
|
||||
"rx_total_s": 0.04341953102266416,
|
||||
"rx_overhead_s": 0.0018992309924215078,
|
||||
"rx_t_start_unix": 1779879224.0899644,
|
||||
"send_t_start_unix": 1779879224.0914912,
|
||||
"req_ids": [
|
||||
"cmpl-8a57776c81d64b2c-0-ace8fb2b"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 4096,
|
||||
"total_bytes": 402653184,
|
||||
"pure_transfer_s": 0.04148670402355492,
|
||||
"rx_total_s": 0.04336977802449837,
|
||||
"rx_overhead_s": 0.0018830740009434521,
|
||||
"rx_t_start_unix": 1779879224.424807,
|
||||
"send_t_start_unix": 1779879224.4262393,
|
||||
"req_ids": [
|
||||
"cmpl-9b1a5dce18758450-0-b17b3649"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 4096,
|
||||
"total_bytes": 402653184,
|
||||
"pure_transfer_s": 0.04146742797456682,
|
||||
"rx_total_s": 0.043769759009592235,
|
||||
"rx_overhead_s": 0.002302331035025418,
|
||||
"rx_t_start_unix": 1779879224.7599711,
|
||||
"send_t_start_unix": 1779879224.7617002,
|
||||
"req_ids": [
|
||||
"cmpl-8c7d412b85f43ed7-0-9dea4add"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 4096,
|
||||
"total_bytes": 402653184,
|
||||
"pure_transfer_s": 0.04143296502297744,
|
||||
"rx_total_s": 0.043612666020635515,
|
||||
"rx_overhead_s": 0.002179700997658074,
|
||||
"rx_t_start_unix": 1779879225.0962389,
|
||||
"send_t_start_unix": 1779879225.0978234,
|
||||
"req_ids": [
|
||||
"cmpl-8860308db3f010a5-0-ad51eb46"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16,
|
||||
"total_bytes": 1572864,
|
||||
"pure_transfer_s": 0.0003991159610450268,
|
||||
"rx_total_s": 0.002386144013144076,
|
||||
"rx_overhead_s": 0.001987028052099049,
|
||||
"rx_t_start_unix": 1779879225.7592747,
|
||||
"send_t_start_unix": 1779879225.760789,
|
||||
"req_ids": [
|
||||
"cmpl-86cca1a2b9427801-0-ba41ade7"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16,
|
||||
"total_bytes": 1572864,
|
||||
"pure_transfer_s": 0.00041423802031204104,
|
||||
"rx_total_s": 0.0023903060355223715,
|
||||
"rx_overhead_s": 0.0019760680152103305,
|
||||
"rx_t_start_unix": 1779879226.384918,
|
||||
"send_t_start_unix": 1779879226.3864496,
|
||||
"req_ids": [
|
||||
"cmpl-a208c6d804293be7-0-94d265ab"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 8192,
|
||||
"total_bytes": 805306368,
|
||||
"pure_transfer_s": 0.08309489500243217,
|
||||
"rx_total_s": 0.08524628396844491,
|
||||
"rx_overhead_s": 0.002151388966012746,
|
||||
"rx_t_start_unix": 1779879227.0092332,
|
||||
"send_t_start_unix": 1779879227.0107942,
|
||||
"req_ids": [
|
||||
"cmpl-b53bea2317cc1211-0-8fcad8a8"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 8192,
|
||||
"total_bytes": 805306368,
|
||||
"pure_transfer_s": 0.08372796402545646,
|
||||
"rx_total_s": 0.08596085698809475,
|
||||
"rx_overhead_s": 0.0022328929626382887,
|
||||
"rx_t_start_unix": 1779879227.7190688,
|
||||
"send_t_start_unix": 1779879227.7207224,
|
||||
"req_ids": [
|
||||
"cmpl-9daf909593bbdf03-0-8fd7d50e"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 8192,
|
||||
"total_bytes": 805306368,
|
||||
"pure_transfer_s": 0.08398396399570629,
|
||||
"rx_total_s": 0.0860762019874528,
|
||||
"rx_overhead_s": 0.002092237991746515,
|
||||
"rx_t_start_unix": 1779879228.4297745,
|
||||
"send_t_start_unix": 1779879228.4314566,
|
||||
"req_ids": [
|
||||
"cmpl-9ef40f3b6d736128-0-8e8e1c30"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16384,
|
||||
"total_bytes": 1610612736,
|
||||
"pure_transfer_s": 0.16950496198842302,
|
||||
"rx_total_s": 0.1721468890318647,
|
||||
"rx_overhead_s": 0.002641927043441683,
|
||||
"rx_t_start_unix": 1779879230.131392,
|
||||
"send_t_start_unix": 1779879230.1334376,
|
||||
"req_ids": [
|
||||
"cmpl-851e5d7e3e83d7ea-0-a66a5e0b"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16384,
|
||||
"total_bytes": 1610612736,
|
||||
"pure_transfer_s": 0.16713789198547602,
|
||||
"rx_total_s": 0.16974544001277536,
|
||||
"rx_overhead_s": 0.0026075480272993445,
|
||||
"rx_t_start_unix": 1779879231.896075,
|
||||
"send_t_start_unix": 1779879231.8981037,
|
||||
"req_ids": [
|
||||
"cmpl-9be12af6a9ccccf5-0-af1230c7"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16384,
|
||||
"total_bytes": 1610612736,
|
||||
"pure_transfer_s": 0.16713115200400352,
|
||||
"rx_total_s": 0.16975757898762822,
|
||||
"rx_overhead_s": 0.0026264269836246967,
|
||||
"rx_t_start_unix": 1779879233.6589305,
|
||||
"send_t_start_unix": 1779879233.6608078,
|
||||
"req_ids": [
|
||||
"cmpl-b61b9b237366297b-0-9832f0e3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16384,
|
||||
"total_bytes": 1610612736,
|
||||
"pure_transfer_s": 0.16709016199456528,
|
||||
"rx_total_s": 0.1695251659839414,
|
||||
"rx_overhead_s": 0.0024350039893761277,
|
||||
"rx_t_start_unix": 1779879235.4181106,
|
||||
"send_t_start_unix": 1779879235.419875,
|
||||
"req_ids": [
|
||||
"cmpl-bae0d0efe47ece8f-0-affbc685"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 16384,
|
||||
"total_bytes": 1610612736,
|
||||
"pure_transfer_s": 0.166486973001156,
|
||||
"rx_total_s": 0.16962904302636161,
|
||||
"rx_overhead_s": 0.003142070025205612,
|
||||
"rx_t_start_unix": 1779879237.1803744,
|
||||
"send_t_start_unix": 1779879237.1821773,
|
||||
"req_ids": [
|
||||
"cmpl-a34bc73c9cd2efc1-0-90d647fc"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 32768,
|
||||
"total_bytes": 3221225472,
|
||||
"pure_transfer_s": 0.31926770601421595,
|
||||
"rx_total_s": 0.32203804596792907,
|
||||
"rx_overhead_s": 0.002770339953713119,
|
||||
"rx_t_start_unix": 1779879241.9859307,
|
||||
"send_t_start_unix": 1779879241.9880297,
|
||||
"req_ids": [
|
||||
"cmpl-89a36c12ee6b0ff3-0-9fddbc0f"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 32768,
|
||||
"total_bytes": 3221225472,
|
||||
"pure_transfer_s": 0.3197040680097416,
|
||||
"rx_total_s": 0.3227974839974195,
|
||||
"rx_overhead_s": 0.003093415987677872,
|
||||
"rx_t_start_unix": 1779879246.9755645,
|
||||
"send_t_start_unix": 1779879246.9779432,
|
||||
"req_ids": [
|
||||
"cmpl-8d65512eb7e3c36c-0-8b23597c"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 32768,
|
||||
"total_bytes": 3221225472,
|
||||
"pure_transfer_s": 0.32088329299585894,
|
||||
"rx_total_s": 0.3240378479822539,
|
||||
"rx_overhead_s": 0.003154554986394942,
|
||||
"rx_t_start_unix": 1779879251.9618897,
|
||||
"send_t_start_unix": 1779879251.9643052,
|
||||
"req_ids": [
|
||||
"cmpl-a13c271ecbbca78b-0-b76a0370"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 32768,
|
||||
"total_bytes": 3221225472,
|
||||
"pure_transfer_s": 0.5439103110111319,
|
||||
"rx_total_s": 0.5924434679909609,
|
||||
"rx_overhead_s": 0.04853315697982907,
|
||||
"rx_t_start_unix": 1779879256.9512377,
|
||||
"send_t_start_unix": 1779879256.9989722,
|
||||
"req_ids": [
|
||||
"cmpl-bada04ec8c556aca-0-a263d637"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 32768,
|
||||
"total_bytes": 3221225472,
|
||||
"pure_transfer_s": 0.5193864739849232,
|
||||
"rx_total_s": 0.5644763479940593,
|
||||
"rx_overhead_s": 0.04508987400913611,
|
||||
"rx_t_start_unix": 1779879262.2127163,
|
||||
"send_t_start_unix": 1779879262.2562187,
|
||||
"req_ids": [
|
||||
"cmpl-9641a077022e6123-0-8c3c0975"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 65536,
|
||||
"total_bytes": 6442450944,
|
||||
"pure_transfer_s": 1.9844180009677075,
|
||||
"rx_total_s": 2.0784930550144054,
|
||||
"rx_overhead_s": 0.09407505404669791,
|
||||
"rx_t_start_unix": 1779879278.1063075,
|
||||
"send_t_start_unix": 1779879278.199048,
|
||||
"req_ids": [
|
||||
"cmpl-bb3a4e5084af8c3a-0-bdfa0931"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 65536,
|
||||
"total_bytes": 6442450944,
|
||||
"pure_transfer_s": 2.1099297259934247,
|
||||
"rx_total_s": 2.2067435560165904,
|
||||
"rx_overhead_s": 0.09681383002316579,
|
||||
"rx_t_start_unix": 1779879295.600993,
|
||||
"send_t_start_unix": 1779879295.6967168,
|
||||
"req_ids": [
|
||||
"cmpl-91b951f85c93a71b-0-8396bee5"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 65536,
|
||||
"total_bytes": 6442450944,
|
||||
"pure_transfer_s": 1.8950715209939517,
|
||||
"rx_total_s": 1.9879729640088044,
|
||||
"rx_overhead_s": 0.0929014430148527,
|
||||
"rx_t_start_unix": 1779879313.2315958,
|
||||
"send_t_start_unix": 1779879313.3236735,
|
||||
"req_ids": [
|
||||
"cmpl-81d236ecb6aadadf-0-ac184d51"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 65536,
|
||||
"total_bytes": 6442450944,
|
||||
"pure_transfer_s": 0.9277855920372531,
|
||||
"rx_total_s": 0.9849357060156763,
|
||||
"rx_overhead_s": 0.05715011397842318,
|
||||
"rx_t_start_unix": 1779879330.6154163,
|
||||
"send_t_start_unix": 1779879330.6715357,
|
||||
"req_ids": [
|
||||
"cmpl-a4c76c62b44c4295-0-b007a6ed"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 65536,
|
||||
"total_bytes": 6442450944,
|
||||
"pure_transfer_s": 0.6652462020283565,
|
||||
"rx_total_s": 0.6725030990201049,
|
||||
"rx_overhead_s": 0.007256896991748363,
|
||||
"rx_t_start_unix": 1779879346.990221,
|
||||
"send_t_start_unix": 1779879346.9950044,
|
||||
"req_ids": [
|
||||
"cmpl-a06d4b774a8af9a5-0-980e9d23"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 131072,
|
||||
"total_bytes": 12884901888,
|
||||
"pure_transfer_s": 1.3330365709844045,
|
||||
"rx_total_s": 1.3384539679973386,
|
||||
"rx_overhead_s": 0.005417397012934089,
|
||||
"rx_t_start_unix": 1779879402.7123013,
|
||||
"send_t_start_unix": 1779879402.7169023,
|
||||
"req_ids": [
|
||||
"cmpl-bf0d435e06e3349f-0-8507c933"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 131072,
|
||||
"total_bytes": 12884901888,
|
||||
"pure_transfer_s": 5.839069904992357,
|
||||
"rx_total_s": 5.973284716019407,
|
||||
"rx_overhead_s": 0.13421481102705002,
|
||||
"rx_t_start_unix": 1779879458.9232886,
|
||||
"send_t_start_unix": 1779879459.0566247,
|
||||
"req_ids": [
|
||||
"cmpl-9f87ae0fb0c7eec8-0-a8a1daea"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 131072,
|
||||
"total_bytes": 12884901888,
|
||||
"pure_transfer_s": 9.862486142024864,
|
||||
"rx_total_s": 10.056511385017075,
|
||||
"rx_overhead_s": 0.19402524299221113,
|
||||
"rx_t_start_unix": 1779879519.7647448,
|
||||
"send_t_start_unix": 1779879519.9567635,
|
||||
"req_ids": [
|
||||
"cmpl-a62e48e40e6c6ad7-0-acca9741"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 131072,
|
||||
"total_bytes": 12884901888,
|
||||
"pure_transfer_s": 2.8350498770014383,
|
||||
"rx_total_s": 2.925714804965537,
|
||||
"rx_overhead_s": 0.09066492796409875,
|
||||
"rx_t_start_unix": 1779879584.888362,
|
||||
"send_t_start_unix": 1779879584.9780834,
|
||||
"req_ids": [
|
||||
"cmpl-824479d53bab40e4-0-af951a11"
|
||||
]
|
||||
},
|
||||
{
|
||||
"input_tokens_est": 131072,
|
||||
"total_bytes": 12884901888,
|
||||
"pure_transfer_s": 1.485496642999351,
|
||||
"rx_total_s": 1.5183607729850337,
|
||||
"rx_overhead_s": 0.032864129985682666,
|
||||
"rx_t_start_unix": 1779879642.6076336,
|
||||
"send_t_start_unix": 1779879642.639775,
|
||||
"req_ids": [
|
||||
"cmpl-9f06f19c981c0b3f-0-b3afb370"
|
||||
]
|
||||
}
|
||||
],
|
||||
"summary": [
|
||||
{
|
||||
"input_tokens": 16,
|
||||
"kv_mib": 1.5,
|
||||
"n": 6,
|
||||
"pure_transfer_ms_mean": 0.39,
|
||||
"pure_transfer_ms_p50": 0.4,
|
||||
"pure_transfer_ms_max": 0.42,
|
||||
"pure_transfer_ms_min": 0.34,
|
||||
"rx_total_ms_mean": 2.27,
|
||||
"rx_overhead_ms_mean": 1.88,
|
||||
"throughput_gbps_mean": 4.09,
|
||||
"throughput_gbps_p50": 3.91,
|
||||
"throughput_gbps_max": 4.63
|
||||
},
|
||||
{
|
||||
"input_tokens": 512,
|
||||
"kv_mib": 48.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 8.9,
|
||||
"pure_transfer_ms_p50": 5.35,
|
||||
"pure_transfer_ms_max": 23.2,
|
||||
"pure_transfer_ms_min": 5.28,
|
||||
"rx_total_ms_mean": 12.17,
|
||||
"rx_overhead_ms_mean": 3.26,
|
||||
"throughput_gbps_mean": 7.99,
|
||||
"throughput_gbps_p50": 9.4,
|
||||
"throughput_gbps_max": 9.53
|
||||
},
|
||||
{
|
||||
"input_tokens": 1024,
|
||||
"kv_mib": 96.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 10.4,
|
||||
"pure_transfer_ms_p50": 10.4,
|
||||
"pure_transfer_ms_max": 10.44,
|
||||
"pure_transfer_ms_min": 10.35,
|
||||
"rx_total_ms_mean": 11.91,
|
||||
"rx_overhead_ms_mean": 1.5,
|
||||
"throughput_gbps_mean": 9.68,
|
||||
"throughput_gbps_p50": 9.68,
|
||||
"throughput_gbps_max": 9.72
|
||||
},
|
||||
{
|
||||
"input_tokens": 2048,
|
||||
"kv_mib": 192.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 20.75,
|
||||
"pure_transfer_ms_p50": 20.64,
|
||||
"pure_transfer_ms_max": 21.17,
|
||||
"pure_transfer_ms_min": 20.58,
|
||||
"rx_total_ms_mean": 22.52,
|
||||
"rx_overhead_ms_mean": 1.77,
|
||||
"throughput_gbps_mean": 9.7,
|
||||
"throughput_gbps_p50": 9.75,
|
||||
"throughput_gbps_max": 9.78
|
||||
},
|
||||
{
|
||||
"input_tokens": 4096,
|
||||
"kv_mib": 384.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 41.47,
|
||||
"pure_transfer_ms_p50": 41.47,
|
||||
"pure_transfer_ms_max": 41.52,
|
||||
"pure_transfer_ms_min": 41.43,
|
||||
"rx_total_ms_mean": 43.5,
|
||||
"rx_overhead_ms_mean": 2.03,
|
||||
"throughput_gbps_mean": 9.71,
|
||||
"throughput_gbps_p50": 9.71,
|
||||
"throughput_gbps_max": 9.72
|
||||
},
|
||||
{
|
||||
"input_tokens": 8192,
|
||||
"kv_mib": 768.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 83.97,
|
||||
"pure_transfer_ms_p50": 83.73,
|
||||
"pure_transfer_ms_max": 85.37,
|
||||
"pure_transfer_ms_min": 83.09,
|
||||
"rx_total_ms_mean": 86.17,
|
||||
"rx_overhead_ms_mean": 2.2,
|
||||
"throughput_gbps_mean": 9.59,
|
||||
"throughput_gbps_p50": 9.62,
|
||||
"throughput_gbps_max": 9.69
|
||||
},
|
||||
{
|
||||
"input_tokens": 16384,
|
||||
"kv_mib": 1536.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 167.47,
|
||||
"pure_transfer_ms_p50": 167.13,
|
||||
"pure_transfer_ms_max": 169.5,
|
||||
"pure_transfer_ms_min": 166.49,
|
||||
"rx_total_ms_mean": 170.16,
|
||||
"rx_overhead_ms_mean": 2.69,
|
||||
"throughput_gbps_mean": 9.62,
|
||||
"throughput_gbps_p50": 9.64,
|
||||
"throughput_gbps_max": 9.67
|
||||
},
|
||||
{
|
||||
"input_tokens": 32768,
|
||||
"kv_mib": 3072.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 404.63,
|
||||
"pure_transfer_ms_p50": 320.88,
|
||||
"pure_transfer_ms_max": 543.91,
|
||||
"pure_transfer_ms_min": 319.27,
|
||||
"rx_total_ms_mean": 425.16,
|
||||
"rx_overhead_ms_mean": 20.53,
|
||||
"throughput_gbps_mean": 8.47,
|
||||
"throughput_gbps_p50": 10.04,
|
||||
"throughput_gbps_max": 10.09
|
||||
},
|
||||
{
|
||||
"input_tokens": 65536,
|
||||
"kv_mib": 6144.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 1516.49,
|
||||
"pure_transfer_ms_p50": 1895.07,
|
||||
"pure_transfer_ms_max": 2109.93,
|
||||
"pure_transfer_ms_min": 665.25,
|
||||
"rx_total_ms_mean": 1586.13,
|
||||
"rx_overhead_ms_mean": 69.64,
|
||||
"throughput_gbps_mean": 5.27,
|
||||
"throughput_gbps_p50": 3.4,
|
||||
"throughput_gbps_max": 9.68
|
||||
},
|
||||
{
|
||||
"input_tokens": 131072,
|
||||
"kv_mib": 12288.0,
|
||||
"n": 5,
|
||||
"pure_transfer_ms_mean": 4271.03,
|
||||
"pure_transfer_ms_p50": 2835.05,
|
||||
"pure_transfer_ms_max": 9862.49,
|
||||
"pure_transfer_ms_min": 1333.04,
|
||||
"rx_total_ms_mean": 4362.47,
|
||||
"rx_overhead_ms_mean": 91.44,
|
||||
"throughput_gbps_mean": 5.28,
|
||||
"throughput_gbps_p50": 4.54,
|
||||
"throughput_gbps_max": 9.67
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,107 +0,0 @@
|
||||
# PD-disagg vs colocation — controlled reuse & concurrency axes (v2)
|
||||
|
||||
Self-contained results for the **controlled-variable** redo of the MB5 PD-vs-colo
|
||||
ablation. Supersedes the confounded first cut (held input fixed and sliced the
|
||||
prefix, so "more reuse" was entangled with "less prefill"). All arms route through
|
||||
the proxy at fair **APC parity** (session-routed producers reach the same prefix-cache
|
||||
hit rate as colo), so PD loses on *structure*, not on broken cache.
|
||||
|
||||
- **Config arms:** `colo` = 8×kv_both (8C-proxy, session-affinity); PD = `6P+2D / 4P+4D / 2P+6D`.
|
||||
- **Driver:** closed-loop N (`REPLAY_MAX_INFLIGHT`) + fixed think-time; `gen_synthetic_trace.py --mode regular`.
|
||||
- **PD-arm wall-cap:** collapsed PD arms drain pathologically slowly, so PD arms run with a
|
||||
wall-deadline (`REPLAY_MAX_DURATION`; un-run turns counted as failures → honest completion%);
|
||||
**colo is uncapped** so the reference is always fully measured.
|
||||
- **Hardware:** run on **dash2** (8×H20). dash0's RDMA NICs were faulted for Mooncake during
|
||||
this work (could not init the transfer engine; needs an admin reset — no sudo); dash2's NICs
|
||||
are healthy. cpfs/venv/data are shared across the boxes.
|
||||
|
||||
---
|
||||
|
||||
## 1. Reuse / APC axis — fixed real prefill, vary cached prefix
|
||||
|
||||
N=8. Hold the **real new-prefill work per turn constant** (`--delta-len`) and grow the cached
|
||||
prefix → reuse = prefix/(prefix+delta). Three shapes isolate output vs delta:
|
||||
|
||||
| | delta (real prefill/turn) | output | role |
|
||||
|---|---|---|---|
|
||||
| **A** | 2048 | 256 | original |
|
||||
| **C** | 2048 | 128 | A vs C = pure **output** 256→128 |
|
||||
| **B** | 1024 | 128 | C vs B = pure **delta** 2048→1024 |
|
||||
|
||||
**PD-best advantage** = colo E2E-p90 / best-PD E2E-p90 (>1 ⇒ PD wins):
|
||||
|
||||
| reuse% | A d2048/o256 | C d2048/o128 | B d1024/o128 |
|
||||
|---|---|---|---|
|
||||
| 20 | 1.34 | 1.41 | — |
|
||||
| 50 | 1.36 | 1.37 | — |
|
||||
| 67 | **1.47** | **1.49** | **1.27** |
|
||||
| 80 | 1.31 | 1.23 | 1.25 |
|
||||
| 90 | 1.15 | 1.01 | — |
|
||||
| 95 | 0.87 | 0.89 | 0.89 |
|
||||
|
||||
|
||||
|
||||
**Findings:**
|
||||
1. **Output length is ~negligible.** A and C (same delta) track each other across the whole
|
||||
range — halving output barely moves PD's advantage.
|
||||
2. **Delta (real prefill/turn) is the dominant shape factor.** B (delta=1024) sits clearly
|
||||
below A/C at mid reuse (67%: 1.27 vs ~1.48). More real prefill per turn → bigger PD win,
|
||||
because PD-disagg's benefit is isolating prefill from decode — more prefill to isolate.
|
||||
3. **Crossover to colo at reuse ~90–95% is robust** across all three shapes: PD always loses
|
||||
the high-reuse / large-resident-context corner (it must KV-transfer the whole resident
|
||||
context every turn for a few hundred new tokens; colo keeps it local).
|
||||
|
||||
*Caveat:* the clean, uncapped, 100%-completion comparison region is reuse **20–80%** (carries
|
||||
findings 1–2). At reuse 90/95% the PD arms collapse and C's points are capped-completion, while
|
||||
A/B are full-drain — comparable in direction, not in exact PD completion%.
|
||||
|
||||
Data: `fig1_reuse_fixed.json` (A), `fig1_reuse_d2048_o128.json` (C), `fig1_reuse_d1024_o128.json` (B).
|
||||
|
||||
---
|
||||
|
||||
## 2. Concurrency axis — agentic corner, sweep N
|
||||
|
||||
in=32768 (prefix 32256 + delta 512, **reuse 0.984**), out=128; closed-loop N ∈ {8,16,32,48,64,96,128};
|
||||
PD arms capped 600s, colo uncapped.
|
||||
|
||||
| N | **colo** completion · E2E-mean · TPS | best PD-arm completion |
|
||||
|---|---|---|
|
||||
| 8 | **256/256** · 2.4s · 326 | 6P+2D 256/256 |
|
||||
| 16 | **512/512** · 3.5s · 462 | 6P+2D 439/512 (86%) |
|
||||
| 32 | **1024/1024** · 13.3s · 190 | all PD **<27%** |
|
||||
| 48 | **1536/1536** · 24.9s · 168 | all PD <32% |
|
||||
| 64 | **2048/2048** · 38.4s · 166 | all PD <31% |
|
||||
| 96 | **3072/3072** · 60.0s · 171 | PD **2–7%** |
|
||||
| 128 | **4096/4096** · 80.8s · 181 | 4P+4D 6%, 2P+6D <1% |
|
||||
|
||||
|
||||
|
||||
**Finding:** **colo completes 100% of requests at every concurrency level** — it degrades
|
||||
*gracefully* (latency rises 2.4s→81s, nothing dropped). **Every static PD split collapses, and
|
||||
progressively earlier as N rises**: PD is viable only at N≤8–16; by N≥32 it drops 70–99% of
|
||||
requests while its prefix-cache hit-rate craters to ~0%. colo's elastic pool absorbs the
|
||||
time-varying P/D demand; the static partition + per-turn 32k KV-transfer cannot. (Latency
|
||||
percentiles count successes only, so they *understate* PD — read them with the completion column.)
|
||||
|
||||
Data: `fig3_conc32k.json`.
|
||||
|
||||
*Caveat:* N=128 6P+2D is missing (one transient vLLM/Mooncake startup flake at the end); does
|
||||
not change the picture (all PD arms are already collapsed by N=128). The SLO auto-stop in the
|
||||
driver is a no-op (a stdout-capture bug), so the full grid ran — more points, not fewer.
|
||||
|
||||
---
|
||||
|
||||
## 3. Reproduce
|
||||
|
||||
```bash
|
||||
# on a box with healthy Mooncake/RDMA NICs (dash2), cpfs mounted:
|
||||
R=/home/admin/cpfs/wjh/agentic-kv-fresh
|
||||
# reuse axis (three shapes): DELTA/OL pick the shape; tag carries _o${OL}
|
||||
ssh dash2 "cd $R && DELTA=2048 OL=256 bash microbench/fresh_setup/run_reuse_fixed.sh"
|
||||
ssh dash2 "cd $R && DELTA=2048 OL=128 bash microbench/fresh_setup/run_reuse_fixed.sh"
|
||||
ssh dash2 "cd $R && DELTA=1024 OL=128 bash microbench/fresh_setup/run_reuse_fixed.sh"
|
||||
# concurrency axis (capped):
|
||||
ssh dash2 "cd $R && NLIST='8 16 32 48 64 96 128' CONC_PD_MAXDUR=600 bash microbench/fresh_setup/run_conc.sh"
|
||||
# render (reads the *.json in this dir):
|
||||
python microbench/fresh_setup/plot_pd_crossover.py
|
||||
```
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,147 +0,0 @@
|
||||
# Migration Trigger Validation (unified_v4) — 2026-05-30
|
||||
|
||||
Hardware: dash2, 8×H20, Qwen3-Coder-30B-A3B, TP=1, kv_both + DR-fix substrate.
|
||||
Trace: `w600_r0.0015_st30_first600s.jsonl` (807 reqs, 600s span).
|
||||
Policy: `unified_v4` = unified hybrid routing + pending-prefill-queue-triggered
|
||||
session migration (commit `3a6bf5d` on `kzlin-dev` branch).
|
||||
|
||||
## Research Question
|
||||
|
||||
Does Pillar 2 (hot-triggered session migration) provide measurable benefit on
|
||||
top of Pillar 1 (affinity-default routing)?
|
||||
|
||||
## Experiment Design
|
||||
|
||||
| Arm | Policy | Substrate | Trace QPS |
|
||||
|---|---|---|---|
|
||||
| unified_1x | unified (affinity-only) | kv_both + DR-fix | ~1.3 (original) |
|
||||
| unified_v4_1x | unified_v4 (affinity + migration) | kv_both + DR-fix | ~1.3 |
|
||||
| unified_v4_2x | unified_v4 (affinity + migration) | kv_both + DR-fix | ~2.7 (2× compressed) |
|
||||
|
||||
The 2× trace was generated by halving inter-request intervals:
|
||||
`ts_new = ts_min + (ts_orig - ts_min) / 2`.
|
||||
|
||||
## Results
|
||||
|
||||
### 1x QPS: unified vs unified_v4
|
||||
|
||||
| Metric | unified | unified_v4 | Delta |
|
||||
|---|---:|---:|---:|
|
||||
| OK/total | 807/807 | 807/807 | — |
|
||||
| TTFT mean | 3.990s | 4.142s | +3.8% |
|
||||
| TTFT p50 | 0.719s | 0.711s | −1.0% |
|
||||
| TTFT p90 | 11.499s | 12.293s | +6.9% |
|
||||
| TPOT p90 | 0.024s | 0.022s | −9.3% |
|
||||
| E2E p50 | 2.265s | 2.293s | +1.2% |
|
||||
| E2E p90 | 24.507s | 23.955s | −2.3% |
|
||||
| **Migrations** | **0** | **0** | — |
|
||||
|
||||
**Conclusion**: At 1x QPS (~1.3 req/s, ~0.16 req/instance/s), the migration
|
||||
trigger NEVER fires. The two arms produce statistically identical results.
|
||||
|
||||
### 2x QPS: unified_v4 under higher load
|
||||
|
||||
| Metric | unified_v4 @ 2x |
|
||||
|---|---:|
|
||||
| OK/total | 807/807 |
|
||||
| TTFT mean | 5.227s |
|
||||
| TTFT p90 | 15.000s |
|
||||
| E2E p90 | 39.401s |
|
||||
| **Migrations** | **4/807 (0.5%)** |
|
||||
|
||||
4 migrated requests (all verified via `v3_decode_target_url` in breakdown):
|
||||
|
||||
| Session | Input | new_local | src_pp | fleet_median | proj_prefill | Target |
|
||||
|---|---:|---:|---:|---:|---:|---|
|
||||
| 1313181 | 22,686 | 22,686 | 13,360 | 6,680 | 5.1s | inst_5 |
|
||||
| 1310590 | 32,440 | 14,520 | 57,051 | 12,630 | 10.2s | inst_4 |
|
||||
| 1373431 | 126,340 | 126,340 | 73,385 | 33,294 | 28.5s | inst_4 |
|
||||
| 1313181 | 60,004 | 17,508 | 19,503 | 3,806 | 5.3s | inst_5 |
|
||||
|
||||
## Root Cause Analysis: Why Zero Migrations at 1x
|
||||
|
||||
The unified_v4 trigger requires BOTH arms to fire simultaneously:
|
||||
- **Absolute SLO arm**: `proj_prefill_s(src) > 2.5s` — fires for 41% of eligible requests
|
||||
- **Relative arm**: `src_pending_prefill > fleet_median × 1.5` — NEVER fires at 1x
|
||||
|
||||
The relative arm fails because `pending_prefill_tokens` (the proxy's shadow
|
||||
counter) is 0 for **95% of routing decisions** at 1x QPS:
|
||||
|
||||
| QPS | src_pp > 0 (% of eligible) | Migrations |
|
||||
|---:|---:|---:|
|
||||
| 1.3 (1×) | 5% (8/241) | 0 |
|
||||
| 2.7 (2×) | 24% (62/257) | 4 |
|
||||
|
||||
**Mechanism**: `pending_prefill_tokens` reflects previously-dispatched requests
|
||||
that haven't finished their prefill yet. At 0.16 req/instance/s, each instance
|
||||
completes its prefill before the next request arrives — the counter is almost
|
||||
always 0 at decision time. Only under genuine queueing pressure (2× and above)
|
||||
does the counter become non-zero and the relative arm can fire.
|
||||
|
||||
The high TTFT at 1x (~11.5s p90) comes from **compute-bound large prefills**
|
||||
(single 60k+ token requests inherently need ~9s), NOT from queue depth.
|
||||
|
||||
## Interpretation for Paper
|
||||
|
||||
1. **The migration mechanism is functionally correct.** At 2x it fires on the
|
||||
right signal (src genuinely overloaded relative to fleet) and selects valid
|
||||
targets (cooler instances with load gap).
|
||||
|
||||
2. **At benchmark scale (8 instances, ~1 QPS), migration is not needed.** The
|
||||
affinity-default routing (Pillar 1) already achieves APC ~79% and the
|
||||
remaining hot-pin issue is mild (max-worker/median-worker ≈ 3.7×). The
|
||||
"dispatch coupling" feedback loop is present but not yet at the catastrophic
|
||||
amplification regime.
|
||||
|
||||
3. **Migration becomes relevant under scale-out + higher load.** With more
|
||||
instances (16–32), session skew concentrates more load per hot instance
|
||||
while cold instances sit idle — exactly the condition where `src_pp >
|
||||
fleet_median × 1.5` naturally fires. The 1x→2x progression (0%→0.5%
|
||||
migration rate) shows the correct scaling direction.
|
||||
|
||||
4. **Paper §3.3 framing**: Migration is a **scale-out insurance mechanism**
|
||||
that gracefully degrades to no-op under low load. Its value is NOT
|
||||
demonstrable at 8-instance single-node benchmark; the argument must rely on
|
||||
(a) the mechanism's correctness (this experiment), (b) the substrate's
|
||||
net-positive property (commit `ef9e010`), and (c) scale-out projection
|
||||
(future: 16+ GPU, multi-node).
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [ ] **Scale-out validation** (16 GPU, 2 nodes): With more instances and the
|
||||
same trace, more sessions compete per-instance → higher pending_prefill →
|
||||
migration triggers naturally. This is the strongest evidence path.
|
||||
- [ ] **3–4× QPS on 8 instances**: Push to saturation to measure migration's
|
||||
effect in the catastrophic regime. Risk: may exceed serving capacity (errors).
|
||||
- [ ] **Threshold sensitivity**: Ablate `v4_rel_hi` (1.5→1.2→1.0) and
|
||||
`v4_ttft_slo_s` (2.5→1.5→1.0) to characterize the trigger landscape.
|
||||
|
||||
## Reproduction
|
||||
|
||||
```bash
|
||||
# On dash2 (local /tmp, does NOT modify shared NAS):
|
||||
# 1x QPS
|
||||
bash /tmp/migration_exp/run_migration_ab.sh # interleaved unified vs unified_v4
|
||||
|
||||
# 2x QPS
|
||||
python3 -c "
|
||||
import json
|
||||
trace_in = '/home/admin/cpfs/wjh/agentic-kv/traces/w600_r0.0015_st30_first600s.jsonl'
|
||||
rows = [json.loads(l) for l in open(trace_in)]
|
||||
ts_min = min(r['timestamp'] for r in rows)
|
||||
for r in rows: r['timestamp'] = ts_min + (r['timestamp'] - ts_min) / 2.0
|
||||
with open('/tmp/migration_exp/trace_2x.jsonl', 'w') as f:
|
||||
for r in rows: f.write(json.dumps(r) + '\n')
|
||||
"
|
||||
bash /tmp/migration_exp/run_2x.sh
|
||||
```
|
||||
|
||||
## Data Locations (dash2 /tmp, ephemeral)
|
||||
|
||||
| Path | Content |
|
||||
|---|---|
|
||||
| `/tmp/migration_exp/outputs/unified_run1/` | Baseline arm (1x) |
|
||||
| `/tmp/migration_exp/outputs/unified_v4_run1/` | Migration arm (1x) |
|
||||
| `/tmp/migration_exp/outputs/unified_v4_2x/` | Migration arm (2x) |
|
||||
| `/tmp/migration_exp/outputs/*/breakdown.json` | Per-request routing decisions with v4_* fields |
|
||||
| `/tmp/migration_exp/outputs/*/metrics.jsonl` | Per-request latency metrics |
|
||||
@@ -1,110 +0,0 @@
|
||||
{
|
||||
"experiment": "migration_trigger_validation",
|
||||
"date": "2026-05-30",
|
||||
"hardware": "dash2, 8xH20, Qwen3-Coder-30B-A3B, TP=1",
|
||||
"substrate": "kv_both + DR-fix (delay_free_blocks + VLLM_EVICT_SENT_BLOCKS gate)",
|
||||
"arms": {
|
||||
"unified_1x": {
|
||||
"metrics": {
|
||||
"ok": 807,
|
||||
"total": 807,
|
||||
"ttft_mean": 3.99,
|
||||
"ttft_p50": 0.719,
|
||||
"ttft_p90": 11.499,
|
||||
"ttft_p99": 45.982,
|
||||
"tpot_p90": 0.0239,
|
||||
"e2e_p50": 2.265,
|
||||
"e2e_p90": 24.507,
|
||||
"e2e_p99": 71.233
|
||||
},
|
||||
"migrations": 0
|
||||
},
|
||||
"unified_v4_1x": {
|
||||
"metrics": {
|
||||
"ok": 807,
|
||||
"total": 807,
|
||||
"ttft_mean": 4.142,
|
||||
"ttft_p50": 0.711,
|
||||
"ttft_p90": 12.293,
|
||||
"ttft_p99": 46.148,
|
||||
"tpot_p90": 0.0217,
|
||||
"e2e_p50": 2.293,
|
||||
"e2e_p90": 23.955,
|
||||
"e2e_p99": 75.915
|
||||
},
|
||||
"trigger_summary": {
|
||||
"trace": "w600_r0.0015_st30_first600s.jsonl",
|
||||
"qps_factor": 1,
|
||||
"total_requests": 807,
|
||||
"migrations_triggered": 13,
|
||||
"size_floor_filtered": 552,
|
||||
"eligible_requests": 255,
|
||||
"slo_arm_true": 100,
|
||||
"src_pp_nonzero": 22
|
||||
}
|
||||
},
|
||||
"unified_v4_2x": {
|
||||
"metrics": {
|
||||
"ok": 807,
|
||||
"total": 807,
|
||||
"ttft_mean": 5.227,
|
||||
"ttft_p50": 0.942,
|
||||
"ttft_p90": 15.0,
|
||||
"ttft_p99": 59.227,
|
||||
"tpot_p90": 0.1087,
|
||||
"e2e_p50": 5.035,
|
||||
"e2e_p90": 39.401,
|
||||
"e2e_p99": 163.032
|
||||
},
|
||||
"trigger_summary": {
|
||||
"trace": "trace_2x.jsonl (timestamps / 2)",
|
||||
"qps_factor": 2,
|
||||
"total_requests": 807,
|
||||
"migrations_triggered": 4,
|
||||
"size_floor_filtered": 550,
|
||||
"eligible_requests": 257,
|
||||
"slo_arm_true": 133,
|
||||
"src_pp_nonzero": 62,
|
||||
"pending_prefill_p90_when_nonzero": 68131,
|
||||
"migrated_details": [
|
||||
{
|
||||
"session_id": "1313181",
|
||||
"input_length": 22686,
|
||||
"new_local": 22686,
|
||||
"src_pending_prefill": 13360,
|
||||
"fleet_median_pp": 6680.0,
|
||||
"proj_prefill_s": 5.15,
|
||||
"target_idx": 5
|
||||
},
|
||||
{
|
||||
"session_id": "1310590",
|
||||
"input_length": 32440,
|
||||
"new_local": 14520,
|
||||
"src_pending_prefill": 57051,
|
||||
"fleet_median_pp": 12630.5,
|
||||
"proj_prefill_s": 10.22,
|
||||
"target_idx": 4
|
||||
},
|
||||
{
|
||||
"session_id": "1373431",
|
||||
"input_length": 126340,
|
||||
"new_local": 126340,
|
||||
"src_pending_prefill": 73385,
|
||||
"fleet_median_pp": 33294.5,
|
||||
"proj_prefill_s": 28.53,
|
||||
"target_idx": 4
|
||||
},
|
||||
{
|
||||
"session_id": "1313181",
|
||||
"input_length": 60004,
|
||||
"new_local": 17508,
|
||||
"src_pending_prefill": 19503,
|
||||
"fleet_median_pp": 3806.5,
|
||||
"proj_prefill_s": 5.29,
|
||||
"target_idx": 5
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -23,22 +23,6 @@ Per-request breakdown shows **87.7% of TTFT** is spent waiting for KV cache memo
|
||||
> Earlier cross-machine comparison (commit `1e86285`) was invalidated — baseline used warm instances. See REPORT.md §3.5.
|
||||
| **Delta** | **-45%** | **-36%** | **-44%** | **+30pp** |
|
||||
|
||||
> ✅⚠️ **2026-05-30 — confirmed + refined by the clean MB5 ablation; one caveat.**
|
||||
> A producer-side contamination (`e13391e`: evicts a producer's prefix-cache on every
|
||||
> KV transfer) was found in the *agentic-kv-fresh* MB5 stack and gated off; everything
|
||||
> was re-run clean.
|
||||
> - **Confirmed:** this doc's central thesis — PD's failure is a **decode-side KV memory
|
||||
> wall**, not transfer/prefill cost — is reproduced on the clean stack (concurrency
|
||||
> axis: at N=64 the split collapses, APC 71%→1.4%, TPS −30%; colo scales). Fig 3.
|
||||
> - **Refined:** "PD separation is net negative" is **regime-dependent**, not universal.
|
||||
> Clean ablation shows PD *wins* at low load / decode-heavy / low-reuse and loses the
|
||||
> **agentic corner** (high reuse + short output + large context + high concurrency).
|
||||
> - **Caveat (cross-check):** if this study's runs used the fork vLLM that contains
|
||||
> `e13391e`, any **producer prefix-cache / APC reuse** figures here (e.g. §5.3.1) may be
|
||||
> understated. The decode-memory-wall result is *not* reuse-dependent and is unaffected.
|
||||
> On the clean stack, session-routed producers reach APC parity with colo (71–82%).
|
||||
> Figures: [`figs/mb5_pd_ablation/`](../figs/mb5_pd_ablation/).
|
||||
|
||||
---
|
||||
|
||||
## 1. Workload Characterization
|
||||
|
||||
@@ -1,165 +0,0 @@
|
||||
# PD-colo vs PD-disagg on the real agentic trace — LMetric (cache-aware) clean-stack anchor
|
||||
|
||||
**Figure:** [`figs/v2/fig4_lmetric_pd_vs_colo.png`](../../figs/v2/fig4_lmetric_pd_vs_colo.png)
|
||||
**Data:** [`analysis/v2/fig4l_lmetric.json`](fig4l_lmetric.json)
|
||||
**Date:** 2026-05-31 · Hardware: dash1, 8×H20 · Model: Qwen3-Coder-30B-A3B-Instruct
|
||||
· vLLM 0.18.1 (V1, chunked-prefill on, `e13391e` eviction gated **off**)
|
||||
· Mooncake 0.3.11 · Routing: cache-aware proxy with **`--policy lmetric`**
|
||||
· Replayer per-request timeout 600 s.
|
||||
|
||||
## TL;DR
|
||||
|
||||
On the production agentic trace with the *right* routing baseline (LMetric, cache-aware),
|
||||
**PD-colo (8× kv_both) keeps 100 % completion on both traces** and matches the daily-bench
|
||||
expectation (~17 min for the high-load first600s, ~50 min for the full trace, with E2E p50
|
||||
~3 s and TTFT p50 ~1 s — **3.5–7× better than the original §3 round-robin baseline at the
|
||||
same wall-clock**). Every static **PD-disagg ratio fails** (14–65 % completion), and the
|
||||
failure mode rotates predictably with the split — **no static partition has a working
|
||||
operating point on this workload**. LMetric improves colo dramatically; it does *not*
|
||||
rescue PD-disagg, confirming the bottleneck is structural (D-pool admission capacity +
|
||||
multi-turn KV accumulation), not routing. A follow-up linear-policy run with PD-disagg
|
||||
**wall-capped at 2× the colo wall** (see end of doc) hits the **identical** success-rate
|
||||
ceiling — confirming the cap is structural, not policy-driven.
|
||||
|
||||
## Setup
|
||||
|
||||
- Trace: `w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions, agentic multi-turn,
|
||||
contexts up to ~112 k tokens; "first600s" variant = same heavy sessions compressed
|
||||
into 600 s → 807 reqs at 3.2× higher arrival rate).
|
||||
- 8 instances on 8 GPUs.
|
||||
- `--mode baseline` for colo (plain vLLM); `--mode pdsep --pd-ratio P:D` for the three PD
|
||||
splits, all with Mooncake KV transfer.
|
||||
- Cache-aware proxy with LMetric scoring (`P_tokens × num_requests`) + session affinity
|
||||
for multi-turn (the colleague's canonical baseline).
|
||||
|
||||
## Results
|
||||
|
||||
### first600s (1.35 req/s, high-load stress)
|
||||
|
||||
| arm | success | E2E mean / p50 / p90 / p99 | TTFT p90 | TPOT p99 | TPS | wall |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **colo (8C)** | **807/807 = 100 %** | 11.1 / 3.27 / 28.6 / 95.9 s | 14.5 s | 388 ms | 226 | 17.0 min |
|
||||
| pd6 (6:2) | 474/807 = **58.7 %** | 83.2 / 6.75 / 382 / 542 s | 380 s | 19 ms | 40 | 55 min |
|
||||
| pd4 (4:4) | 348/807 = **43.1 %** | 203 / 215 / 477 / 575 s | 475 s | 25 ms | 15 | 114 min |
|
||||
| pd2 (2:6) | 180/807 = **22.3 %** | 380 / 536 / 579 / 602 s | 577 s | 18 ms | 34 | 321 min* |
|
||||
|
||||
### Full trace (0.42 req/s, original §3 anchor load)
|
||||
|
||||
| arm | success | E2E mean / p50 / p90 / p99 | TTFT p90 | TPOT p99 | TPS | wall |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **colo (8C)** | **1214/1214 = 100 %** | 10.9 / 3.13 / 29.6 / 93.7 s | 16.9 s | 254 ms | 125 | 49.9 min |
|
||||
| pd6 (6:2) | 793/1214 = **65.3 %** | 61.9 / 3.70 / 307 / 477 s | 300 s | 18 ms | 46 | 94 min |
|
||||
| pd4 (4:4) | 533/1214 = **43.9 %** | 131 / 8.22 / 468 / 531 s | 467 s | 21 ms | 13 | 231 min |
|
||||
| pd2 (2:6) | 169/1214 = **13.9 %** | 195 / 6.82 / 552 / 593 s | 549 s | 13 ms | 1 | 563 min |
|
||||
|
||||
\* The pd2 wall-clock is dominated by per-request timeouts (`request_timeout=600 s`)
|
||||
draining concurrently behind the multi-turn session causality.
|
||||
|
||||
## Five clean findings
|
||||
|
||||
1. **LMetric+colo is the right baseline.** Full-trace colo wall **49.9 min ≈ the original
|
||||
§3 RR's 49.9 min**, but E2E p50 **3.13 s vs §3's 10.8 s (3.5×)** and TTFT p50
|
||||
**1.02 s vs §3's 7.0 s (7×)**. Same throughput envelope, far better latency — by virtue
|
||||
of cache-aware routing concentrating each session's turns onto one instance for
|
||||
prefix-cache reuse. The original §3 RR was an *unfairly weak* colo baseline.
|
||||
|
||||
2. **Every static PD-disagg ratio fails on the agentic workload.** Completion drops to
|
||||
14–65 %, on *both* traces. The drop is not a high-load artifact (it holds at the
|
||||
original §3 arrival rate of 0.42 req/s); it is structural.
|
||||
|
||||
3. **Failure mode rotates predictably with the P:D split:**
|
||||
- **pd2 (2 producers)** → prefill-bound → 78–86 % TTFT timeouts.
|
||||
- **pd6 (2 decode)** → decode-admission-bound → 35–41 % TTFT timeouts.
|
||||
- **pd4 (4P+4D)** → both bottlenecks hit → 57 % TTFT timeouts.
|
||||
- **No static ratio works.** Colo's elastic 8-GPU pool absorbs whichever phase is
|
||||
hot at the moment.
|
||||
|
||||
4. **Decode isolation works, but doesn't matter under failure.** TPOT p99 on every PD
|
||||
arm is **13–25 ms** — an order of magnitude better than colo's 254–388 ms — but the
|
||||
win applies only to the 14–65 % of requests that get admitted. The other 35–86 %
|
||||
time out before ever seeing a first token, so the TPOT win is invisible to the end user.
|
||||
|
||||
5. **The §3 RR "100 % PD completion" was a measurement artifact.** Original §3
|
||||
(contaminated stack, RR routing) reported 100 % completion for pd6/pd4. LMetric on
|
||||
the clean stack shows 44–65 %. Most plausible mechanism: `e13391e`'s eviction of
|
||||
producer KV on every transfer acted as a **relief valve**, reducing producer-pool
|
||||
pressure and letting more requests squeeze under the 600 s timeout — at the (uncosted)
|
||||
price of cross-turn re-prefill. With the eviction off, producers retain prefix
|
||||
correctly → cache works on PD too → but the cache itself contends for producer
|
||||
pool capacity, and the decode-pool admission ceiling tips earlier. **PD-disagg is
|
||||
worse on agentic than §3 advertised, not better.**
|
||||
|
||||
## Linear-policy + wall-cap follow-up (2026-06-01) — the success ceiling is policy-invariant
|
||||
|
||||
To check whether the LMetric routing was secretly handicapping PD-disagg, we re-ran
|
||||
first600s with the **default `--policy linear`** (cache-aware load score + sticky
|
||||
session affinity — the original baseline of the cache_aware_proxy stack) and
|
||||
**wall-capped each PD-disagg arm at 2 × the colo wall** (kill bench.sh + cleanup
|
||||
GPUs once cap is exceeded, record `records_at_cap`).
|
||||
|
||||
| arm | linear success | linear wall | linear @-cap? | lmetric success | lmetric wall |
|
||||
|---|---|---|---|---|---|
|
||||
| **colo** | 807/807 = **100 %** | 964 s | — | 807/807 = **100 %** | 1021 s |
|
||||
| **pd6 (6:2)** | **472/807 = 58 %** | 2280 s | ⊗ cap (706 dispatched) | 474/807 = 59 % | 3325 s |
|
||||
| **pd4 (4:4)** | **349/807 = 43 %** | 2281 s | ⊗ cap (577 dispatched) | 348/807 = 43 % | 6850 s |
|
||||
| **pd2 (2:6)** | **176/807 = 22 %** | 2280 s | ⊗ cap (521 dispatched) | 180/807 = 22 % | 19275 s |
|
||||
|
||||
→ Figure: [`figs/v2/fig4_linear_vs_lmetric.png`](../../figs/v2/fig4_linear_vs_lmetric.png) ·
|
||||
data: [`fig4r_linear.json`](fig4r_linear.json)
|
||||
|
||||
**Three clean conclusions from the wall-cap experiment:**
|
||||
|
||||
1. **The success-rate ceiling is structural, not a routing artifact.** Linear and
|
||||
LMetric — two very different scoring policies (one session-sticky cache-aware,
|
||||
the other non-sticky pure load) — converge on **identical success rates**
|
||||
(58 / 43 / 22 %) for every PD-disagg ratio. Routing has *zero* effect on the
|
||||
completion ceiling. The bottleneck is the static P:D split itself.
|
||||
|
||||
2. **LMetric's longer wall was wall *wasted on requests that will never succeed*.**
|
||||
When the cap is enforced, linear hits the same ceiling in 2280 s as LMetric did
|
||||
in 3300–19000 s — the extra wall just slowly times out the unreachable
|
||||
requests at 600 s each.
|
||||
|
||||
3. **The wall-cap is the right way to bench PD-disagg.** Reporting "completion %"
|
||||
without a wall budget is misleading (the bench eventually completes if you wait
|
||||
forever, but only by counting timeouts as failures over hours). The honest
|
||||
metric is **success-in-2×-colo-wall**, which gives the same answer for both
|
||||
routings and matches what an end user would observe on a real SLO.
|
||||
|
||||
This **strengthens** the §5 D-pool capacity-ceiling thesis: even with
|
||||
session-affinity routing serving every request to a warm prefix cache (which
|
||||
*should* maximise PD's throughput), the static D-pool can't admit more than
|
||||
~22 / 43 / 58 % of the agentic trace within 2× the colo budget. Colo wins not
|
||||
because routing is smarter, but because its **elastic pool** absorbs whichever
|
||||
phase is hot — there's no cap to hit.
|
||||
|
||||
---
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
# On dash1, from the main repo /home/admin/cpfs/wjh/agentic-kv:
|
||||
for TR in w600_r0.0015_st30.jsonl w600_r0.0015_st30_first600s.jsonl; do
|
||||
TRACE=traces/$TR bash scripts/bench.sh --tag fig4l_lmetric_colo_${TR%.*} \
|
||||
--mode baseline --policy lmetric
|
||||
for r in 6:2 4:4 2:6; do
|
||||
TRACE=traces/$TR bash scripts/bench.sh --tag fig4l_lmetric_${r/:/p}_${TR%.*} \
|
||||
--mode pdsep --pd-ratio $r --policy lmetric
|
||||
done
|
||||
done
|
||||
|
||||
python microbench/fresh_setup/plot_fig4l_lmetric.py
|
||||
python microbench/fresh_setup/plot_fig4_linear_vs_lmetric.py
|
||||
```
|
||||
|
||||
For the linear + 2× wall-cap variant, run colo first to get `wall_clock_s`,
|
||||
compute `CAP=2*wall`, then launch each PD-disagg arm in the background and
|
||||
`SIGTERM` it (so bench.sh's cleanup trap fires) once `date +%s` minus the
|
||||
arm's start time exceeds `CAP`. The capped runs lack `metrics.summary.json`
|
||||
(replayer was killed before it could write); compute the summary directly from
|
||||
`metrics.jsonl` (see the inline collector used to build
|
||||
`analysis/v2/fig4r_linear.json`).
|
||||
|
||||
Source `bench.sh` cleans GPUs before each arm and writes `metrics.jsonl` +
|
||||
`metrics.summary.json` per tag. Aggregation script: see the inline JSON dump used
|
||||
to build `analysis/v2/fig4l_lmetric.json`.
|
||||
@@ -1 +0,0 @@
|
||||
[{"tag": "fig4l_lmetric_colo_first600s", "arm": "colo", "trace": "first600s", "n": 807, "req": 807, "e2e": {"count": 807.0, "mean": 11.066699584425269, "p50": 3.27055042097345, "p90": 28.745733462180937, "p99": 97.40008939541167}, "ttft": {"count": 807.0, "mean": 5.119651803458883, "p50": 1.2114678020589054, "p90": 14.777630288852365, "p99": 50.68302261995841}, "tpot": {"count": 807.0, "mean": 0.03004899278845205, "p50": 0.009643197803618922, "p90": 0.042092699501536976, "p99": 0.3919741264067197}, "wall": 1020.5351374909515, "tps": 226.12940164644368}, {"tag": "fig4l_lmetric_colo_full", "arm": "colo", "trace": "full", "n": 1214, "req": 1214, "e2e": {"count": 1214.0, "mean": 10.928977524270508, "p50": 3.1279119075043127, "p90": 30.011970606888667, "p99": 94.77313101590481}, "ttft": {"count": 1214.0, "mean": 5.533819193267678, "p50": 1.017395684029907, "p90": 17.36427243486981, "p99": 51.49416554694993}, "tpot": {"count": 1214.0, "mean": 0.02049970290344434, "p50": 0.009544484575988867, "p90": 0.032480608771520716, "p99": 0.26057810739537074}, "wall": 2993.276069591986, "tps": 125.38402448497122}, {"tag": "fig4l_lmetric_pd2_first600s", "arm": "2P+6D", "trace": "first600s", "n": 180, "req": 807, "e2e": {"count": 180.0, "mean": 380.2505690135715, "p50": 535.6594606440049, "p90": 579.5011055286858, "p99": 601.5567972306756}, "ttft": {"count": 180.0, "mean": 378.7133691522933, "p50": 534.4269686369807, "p90": 577.3534130641376, "p99": 596.404559875431}, "tpot": {"count": 180.0, "mean": 0.007975266077679418, "p50": 0.007166497974743372, "p90": 0.012511071875514153, "p99": 0.017508981961061446}, "wall": 19275.367093455978, "tps": 1.8895100582735462}, {"tag": "fig4l_lmetric_pd2_full", "arm": "2P+6D", "trace": "full", "n": 169, "req": 1214, "e2e": {"count": 169.0, "mean": 194.88523891245458, "p50": 6.817620265996084, "p90": 552.1569225640735, "p99": 595.3934216396092}, "ttft": {"count": 169.0, "mean": 193.4153314989016, "p50": 5.60239192598965, "p90": 549.3611521873856, "p99": 582.4436428000824}, "tpot": {"count": 169.0, "mean": 0.007747395842651413, "p50": 0.007691574401794991, "p90": 0.011201243427351017, "p99": 0.013311375577245894}, "wall": 33770.57413210906, "tps": 0.9869539045920406}, {"tag": "fig4l_lmetric_pd4_first600s", "arm": "4P+4D", "trace": "first600s", "n": 348, "req": 807, "e2e": {"count": 348.0, "mean": 202.63302869595395, "p50": 214.03008900902933, "p90": 477.40967412578175, "p99": 576.6393926549597}, "ttft": {"count": 348.0, "mean": 199.96385804087797, "p50": 213.50966987549327, "p90": 475.7766476540827, "p99": 559.6153268160638}, "tpot": {"count": 348.0, "mean": 0.008873619369764751, "p50": 0.007645836479973812, "p90": 0.013845969236959285, "p99": 0.02567216653158788}, "wall": 6850.181333696004, "tps": 15.00296050477674}, {"tag": "fig4l_lmetric_pd4_full", "arm": "4P+4D", "trace": "full", "n": 533, "req": 1214, "e2e": {"count": 533.0, "mean": 130.94711188977982, "p50": 8.219856544979848, "p90": 473.44134307731883, "p99": 533.2597587251009}, "ttft": {"count": 533.0, "mean": 127.83193208824007, "p50": 4.8246813879814, "p90": 467.54664219671395, "p99": 528.8304683346115}, "tpot": {"count": 533.0, "mean": 0.008886429490232585, "p50": 0.007981476340708988, "p90": 0.013570741891233497, "p99": 0.023050950961825044}, "wall": 13884.384965199977, "tps": 12.621372890425038}, {"tag": "fig4l_lmetric_pd6_first600s", "arm": "6P+2D", "trace": "first600s", "n": 474, "req": 807, "e2e": {"count": 474.0, "mean": 83.15809065495806, "p50": 6.7270191764691845, "p90": 391.6558471220078, "p99": 544.7372293809171}, "ttft": {"count": 474.0, "mean": 80.70155321074382, "p50": 4.1273433425230905, "p90": 390.00296151017517, "p99": 539.0574236416071}, "tpot": {"count": 474.0, "mean": 0.008519881756330928, "p50": 0.00803907146806204, "p90": 0.012583933303093976, "p99": 0.018606097790947705}, "wall": 3325.2749515309697, "tps": 39.705588838364164}, {"tag": "fig4l_lmetric_pd6_full", "arm": "6P+2D", "trace": "full", "n": 793, "req": 1214, "e2e": {"count": 793.0, "mean": 61.907526705667, "p50": 3.69814173609484, "p90": 308.2633092067672, "p99": 477.48038318102715}, "ttft": {"count": 793.0, "mean": 59.25069201986225, "p50": 1.402295546955429, "p90": 302.5604081378088, "p99": 475.3738951798529}, "tpot": {"count": 793.0, "mean": 0.009137289999448822, "p50": 0.008635683270933276, "p90": 0.013065757584108427, "p99": 0.01816783740464599}, "wall": 5662.029295974993, "tps": 39.24494000021532}]
|
||||
@@ -1 +0,0 @@
|
||||
[{"tag": "fig4r_linear_colo_first600s", "arm": "colo", "trace": "first600s", "policy": "linear", "n": 807, "req": 807, "dispatched": 807, "e2e": {"count": 807.0, "mean": 8.436370009274967, "p50": 2.5224755640374497, "p90": 22.65510415879542, "p99": 75.54369598095519}, "ttft": {"count": 807.0, "mean": 4.2332503390957195, "p50": 0.8872958200518042, "p90": 11.684667797433207, "p99": 44.98891795879462}, "tpot": {"count": 807.0, "mean": 0.020958194728517718, "p50": 0.00851320761584622, "p90": 0.026440129078245465, "p99": 0.30344440533287176}, "wall": 963.6191155100241, "tps": 239.4857016486815, "capped": false}, {"tag": "fig4r_linear_pd2_first600s", "arm": "2P+6D", "trace": "first600s", "policy": "linear", "n": 176, "req": 807, "dispatched": 521, "e2e": {"count": 176, "mean": 378.5561210460834, "p50": 536.7719694490079, "p90": 583.832092280034, "p99": 601.3415494390065}, "ttft": {"count": 176, "mean": 377.12570991374446, "p50": 536.1157373189926, "p90": 580.3465002350276, "p99": 598.0943597999867}, "tpot": {"count": 176, "mean": 0.007864906140929698, "p50": 0.007212154543958604, "p90": 0.011962352272927423, "p99": 0.017870794738764347}, "wall": 2280, "tps": 14.419736842105262, "capped": true}, {"tag": "fig4r_linear_pd4_first600s", "arm": "4P+4D", "trace": "first600s", "policy": "linear", "n": 349, "req": 807, "dispatched": 577, "e2e": {"count": 349, "mean": 264.8537863784421, "p50": 306.6853819829412, "p90": 488.64622142596636, "p99": 596.5830293919425}, "ttft": {"count": 349, "mean": 262.3163347712099, "p50": 299.75751709297765, "p90": 485.475125996978, "p99": 596.4081599479541}, "tpot": {"count": 349, "mean": 0.010442244895290958, "p50": 0.008213572105774598, "p90": 0.019443845545703716, "p99": 0.028178529054794}, "wall": 2281, "tps": 38.306882946076286, "capped": true}, {"tag": "fig4r_linear_pd6_first600s", "arm": "6P+2D", "trace": "first600s", "policy": "linear", "n": 472, "req": 807, "dispatched": 706, "e2e": {"count": 472, "mean": 118.632779156234, "p50": 12.702161715948023, "p90": 458.1609142010566, "p99": 526.5488834320568}, "ttft": {"count": 472, "mean": 115.80202843308507, "p50": 9.745031949947588, "p90": 455.81679951993283, "p99": 516.5850186559837}, "tpot": {"count": 472, "mean": 0.00950947083585719, "p50": 0.008435572332624966, "p90": 0.015233499645638644, "p99": 0.023447183093280886}, "wall": 2280, "tps": 61.69210526315789, "capped": true}]
|
||||
@@ -1,67 +0,0 @@
|
||||
# KV-cache Working-Set Sizing — GLM-5.1-FP8 · TP=8 · 1× B300 node
|
||||
|
||||
工具：`scripts/working_set_analysis.py`（可配置 GPU 型号 / 并行度 TP·PP·EP / 模型 config.json /
|
||||
KV dtype / 权重大小）。图：`figs/working_set/glm5_fp8_tp8_b300.png`。
|
||||
|
||||
## 复现
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/working_set_analysis.py \
|
||||
/home/gahow/phd/kvcache-simulator/bailian-traces/glm_coder_blksz_512_040915-040917.jsonl \
|
||||
--model-config /home/gahow/phd/kvcache-simulator/models/GLM-5/config.json \
|
||||
--gpu B300 --tp 8 --ep 8 --kv-dtype-bytes 1 --weight-gb 744 --min-ts 0 \
|
||||
--out figs/working_set/glm5_fp8_tp8_b300.png
|
||||
```
|
||||
|
||||
## 方法
|
||||
|
||||
`hash_ids` 是全局内容寻址 block id（同内容=同 id，复用=同 id 再现）。vLLM prefix cache 是
|
||||
block 级，所以**集群级 KV footprint = 任一时刻必须常驻的 distinct block 数**，与 placement 无关
|
||||
（affinity 只搬运 block，不改总量）。三种 working set：
|
||||
- `W_all` 永不淘汰（真上界）
|
||||
- `W_oracle` 每 block 只在 `[首次, 末次复用]` 常驻（Belady 完美预知 → 满 APC 上界的最小 HBM）
|
||||
- `W_denning(T)` 滑窗 T 内被访问的 distinct block（现实 TTL-LRU）
|
||||
|
||||
KV/token：MLA → `L·(kv_lora_rank+qk_rope_head_dim)·dtype`；GQA → `2·L·kv_heads·head_dim·dtype`
|
||||
（与 `kvcache-simulator/src/config.rs::kv_block_bytes` 一致）。
|
||||
|
||||
## 配置
|
||||
|
||||
| 项 | 值 |
|
||||
|---|---|
|
||||
| 模型 | GLM-5.1-FP8（MLA, L=78, kv_lora=512+rope=64） |
|
||||
| KV/token · KV/block(512) | **43.9 KiB** · **23.0 MB**（≈ Qwen3 GQA 96 KiB 的一半） |
|
||||
| 硬件 | 8× B300 (288 GB) = 2304 GB HBM/replica |
|
||||
| 预算 | FP8 权重 744 GB + act 32 GB → **KV pool = 1528 GB/node** |
|
||||
| trace | dash0 glm_coder，475k req，**1.25h active @ 106 QPS**，~40k tok/req（剔除 77 条负 ts 暖机） |
|
||||
| APC 上界 | **80.4%** |
|
||||
|
||||
## 结果
|
||||
|
||||
| 保留窗口 T | peak footprint | = 节点 (GPU) | APC@T |
|
||||
|---:|---:|---:|---:|
|
||||
| 2s（在飞下限）| 533 GB | 0.3 (3) | 1.7% |
|
||||
| 10s | 2,068 GB | 1.4 (11) | 15% |
|
||||
| 30s | 4,906 GB | 3.2 (26) | 42% |
|
||||
| 60s | 7,698 GB | 5.0 (40) | 56% |
|
||||
| 300s | 21,960 GB | 14.4 (115) | 74% |
|
||||
| **oracle（满 80.4%）** | **21,399 GB** | **14.0 (112)** | 80.4% |
|
||||
| retain-forever | 167,018 GB | 109 (874) | — |
|
||||
|
||||
## 结论
|
||||
|
||||
1. **Serving：1 节点绰绰有余。** 在飞 KV（τ≈2-5s）仅 533–1157 GB ≪ 单节点 1528 GB。
|
||||
MLA + B300 大 HBM 让 live footprint 微不足道——跑起来根本不缺显存。
|
||||
2. **缓存全部复用（80.4%）：1 节点差 ~14×。** oracle 下限 21.4 TB = 14 节点（112 GPU），
|
||||
真实 LRU ~2× → ~28 节点。单节点（1528 GB）只能 hold ~10s 窗口 → cache 侧 APC 仅 ~10-15%。
|
||||
要 ~56% 需 5 节点，~74% 需 ~14 节点。
|
||||
3. **瓶颈在长尾，不在 live。** 把 APC 50%→80% 装进 GPU HBM 要 5→14 节点，极不经济
|
||||
→ offload/migration 到 CPU DRAM（每节点 ~1.5 TB）是定量动机。与 Qwen 结论方向一致。
|
||||
|
||||
## 注意
|
||||
|
||||
- footprint 是 TTL-LRU（最浪费）+ shared-cache 下限：真实 capacity-LRU 同容量下 APC 更高，
|
||||
但分区/affinity 不均衡又抬高需求；oracle / retain-forever 给出下/上界。
|
||||
- GLM trace mean ~40k tok/req，是 Qwen trace（11k）的 ~3.5×（tokenizer + 抽取不同），
|
||||
**绝对 GB 不可跨模型横比**，方法与定性结论可比。
|
||||
- EP 不改变 KV 总量（只影响 expert 权重分布），`--ep` 仅作标注。
|
||||
@@ -1,81 +0,0 @@
|
||||
# Agentic workload characterization C1–C3 (full 051315 production trace)
|
||||
|
||||
Date 2026-05-29. Source: `trace-glm5.1-formatted/051315-051317.jsonl` on dash1
|
||||
(release file, 2,114,220 requests / 1,307,276 sessions / 2h, type=100% `coder`).
|
||||
This release file **is the full cluster-level production trace** — session skew
|
||||
reproduces 46.5/66.5/74.6/87.5/96.0 exactly. Compute: `compute_chars.py`
|
||||
(2-pass, ~65s, `~/ali-trace/.venv` python). Numbers: `chars.json`.
|
||||
|
||||
> ⚠️ **Cluster-level, not per-instance.** This is one cluster's aggregate stream.
|
||||
> Concurrent-session counts have NO denominator of "8 instances" — do not compare
|
||||
> them to a single deployment's instance count.
|
||||
|
||||
These three are NOT in the existing 13 analyzer figures (which are single-variable
|
||||
marginals on the older 041x traces). C1–C3 are joint/temporal and argument-bearing.
|
||||
|
||||
## C1 — the workload is a MIXTURE, not "multi-turn agentic" (`c1_session_mixture.png`)
|
||||
|
||||
- **90.3%** of sessions are single-turn; mean 1.62 turns, p99=18, max=3091.
|
||||
- But multi-turn sessions (9.7%) = **44.2% of requests** and **66.9% of input
|
||||
(prefill) mass**. Single-turn = **60.2% of output (decode) mass**.
|
||||
- Continuation hazard P(reach k+1 | reached k): turn1→2 only **10.2%**, but
|
||||
turn2→3 50.6%, turn5→6 87%, turn12→13 **94.3%** (Lindy / Pareto).
|
||||
- Predictability of heaviness at cold-start is near-zero:
|
||||
corr(turn1_input, session_mass)=0.15, corr(turn1_input, n_turns)=**0.04**.
|
||||
|
||||
**Routing:** heaviness is unpredictable at session start → proactive placement
|
||||
cannot pre-empt hot-pin → a REACTIVE mechanism (observable-load routing /
|
||||
migration) is required. But once a session has shown depth, it almost surely
|
||||
continues → "observed accumulated load" is the signal that works (not turn-1
|
||||
features, not cost-model prediction). The single/multi optimal strategies are
|
||||
opposite (load-balance the 90% one-shot sea vs affinity-pin the deep tail) and
|
||||
you can't tell them apart at turn 1 → the only viable policy starts everyone
|
||||
load-balanced and becomes sticky as turns accrue. This is exactly LPWL's
|
||||
emergent behavior (`new_uncached≈input`→by-load; `new_uncached≈0`→sticks), so
|
||||
C1 explains *why* a cache-aware-load score is the right shape — it auto-segments
|
||||
the mixture with no classifier.
|
||||
|
||||
## C2 — marginal work collapses while resident state explodes (`c2_work_amortization.png`)
|
||||
|
||||
Per turn: resident context grows 11k→56k+ tokens while new prefill collapses
|
||||
2.7k→~200 tokens; per-turn reuse climbs 83%→**99.6%**; resident/new ratio
|
||||
("the PD tax") grows to ~250× by turn 12, ~450× by turn 30.
|
||||
|
||||
**PD-colocation:** the dominant cost is keeping ~50k+ resident KV available for
|
||||
the next turn's tiny delta. Disaggregation physically splits a turn's prefill-KV
|
||||
(P) and decode-KV (D), and the next turn's prefix = [prevPrompt + prevAnswer]
|
||||
spans both → must be gathered/transferred; colocation keeps it local for free.
|
||||
**Routing:** route on delta (`input − cache_hit`), never total input — C2 is the
|
||||
trace-level justification for LPWL's score function.
|
||||
|
||||
## C3 — prefill/decode BALANCE (honest reframe) (`c3_prefill_decode_balance.png`)
|
||||
|
||||
- Token mass: 98.7% input / **1.3% output**; of input, 60% reused-prefix, 40%
|
||||
new-prefill (28.6B new-prefill tokens vs 0.94B decode tokens).
|
||||
- **But tokens ≠ time.** Under a per-request latency model (prefill@7k tok/s,
|
||||
TPOT 10ms), aggregate decode-time share ≈ **70% (robust 68–71% across
|
||||
constants)** — each decode token costs ~70–140× a prefill token. So this is
|
||||
NOT a "decode is negligible" workload.
|
||||
- Per-request the bottleneck FLIPS within a session: turn-1 (and the 90%
|
||||
single-turn) is prefill-bound; turns ≥3 are strongly decode-bound.
|
||||
|
||||
**PD-colocation (correct argument):** the workload has *substantial* work on both
|
||||
sides of the roofline — compute-bound prefill (~30% of time) and memory-bound
|
||||
decode (~70%). Colocation interleaves them on one GPU (chunked prefill +
|
||||
continuous batching) so compute and HBM bandwidth are both used; static
|
||||
disaggregation strands P-instances bandwidth-idle and D-instances compute-idle.
|
||||
The earlier "decode is 1.3% so nothing to isolate" instinct was WRONG (token vs
|
||||
time confusion) — C3b is the correction.
|
||||
|
||||
**Caveat:** C3b's 70% is a per-request-latency-weighted estimate; batched decode
|
||||
throughput will shift it. Ground-truth needs `-raw.jsonl` (`usage.cached_tokens`
|
||||
for exact reuse; `backend_first_response_time_ms` / `total_cost_time_ms` for real
|
||||
prefill vs decode wall time). Sampling that 522GB file is the next step.
|
||||
|
||||
## Goal mapping
|
||||
|
||||
| | argue PD-colocation | guide routing |
|
||||
|---|---|---|
|
||||
| C1 mixture + hazard | both segments favor colo (diff reasons) | reactive + auto-segment ⇒ LPWL shape |
|
||||
| C2 resident/delta | the PD tax (transfer/split resident KV) | route on delta, not total |
|
||||
| C3 prefill/decode | roofline complementarity (interleave) | per-req bottleneck flips within session |
|
||||
@@ -1,964 +0,0 @@
|
||||
{
|
||||
"mixture": {
|
||||
"single_sessions": 1179990,
|
||||
"multi_sessions": 127286,
|
||||
"req_single_pct": 55.81207253738968,
|
||||
"req_multi_pct": 44.187927462610325,
|
||||
"in_single_pct": 33.12487590117447,
|
||||
"in_multi_pct": 66.87512409882554,
|
||||
"out_single_pct": 60.24502960903973,
|
||||
"out_multi_pct": 39.75497039096027
|
||||
},
|
||||
"turns": {
|
||||
"mean": 1.6172713336739908,
|
||||
"p99": 18.0,
|
||||
"max": 3091,
|
||||
"single_turn_pct": 90.26326498765371
|
||||
},
|
||||
"hazard": {
|
||||
"1": 0.102101621998721,
|
||||
"2": 0.5062146469376287,
|
||||
"3": 0.7351961756478754,
|
||||
"4": 0.8113739305485657,
|
||||
"5": 0.8723731546954472,
|
||||
"6": 0.8669264241631353,
|
||||
"7": 0.9093235352011023,
|
||||
"8": 0.9240204920989971,
|
||||
"9": 0.901725753553022,
|
||||
"10": 0.9346178826585841,
|
||||
"11": 0.9260597637248089,
|
||||
"12": 0.9427685226874781,
|
||||
"13": 0.91950119395065,
|
||||
"14": 0.936865189289012,
|
||||
"15": 0.9382160896883085,
|
||||
"16": 0.9308646838684262,
|
||||
"17": 0.9371561574269995,
|
||||
"18": 0.9312862196131557,
|
||||
"19": 0.9333279456925813,
|
||||
"20": 0.9351459000779289,
|
||||
"21": 0.9399074074074074,
|
||||
"22": 0.9404984730568416,
|
||||
"23": 0.9473132921336546,
|
||||
"24": 0.9193940734188413,
|
||||
"25": 0.9497294046903187,
|
||||
"26": 0.9323793845764214,
|
||||
"27": 0.9483906016569333,
|
||||
"28": 0.9368466275239868,
|
||||
"29": 0.9472638336900031
|
||||
},
|
||||
"token_mass": {
|
||||
"total_input": 71116829368,
|
||||
"total_output": 940765734,
|
||||
"out_in_ratio_pct": 1.3228454394837104,
|
||||
"new_prefill": 28616906067,
|
||||
"reused_prefix": 42499923301,
|
||||
"new_prefill_pct_of_input": 40.23928839532401
|
||||
},
|
||||
"decode_time_fraction": {
|
||||
"optimistic_for_prefill": 0.6812079219496285,
|
||||
"mid": 0.6970810590484581,
|
||||
"pessimistic": 0.711448473592609
|
||||
},
|
||||
"per_turn": {
|
||||
"turn": [
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
14,
|
||||
15,
|
||||
16,
|
||||
17,
|
||||
18,
|
||||
19,
|
||||
20,
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
30,
|
||||
31,
|
||||
32,
|
||||
33,
|
||||
34,
|
||||
35,
|
||||
36,
|
||||
37,
|
||||
38,
|
||||
39,
|
||||
40,
|
||||
41,
|
||||
42,
|
||||
43,
|
||||
44,
|
||||
45,
|
||||
46,
|
||||
47,
|
||||
48,
|
||||
49,
|
||||
50,
|
||||
51,
|
||||
52,
|
||||
53,
|
||||
54,
|
||||
55,
|
||||
56,
|
||||
57,
|
||||
58,
|
||||
59,
|
||||
60,
|
||||
61,
|
||||
62,
|
||||
63,
|
||||
64,
|
||||
65,
|
||||
66,
|
||||
67,
|
||||
68,
|
||||
69,
|
||||
70,
|
||||
71,
|
||||
72,
|
||||
73,
|
||||
74,
|
||||
75,
|
||||
76,
|
||||
77,
|
||||
78,
|
||||
79,
|
||||
80,
|
||||
81,
|
||||
82,
|
||||
83,
|
||||
84,
|
||||
85,
|
||||
86,
|
||||
87,
|
||||
88,
|
||||
89,
|
||||
90,
|
||||
91,
|
||||
92,
|
||||
93,
|
||||
94,
|
||||
95,
|
||||
96,
|
||||
97,
|
||||
98,
|
||||
99,
|
||||
100,
|
||||
101,
|
||||
102,
|
||||
103,
|
||||
104,
|
||||
105,
|
||||
106,
|
||||
107,
|
||||
108,
|
||||
109,
|
||||
110,
|
||||
111,
|
||||
112,
|
||||
113,
|
||||
114,
|
||||
115,
|
||||
116,
|
||||
117,
|
||||
118,
|
||||
119,
|
||||
120,
|
||||
121,
|
||||
122,
|
||||
123,
|
||||
124,
|
||||
125,
|
||||
126,
|
||||
127,
|
||||
128,
|
||||
129,
|
||||
130,
|
||||
131,
|
||||
132,
|
||||
133,
|
||||
134,
|
||||
135,
|
||||
136,
|
||||
137,
|
||||
138,
|
||||
139,
|
||||
140,
|
||||
141,
|
||||
142,
|
||||
143,
|
||||
144,
|
||||
145,
|
||||
146,
|
||||
147,
|
||||
148
|
||||
],
|
||||
"med_resident_input": [
|
||||
11035.0,
|
||||
19505.0,
|
||||
28059.0,
|
||||
35089.0,
|
||||
41215.0,
|
||||
44750.0,
|
||||
47419.5,
|
||||
49874.0,
|
||||
51905.0,
|
||||
53068.0,
|
||||
54782.0,
|
||||
56414.0,
|
||||
58229.0,
|
||||
59123.5,
|
||||
60434.5,
|
||||
61320.0,
|
||||
62243.0,
|
||||
63411.0,
|
||||
64510.5,
|
||||
65423.0,
|
||||
66942.5,
|
||||
67965.0,
|
||||
68826.0,
|
||||
70165.5,
|
||||
70052.0,
|
||||
70936.0,
|
||||
71547.0,
|
||||
72648.0,
|
||||
73406.0,
|
||||
73844.0,
|
||||
73604.0,
|
||||
74937.5,
|
||||
74778.0,
|
||||
75460.0,
|
||||
75029.0,
|
||||
74978.0,
|
||||
75933.0,
|
||||
76590.0,
|
||||
74695.0,
|
||||
76813.0,
|
||||
77079.5,
|
||||
78310.0,
|
||||
77848.0,
|
||||
77549.0,
|
||||
78203.0,
|
||||
79102.0,
|
||||
79202.0,
|
||||
78821.0,
|
||||
79868.0,
|
||||
80229.5,
|
||||
80912.0,
|
||||
81620.0,
|
||||
81612.5,
|
||||
81836.5,
|
||||
82506.0,
|
||||
82948.0,
|
||||
82633.0,
|
||||
84107.5,
|
||||
84176.0,
|
||||
84441.0,
|
||||
84101.0,
|
||||
85192.0,
|
||||
84127.0,
|
||||
84783.5,
|
||||
85087.0,
|
||||
85771.5,
|
||||
86110.0,
|
||||
85374.5,
|
||||
87137.0,
|
||||
87677.0,
|
||||
88587.0,
|
||||
88656.0,
|
||||
88882.0,
|
||||
89284.0,
|
||||
91512.0,
|
||||
89850.0,
|
||||
90596.0,
|
||||
91244.0,
|
||||
92102.0,
|
||||
93431.0,
|
||||
92333.5,
|
||||
96682.0,
|
||||
94999.0,
|
||||
95226.5,
|
||||
95173.0,
|
||||
95910.0,
|
||||
96528.0,
|
||||
96508.0,
|
||||
97270.0,
|
||||
97301.0,
|
||||
97076.5,
|
||||
97105.0,
|
||||
98032.0,
|
||||
97962.5,
|
||||
97968.5,
|
||||
98310.0,
|
||||
97061.0,
|
||||
97631.0,
|
||||
100126.0,
|
||||
97765.0,
|
||||
101076.0,
|
||||
98198.5,
|
||||
98678.0,
|
||||
98307.0,
|
||||
99174.0,
|
||||
99882.0,
|
||||
99974.0,
|
||||
99757.0,
|
||||
100065.5,
|
||||
99943.0,
|
||||
100612.0,
|
||||
101138.0,
|
||||
106738.0,
|
||||
99621.0,
|
||||
101980.0,
|
||||
102252.0,
|
||||
103018.0,
|
||||
101238.0,
|
||||
102005.0,
|
||||
101897.0,
|
||||
103576.0,
|
||||
102159.5,
|
||||
102695.5,
|
||||
100590.5,
|
||||
103236.0,
|
||||
101812.0,
|
||||
103074.0,
|
||||
99966.0,
|
||||
102183.5,
|
||||
101882.0,
|
||||
102572.5,
|
||||
105622.5,
|
||||
106066.0,
|
||||
103974.0,
|
||||
105443.5,
|
||||
104716.0,
|
||||
105041.0,
|
||||
106628.0,
|
||||
108320.0,
|
||||
108022.5,
|
||||
107621.5,
|
||||
107664.0,
|
||||
107913.0,
|
||||
108630.0,
|
||||
108382.0,
|
||||
107216.5,
|
||||
105731.0,
|
||||
103986.0
|
||||
],
|
||||
"med_new_prefill": [
|
||||
11035.0,
|
||||
2920.0,
|
||||
1249.0,
|
||||
767.0,
|
||||
628.0,
|
||||
485.0,
|
||||
400.0,
|
||||
359.0,
|
||||
314.0,
|
||||
274.0,
|
||||
263.0,
|
||||
258.0,
|
||||
244.0,
|
||||
231.0,
|
||||
227.0,
|
||||
222.0,
|
||||
201.0,
|
||||
200.0,
|
||||
198.0,
|
||||
189.0,
|
||||
182.5,
|
||||
184.0,
|
||||
179.0,
|
||||
188.0,
|
||||
173.0,
|
||||
180.0,
|
||||
164.0,
|
||||
167.0,
|
||||
159.5,
|
||||
168.0,
|
||||
156.0,
|
||||
174.0,
|
||||
156.0,
|
||||
159.0,
|
||||
166.0,
|
||||
165.0,
|
||||
153.0,
|
||||
158.0,
|
||||
182.0,
|
||||
149.0,
|
||||
184.0,
|
||||
172.0,
|
||||
149.0,
|
||||
167.0,
|
||||
163.0,
|
||||
152.0,
|
||||
153.0,
|
||||
171.0,
|
||||
151.0,
|
||||
146.0,
|
||||
162.0,
|
||||
153.0,
|
||||
156.0,
|
||||
164.0,
|
||||
148.0,
|
||||
143.0,
|
||||
143.0,
|
||||
149.0,
|
||||
170.5,
|
||||
159.0,
|
||||
144.0,
|
||||
168.0,
|
||||
148.0,
|
||||
144.5,
|
||||
142.5,
|
||||
146.5,
|
||||
147.0,
|
||||
157.0,
|
||||
168.0,
|
||||
153.0,
|
||||
155.0,
|
||||
127.5,
|
||||
145.0,
|
||||
143.0,
|
||||
146.0,
|
||||
123.0,
|
||||
139.0,
|
||||
137.0,
|
||||
115.0,
|
||||
139.5,
|
||||
117.0,
|
||||
154.0,
|
||||
111.0,
|
||||
124.0,
|
||||
118.0,
|
||||
90.0,
|
||||
104.0,
|
||||
116.0,
|
||||
112.0,
|
||||
76.5,
|
||||
110.0,
|
||||
101.0,
|
||||
123.0,
|
||||
114.0,
|
||||
86.0,
|
||||
92.0,
|
||||
108.0,
|
||||
85.0,
|
||||
146.0,
|
||||
77.5,
|
||||
101.0,
|
||||
102.0,
|
||||
85.0,
|
||||
77.0,
|
||||
114.0,
|
||||
66.0,
|
||||
105.0,
|
||||
90.0,
|
||||
89.0,
|
||||
100.0,
|
||||
108.5,
|
||||
100.0,
|
||||
169.0,
|
||||
89.0,
|
||||
106.5,
|
||||
78.0,
|
||||
75.0,
|
||||
90.0,
|
||||
77.0,
|
||||
88.0,
|
||||
102.0,
|
||||
83.5,
|
||||
123.5,
|
||||
116.5,
|
||||
108.0,
|
||||
119.0,
|
||||
82.0,
|
||||
80.0,
|
||||
105.0,
|
||||
90.0,
|
||||
91.0,
|
||||
113.0,
|
||||
122.0,
|
||||
102.0,
|
||||
101.5,
|
||||
64.0,
|
||||
78.0,
|
||||
52.5,
|
||||
98.5,
|
||||
72.0,
|
||||
87.0,
|
||||
102.0,
|
||||
97.0,
|
||||
123.0,
|
||||
80.0,
|
||||
132.5,
|
||||
86.5,
|
||||
111.0
|
||||
],
|
||||
"med_output": [
|
||||
63.0,
|
||||
67.0,
|
||||
111.0,
|
||||
142.0,
|
||||
158.0,
|
||||
162.0,
|
||||
164.0,
|
||||
164.0,
|
||||
159.0,
|
||||
160.0,
|
||||
159.0,
|
||||
161.0,
|
||||
160.0,
|
||||
158.0,
|
||||
154.0,
|
||||
154.0,
|
||||
154.0,
|
||||
149.0,
|
||||
146.0,
|
||||
147.0,
|
||||
142.0,
|
||||
144.0,
|
||||
143.0,
|
||||
142.0,
|
||||
140.0,
|
||||
136.0,
|
||||
137.0,
|
||||
139.0,
|
||||
136.0,
|
||||
133.0,
|
||||
130.0,
|
||||
131.0,
|
||||
125.0,
|
||||
123.0,
|
||||
122.0,
|
||||
122.0,
|
||||
118.0,
|
||||
122.0,
|
||||
114.0,
|
||||
112.0,
|
||||
115.0,
|
||||
111.0,
|
||||
109.0,
|
||||
112.0,
|
||||
109.0,
|
||||
107.0,
|
||||
111.0,
|
||||
105.0,
|
||||
108.0,
|
||||
107.0,
|
||||
100.0,
|
||||
100.0,
|
||||
95.0,
|
||||
105.0,
|
||||
103.0,
|
||||
102.0,
|
||||
100.0,
|
||||
100.0,
|
||||
98.0,
|
||||
98.0,
|
||||
101.0,
|
||||
99.0,
|
||||
101.0,
|
||||
102.0,
|
||||
97.0,
|
||||
91.0,
|
||||
100.0,
|
||||
97.0,
|
||||
94.0,
|
||||
98.5,
|
||||
92.5,
|
||||
97.0,
|
||||
102.0,
|
||||
92.0,
|
||||
95.0,
|
||||
91.0,
|
||||
91.0,
|
||||
92.0,
|
||||
85.0,
|
||||
98.0,
|
||||
96.0,
|
||||
99.0,
|
||||
94.0,
|
||||
96.0,
|
||||
90.0,
|
||||
85.0,
|
||||
99.0,
|
||||
86.0,
|
||||
99.0,
|
||||
93.0,
|
||||
92.0,
|
||||
93.0,
|
||||
87.0,
|
||||
83.0,
|
||||
87.5,
|
||||
82.0,
|
||||
80.0,
|
||||
90.0,
|
||||
92.0,
|
||||
80.0,
|
||||
77.0,
|
||||
82.0,
|
||||
87.0,
|
||||
74.0,
|
||||
83.0,
|
||||
79.0,
|
||||
84.0,
|
||||
80.5,
|
||||
79.0,
|
||||
76.0,
|
||||
78.5,
|
||||
71.5,
|
||||
81.0,
|
||||
87.0,
|
||||
82.0,
|
||||
85.0,
|
||||
87.0,
|
||||
75.0,
|
||||
75.0,
|
||||
82.0,
|
||||
86.0,
|
||||
76.5,
|
||||
77.5,
|
||||
70.0,
|
||||
78.0,
|
||||
85.0,
|
||||
77.0,
|
||||
67.0,
|
||||
76.5,
|
||||
107.0,
|
||||
92.0,
|
||||
80.5,
|
||||
85.0,
|
||||
83.0,
|
||||
77.0,
|
||||
70.0,
|
||||
84.0,
|
||||
69.0,
|
||||
97.0,
|
||||
72.0,
|
||||
81.0,
|
||||
87.0,
|
||||
89.0,
|
||||
102.0,
|
||||
83.0,
|
||||
82.5,
|
||||
91.0,
|
||||
79.5
|
||||
],
|
||||
"resident_over_new": [
|
||||
1.0,
|
||||
6.679794520547945,
|
||||
22.46517213771017,
|
||||
45.748370273794,
|
||||
65.62898089171975,
|
||||
92.26804123711341,
|
||||
118.54875,
|
||||
138.92479108635098,
|
||||
165.30254777070064,
|
||||
193.67883211678833,
|
||||
208.29657794676805,
|
||||
218.65891472868216,
|
||||
238.64344262295083,
|
||||
255.94588744588745,
|
||||
266.23127753303964,
|
||||
276.2162162162162,
|
||||
309.6666666666667,
|
||||
317.055,
|
||||
325.81060606060606,
|
||||
346.15343915343914,
|
||||
366.8082191780822,
|
||||
369.375,
|
||||
384.5027932960894,
|
||||
373.22074468085106,
|
||||
404.9248554913295,
|
||||
394.0888888888889,
|
||||
436.2621951219512,
|
||||
435.0179640718563,
|
||||
460.2257053291536,
|
||||
439.54761904761904,
|
||||
471.8205128205128,
|
||||
430.67528735632186,
|
||||
479.34615384615387,
|
||||
474.59119496855345,
|
||||
451.98192771084337,
|
||||
454.41212121212124,
|
||||
496.29411764705884,
|
||||
484.746835443038,
|
||||
410.4120879120879,
|
||||
515.5234899328859,
|
||||
418.9103260869565,
|
||||
455.2906976744186,
|
||||
522.4697986577181,
|
||||
464.36526946107784,
|
||||
479.7730061349693,
|
||||
520.4078947368421,
|
||||
517.6601307189543,
|
||||
460.94152046783626,
|
||||
528.9271523178808,
|
||||
549.5171232876712,
|
||||
499.4567901234568,
|
||||
533.4640522875817,
|
||||
523.1570512820513,
|
||||
499.0030487804878,
|
||||
557.472972972973,
|
||||
580.0559440559441,
|
||||
577.8531468531469,
|
||||
564.4798657718121,
|
||||
493.7008797653959,
|
||||
531.0754716981132,
|
||||
584.0347222222222,
|
||||
507.0952380952381,
|
||||
568.4256756756756,
|
||||
586.7370242214533,
|
||||
597.1017543859649,
|
||||
585.4709897610921,
|
||||
585.7823129251701,
|
||||
543.7866242038217,
|
||||
518.672619047619,
|
||||
573.0522875816994,
|
||||
571.5290322580645,
|
||||
695.3411764705883,
|
||||
612.9793103448276,
|
||||
624.3636363636364,
|
||||
626.7945205479452,
|
||||
730.4878048780488,
|
||||
651.7697841726618,
|
||||
666.014598540146,
|
||||
800.8869565217391,
|
||||
669.7562724014336,
|
||||
789.1752136752136,
|
||||
627.8051948051948,
|
||||
855.8468468468468,
|
||||
767.9556451612904,
|
||||
806.5508474576271,
|
||||
1065.6666666666667,
|
||||
928.1538461538462,
|
||||
831.9655172413793,
|
||||
868.4821428571429,
|
||||
1271.9084967320262,
|
||||
882.5136363636364,
|
||||
961.4356435643564,
|
||||
797.0081300813008,
|
||||
859.3201754385965,
|
||||
1139.1686046511627,
|
||||
1068.5869565217392,
|
||||
898.7129629629629,
|
||||
1148.6,
|
||||
685.7945205479452,
|
||||
1261.483870967742,
|
||||
1000.7524752475248,
|
||||
962.7303921568628,
|
||||
1160.9176470588236,
|
||||
1276.7142857142858,
|
||||
869.9473684210526,
|
||||
1513.3636363636363,
|
||||
952.1333333333333,
|
||||
1108.411111111111,
|
||||
1124.3314606741574,
|
||||
999.43,
|
||||
927.2995391705069,
|
||||
1011.38,
|
||||
631.5857988165681,
|
||||
1119.3370786516855,
|
||||
957.5586854460093,
|
||||
1310.923076923077,
|
||||
1373.5733333333333,
|
||||
1124.8666666666666,
|
||||
1324.7402597402597,
|
||||
1157.9204545454545,
|
||||
1015.4509803921569,
|
||||
1223.4670658682635,
|
||||
831.5425101214574,
|
||||
863.4377682403433,
|
||||
955.8888888888889,
|
||||
855.563025210084,
|
||||
1257.0,
|
||||
1249.575,
|
||||
973.1761904761905,
|
||||
1132.0222222222221,
|
||||
1127.1703296703297,
|
||||
934.712389380531,
|
||||
869.3934426229508,
|
||||
1019.3529411764706,
|
||||
1038.8522167487686,
|
||||
1636.1875,
|
||||
1346.679487179487,
|
||||
2031.009523809524,
|
||||
1099.6954314720813,
|
||||
1500.3125,
|
||||
1237.028735632184,
|
||||
1055.5294117647059,
|
||||
1112.5051546391753,
|
||||
883.170731707317,
|
||||
1354.775,
|
||||
809.1811320754717,
|
||||
1222.3236994219653,
|
||||
936.8108108108108
|
||||
],
|
||||
"reuse_pct": [
|
||||
0.0,
|
||||
85.02947962061009,
|
||||
95.5486653123775,
|
||||
97.81412978426287,
|
||||
98.47628290670872,
|
||||
98.91620111731844,
|
||||
99.1564651672835,
|
||||
99.28018606889361,
|
||||
99.39504864656584,
|
||||
99.48368131453984,
|
||||
99.5199153006462,
|
||||
99.54266671393626,
|
||||
99.5809648113483,
|
||||
99.60929241333818,
|
||||
99.62438673274372,
|
||||
99.63796477495107,
|
||||
99.67707212055974,
|
||||
99.68459730961506,
|
||||
99.69307322063851,
|
||||
99.71111077144124,
|
||||
99.72737797363409,
|
||||
99.72927241962775,
|
||||
99.73992386598088,
|
||||
99.73206205328829,
|
||||
99.75304059841261,
|
||||
99.74625014097215,
|
||||
99.7707800466826,
|
||||
99.77012443563484,
|
||||
99.78271530937526,
|
||||
99.77249336438979,
|
||||
99.78805499701103,
|
||||
99.76780650542119,
|
||||
99.79138249217684,
|
||||
99.78929234031276,
|
||||
99.77875221580989,
|
||||
99.77993544773133,
|
||||
99.7985065781676,
|
||||
99.7937067502285,
|
||||
99.75634245933462,
|
||||
99.80602241808027,
|
||||
99.76128542608606,
|
||||
99.78036010726599,
|
||||
99.80860137704244,
|
||||
99.78465228436214,
|
||||
99.79156809841055,
|
||||
99.8078430381027,
|
||||
99.80682306002375,
|
||||
99.7830527397521,
|
||||
99.81093804777883,
|
||||
99.81802204924622,
|
||||
99.79978247973106,
|
||||
99.8125459446214,
|
||||
99.8088528105376,
|
||||
99.79960042279423,
|
||||
99.82061910648923,
|
||||
99.82760283551141,
|
||||
99.82694565125313,
|
||||
99.822845762863,
|
||||
99.79744820376354,
|
||||
99.81170284577398,
|
||||
99.82877730348034,
|
||||
99.80279838482487,
|
||||
99.82407550489143,
|
||||
99.82956589430727,
|
||||
99.8325243574224,
|
||||
99.82919734410615,
|
||||
99.82928811984671,
|
||||
99.81610434028896,
|
||||
99.80720015607606,
|
||||
99.82549585410084,
|
||||
99.8250307607211,
|
||||
99.85618570655116,
|
||||
99.83686235683265,
|
||||
99.83983692486895,
|
||||
99.84045808200017,
|
||||
99.86310517529216,
|
||||
99.84657159256479,
|
||||
99.84985314102846,
|
||||
99.87513843347593,
|
||||
99.85069195449047,
|
||||
99.87328542728262,
|
||||
99.84071492108149,
|
||||
99.88315666480699,
|
||||
99.8697841462198,
|
||||
99.87601525642776,
|
||||
99.90616202690022,
|
||||
99.89225924084204,
|
||||
99.87980271065611,
|
||||
99.88485658476407,
|
||||
99.9213779920042,
|
||||
99.88668730331234,
|
||||
99.89598887801864,
|
||||
99.87453076546434,
|
||||
99.88362893964528,
|
||||
99.91221668189266,
|
||||
99.90641847217984,
|
||||
99.88872976787793,
|
||||
99.91293748911718,
|
||||
99.8541837285021,
|
||||
99.92072827699074,
|
||||
99.90007519094543,
|
||||
99.89612875960428,
|
||||
99.91386124566772,
|
||||
99.92167393980083,
|
||||
99.88505051727267,
|
||||
99.93392202799302,
|
||||
99.89497269290015,
|
||||
99.90978076726445,
|
||||
99.91105825684177,
|
||||
99.89994296749147,
|
||||
99.89215998091679,
|
||||
99.90112519527774,
|
||||
99.84166838426802,
|
||||
99.91066140673152,
|
||||
99.89556775838399,
|
||||
99.92371787348903,
|
||||
99.9271971888408,
|
||||
99.91110057488295,
|
||||
99.92451350423998,
|
||||
99.91363828179436,
|
||||
99.90152158801267,
|
||||
99.91826506590185,
|
||||
99.87974156608614,
|
||||
99.8841838941053,
|
||||
99.89538533069859,
|
||||
99.883117903587,
|
||||
99.92044550517105,
|
||||
99.91997279074886,
|
||||
99.89724368415645,
|
||||
99.91166251153295,
|
||||
99.91128226376466,
|
||||
99.89301521929514,
|
||||
99.88497727829841,
|
||||
99.90189855156096,
|
||||
99.9037399175862,
|
||||
99.93888231024867,
|
||||
99.92574328119497,
|
||||
99.95076340173313,
|
||||
99.90906573116692,
|
||||
99.9333472193293,
|
||||
99.91916113415999,
|
||||
99.90526081141329,
|
||||
99.91011277603255,
|
||||
99.88677161005248,
|
||||
99.92618700522226,
|
||||
99.8764182751722,
|
||||
99.91818861071965,
|
||||
99.89325486123131
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -1,180 +0,0 @@
|
||||
import json, sys, math, statistics as st
|
||||
from collections import defaultdict, Counter
|
||||
import matplotlib; matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
|
||||
PATH="/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl"
|
||||
OUT="/tmp/wlc_out"; import os; os.makedirs(OUT, exist_ok=True)
|
||||
BLOCK=512
|
||||
# --- transparent cost model for C3 (clearly-labeled estimate; raw-timing validation pending) ---
|
||||
PREFILL_TOK_S=7000.0 # MB1: 32k->4.5s ~7100 tok/s effective on H20 / 30B-A3B
|
||||
TPOT_S=0.010 # ~10ms/token decode (crossover unloaded ~5ms, loaded ~25ms)
|
||||
|
||||
def pct(v,p):
|
||||
if not v: return float('nan')
|
||||
s=sorted(v);k=(len(s)-1)*p;f=int(k)
|
||||
return s[f] if f+1>=len(s) else s[f]+(s[f+1]-s[f])*(k-f)
|
||||
|
||||
# ---------- Pass A: structure (scalars only) ----------
|
||||
parents={}; recs={}; childcount=Counter()
|
||||
for line in open(PATH):
|
||||
if not line.strip(): continue
|
||||
d=json.loads(line); cid=d["chat_id"]; pid=d["parent_chat_id"]
|
||||
parents[cid]=pid
|
||||
recs[cid]=(float(d["timestamp"]),int(d["input_length"]),int(d["output_length"]),int(d["turn"]))
|
||||
if pid!="-1": childcount[pid]+=1
|
||||
print(f"[A] records={len(recs)}", file=sys.stderr)
|
||||
|
||||
root_of={}
|
||||
def root(cid):
|
||||
path=[];c=cid
|
||||
while True:
|
||||
if c in root_of:r=root_of[c];break
|
||||
p=parents.get(c,"-1")
|
||||
if p=="-1" or p not in recs:r=c;break
|
||||
path.append(c);c=p
|
||||
for x in path:root_of[x]=r
|
||||
root_of[cid]=r;return r
|
||||
sessions=defaultdict(list)
|
||||
for cid in recs: sessions[root(cid)].append(cid)
|
||||
seq={r:sorted(m,key=lambda c:(recs[c][3],recs[c][0])) for r,m in sessions.items()}
|
||||
print(f"[A] sessions={len(seq)}", file=sys.stderr)
|
||||
|
||||
# ---------- C1: mixture + turn tail + hazard ----------
|
||||
sr=mr=sm=mm=so=mo=0
|
||||
turns_per=[]
|
||||
for r,s in seq.items():
|
||||
multi=len(s)>1; turns_per.append(len(s))
|
||||
for c in s:
|
||||
_,inl,outl,_=recs[c]
|
||||
if multi: mr+=1;mm+=inl;mo+=outl
|
||||
else: sr+=1;sm+=inl;so+=outl
|
||||
tot_r=sr+mr; tot_in=sm+mm; tot_out=so+mo
|
||||
cnt_turn=Counter()
|
||||
for r,s in seq.items():
|
||||
for c in s: cnt_turn[recs[c][3]]+=1
|
||||
hazard={k: (cnt_turn[k+1]/cnt_turn[k] if cnt_turn[k] else 0) for k in range(1,30)}
|
||||
|
||||
# ---------- C2/C3: per-turn resident vs new-prefill (scalar) + hash_ids reuse ----------
|
||||
by_in=defaultdict(list); by_new=defaultdict(list); by_out=defaultdict(list)
|
||||
by_reuse_hash=defaultdict(list) # hash-block prefix stability: reused/parent_blocks
|
||||
store={} # cid -> (blockset, in, out) for chats with pending children
|
||||
tot_new_prefill=0; tot_reused=0
|
||||
for line in open(PATH):
|
||||
if not line.strip(): continue
|
||||
d=json.loads(line); cid=d["chat_id"]; pid=d["parent_chat_id"]
|
||||
inl=int(d["input_length"]); outl=int(d["output_length"]); turn=int(d["turn"])
|
||||
blocks=set(d["hash_ids"])
|
||||
if pid in store:
|
||||
pblk,pin,pout=store[pid]
|
||||
new_prefill=max(0, inl - pin - pout) # actual recompute (accounts for cached answer)
|
||||
reused_blk=len(blocks & pblk)
|
||||
by_reuse_hash[turn].append(reused_blk/len(pblk) if pblk else 0)
|
||||
childcount[pid]-=1
|
||||
if childcount[pid]<=0: del store[pid]
|
||||
tot_reused += (inl-new_prefill)
|
||||
else:
|
||||
new_prefill=inl # session start: all new (intra-session)
|
||||
tot_new_prefill+=new_prefill
|
||||
by_in[turn].append(inl); by_new[turn].append(new_prefill); by_out[turn].append(outl)
|
||||
if childcount[cid]>0: store[cid]=(blocks,inl,outl)
|
||||
print(f"[B] done; store residual={len(store)}", file=sys.stderr)
|
||||
|
||||
TURNS=[t for t in sorted(by_in) if len(by_in[t])>=50]
|
||||
med_in=[pct(by_in[t],.5) for t in TURNS]
|
||||
med_new=[max(pct(by_new[t],.5),1) for t in TURNS]
|
||||
med_out=[pct(by_out[t],.5) for t in TURNS]
|
||||
ratio=[med_in[i]/med_new[i] for i in range(len(TURNS))]
|
||||
reuse_pct=[(1-med_new[i]/med_in[i])*100 for i in range(len(TURNS))]
|
||||
# C3 time per turn (cost model)
|
||||
t_pref=[med_new[i]/PREFILL_TOK_S for i in range(len(TURNS))]
|
||||
t_dec=[med_out[i]*TPOT_S for i in range(len(TURNS))]
|
||||
|
||||
# aggregate decode/prefill time fraction over a RANGE of constants
|
||||
def agg_time(prate,tpot):
|
||||
tp=tot_new_prefill/prate; td=tot_out*tpot; return td/(tp+td)
|
||||
frac_lo=agg_time(13000,0.005); frac_mid=agg_time(7000,0.010); frac_hi=agg_time(3000,0.025)
|
||||
|
||||
chars={
|
||||
"mixture":{"single_sessions":sr if False else sum(1 for s in seq.values() if len(s)==1),
|
||||
"multi_sessions":sum(1 for s in seq.values() if len(s)>1),
|
||||
"req_single_pct":sr/tot_r*100,"req_multi_pct":mr/tot_r*100,
|
||||
"in_single_pct":sm/tot_in*100,"in_multi_pct":mm/tot_in*100,
|
||||
"out_single_pct":so/tot_out*100,"out_multi_pct":mo/tot_out*100},
|
||||
"turns":{"mean":st.mean(turns_per),"p99":pct(turns_per,.99),"max":max(turns_per),
|
||||
"single_turn_pct":sum(1 for x in turns_per if x==1)/len(turns_per)*100},
|
||||
"hazard":hazard,
|
||||
"token_mass":{"total_input":tot_in,"total_output":tot_out,"out_in_ratio_pct":tot_out/tot_in*100,
|
||||
"new_prefill":tot_new_prefill,"reused_prefix":tot_reused,
|
||||
"new_prefill_pct_of_input":tot_new_prefill/tot_in*100},
|
||||
"decode_time_fraction":{"optimistic_for_prefill":frac_lo,"mid":frac_mid,"pessimistic":frac_hi},
|
||||
"per_turn":{"turn":TURNS,"med_resident_input":med_in,"med_new_prefill":med_new,
|
||||
"med_output":med_out,"resident_over_new":ratio,"reuse_pct":reuse_pct},
|
||||
}
|
||||
json.dump(chars, open(f"{OUT}/chars.json","w"), indent=2)
|
||||
|
||||
# ================= FIGURES =================
|
||||
plt.rcParams.update({"figure.dpi":140,"font.size":10,"axes.grid":True,"grid.alpha":.3})
|
||||
|
||||
# ---- C1 ----
|
||||
fig,ax=plt.subplots(1,3,figsize=(15,4.2))
|
||||
cats=["% sessions","% requests","% input\ntokens","% output\ntokens"];
|
||||
singv=[chars["mixture"]["single_sessions"]/len(seq)*100, chars["mixture"]["req_single_pct"],
|
||||
chars["mixture"]["in_single_pct"], chars["mixture"]["out_single_pct"]]
|
||||
multv=[100-x for x in singv]
|
||||
x=np.arange(len(cats))
|
||||
ax[0].bar(x,singv,label="single-turn",color="#7fb3d5")
|
||||
ax[0].bar(x,multv,bottom=singv,label="multi-turn",color="#e74c3c")
|
||||
for i in range(len(cats)):
|
||||
ax[0].text(i,singv[i]/2,f"{singv[i]:.0f}",ha="center",va="center",fontsize=9)
|
||||
ax[0].text(i,singv[i]+multv[i]/2,f"{multv[i]:.0f}",ha="center",va="center",color="white",fontsize=9)
|
||||
ax[0].set_xticks(x);ax[0].set_xticklabels(cats);ax[0].set_ylabel("%");ax[0].set_ylim(0,100)
|
||||
ax[0].set_title("C1a Mixture: 90% sessions single-turn,\nbut multi-turn carries 2/3 prefill mass");ax[0].legend(loc="center right")
|
||||
# turn CCDF log-log
|
||||
tc=sorted(turns_per); n=len(tc); xs=sorted(set(tc))
|
||||
ccdf=[sum(1 for v in tc if v>=xx)/n for xx in xs]
|
||||
ax[1].loglog(xs,ccdf,marker=".",ms=3,color="#34495e")
|
||||
ax[1].set_xlabel("turns per session (k)");ax[1].set_ylabel("P(turns >= k)")
|
||||
ax[1].set_title(f"C1b Heavy-tailed session length\n(p99={chars['turns']['p99']:.0f}, max={chars['turns']['max']})")
|
||||
# hazard
|
||||
hk=list(range(1,20)); hv=[hazard[k]*100 for k in hk]
|
||||
ax[2].plot(hk,hv,marker="o",color="#16a085")
|
||||
ax[2].set_xlabel("reached turn k");ax[2].set_ylabel("P(continue to k+1) %");ax[2].set_ylim(0,100)
|
||||
ax[2].set_title("C1c Continuation hazard rises 10%->94%\n(unpredictable at start, Lindy after)")
|
||||
fig.tight_layout(); fig.savefig(f"{OUT}/c1_session_mixture.png"); plt.close(fig)
|
||||
|
||||
# ---- C2 ----
|
||||
fig,ax=plt.subplots(1,3,figsize=(15,4.2))
|
||||
ax[0].semilogy(TURNS,med_in,marker="o",label="resident context (input)",color="#e74c3c")
|
||||
ax[0].semilogy(TURNS,med_new,marker="s",label="new prefill this turn",color="#2980b9")
|
||||
ax[0].set_xlabel("turn");ax[0].set_ylabel("tokens (median, log)");ax[0].legend()
|
||||
ax[0].set_xlim(1,30)
|
||||
ax[0].set_title("C2a Resident state explodes,\nmarginal work collapses")
|
||||
ax[1].plot(TURNS,ratio,marker="o",color="#8e44ad")
|
||||
ax[1].set_xlabel("turn");ax[1].set_ylabel("resident / new-prefill");ax[1].set_xlim(1,30)
|
||||
ax[1].set_title("C2b The PD tax = resident/delta\n(grows to ~250x by deep turns)")
|
||||
ax[2].plot(TURNS,reuse_pct,marker="o",color="#27ae60")
|
||||
ax[2].set_xlabel("turn");ax[2].set_ylabel("per-turn reuse %");ax[2].set_ylim(50,100);ax[2].set_xlim(1,30)
|
||||
ax[2].set_title("C2c Per-turn reuse climbs to 99.6%\n(deep turns are near-pure cache hits)")
|
||||
fig.tight_layout(); fig.savefig(f"{OUT}/c2_work_amortization.png"); plt.close(fig)
|
||||
|
||||
# ---- C3 ----
|
||||
fig,ax=plt.subplots(1,2,figsize=(11,4.4))
|
||||
# token mass decomposition
|
||||
vals=[tot_reused/1e9, tot_new_prefill/1e9, tot_out/1e9]
|
||||
labs=[f"reused prefix\n{tot_reused/tot_in*100:.0f}% of input",
|
||||
f"new prefill\n{tot_new_prefill/tot_in*100:.0f}% of input",
|
||||
f"decode output\n{tot_out/tot_in*100:.1f}% of input"]
|
||||
ax[0].bar(range(3),vals,color=["#95a5a6","#2980b9","#e67e22"])
|
||||
ax[0].set_xticks(range(3));ax[0].set_xticklabels(labs,fontsize=8.5)
|
||||
ax[0].set_ylabel("tokens (billions)")
|
||||
ax[0].set_title("C3a Token mass: prefill-dominated\n(but tokens != time, see C3b)")
|
||||
# per-turn prefill vs decode TIME (cost model)
|
||||
ax[1].semilogy(TURNS,t_pref,marker="o",label="prefill time (new tok / 7k·s⁻¹)",color="#2980b9")
|
||||
ax[1].semilogy(TURNS,t_dec,marker="s",label="decode time (out·10ms)",color="#e67e22")
|
||||
ax[1].set_xlabel("turn");ax[1].set_ylabel("seconds (median, log)");ax[1].legend(fontsize=8);ax[1].set_xlim(1,30)
|
||||
ax[1].set_title(f"C3b Prefill→decode bottleneck flips within a session\n(agg decode-time share ≈ {frac_mid*100:.0f}%, range {frac_lo*100:.0f}–{frac_hi*100:.0f}%)")
|
||||
fig.tight_layout(); fig.savefig(f"{OUT}/c3_prefill_decode_balance.png"); plt.close(fig)
|
||||
print("FIGURES + chars.json written to", OUT)
|
||||
print(json.dumps({k:chars[k] for k in ["mixture","turns","token_mass","decode_time_fraction"]}, indent=2))
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 169 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 119 KiB After Width: | Height: | Size: 55 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 77 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 73 KiB After Width: | Height: | Size: 36 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 132 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 35 KiB After Width: | Height: | Size: 38 KiB |
BIN
figs/f4c_apc_vs_hotspot_tradeoff.png
Normal file
BIN
figs/f4c_apc_vs_hotspot_tradeoff.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 49 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 54 KiB After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 110 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 114 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 78 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 71 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 72 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 104 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 95 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 92 KiB |
@@ -1,19 +0,0 @@
|
||||
# ⚠️ Correction notice for figs/mb5/ (2026-05-30)
|
||||
|
||||
These figures back `microbench/fresh_setup/PD_DISAGG_RESULTS.md`. A producer-side
|
||||
contamination was later found in the stack that produced them: commit **`e13391e`**
|
||||
(deployed over the "fresh" pip vLLM by `scripts/deploy_vllm_patches.sh`) evicts a
|
||||
producer's prefix-cache blocks on every KV transfer, so a disaggregated producer
|
||||
could never keep a session's prefix warm. It is now gated behind
|
||||
`VLLM_EVICT_SENT_BLOCKS` (default off) and everything was re-run clean.
|
||||
|
||||
| figure | section | status |
|
||||
|---|---|---|
|
||||
| `mb5_producer_hotspot.png` | §6.3 session-affinity hot-pinning | 🛑 **RETRACTED** — pure `e13391e` artifact. On the clean stack, session-routed producers reach APC parity with colo (71–82%); there is no 0%-util stall / hot-pin pathology. |
|
||||
| `mb5_latency_compare.png` | §3 round-robin headline | ✅ stands — RR's ~0% prefix-hit is a *routing* artifact (consecutive turns → different producers), not the eviction bug; reproduced clean. |
|
||||
| `mb5_kv_timeline.png`, `mb5_role_split.png`, `mb5_peak_utilization.png` | §5 per-role KV pool occupancy | ✅ D-pool capacity-ceiling mechanism stands (decode pegs while prefill strands). P-pool occupancy may read slightly low under eviction; the qualitative split is unaffected. |
|
||||
| `mb5_summary.csv` | aggregate | mixed — §3/§5 rows valid; any session-affinity rows superseded. |
|
||||
|
||||
**Superseded by the corrected three-axis ablation:** [`../mb5_pd_ablation/`](../mb5_pd_ablation/)
|
||||
(reuse / shape / concurrency), data in [`../../analysis/mb5_pd_ablation/`](../../analysis/mb5_pd_ablation/).
|
||||
Raw §6 data `analysis/mb5/session_prod.json` is contaminated; `analysis/mb5/rr_prod.json` (round-robin) stands.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 176 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 271 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 375 KiB |
@@ -1,5 +0,0 @@
|
||||
config,rep,n_requests,n_success,wall_clock_s,peak_pool_frac,steady_pool_frac,p_pool_peak_frac,p_pool_steady_frac,d_pool_peak_frac,d_pool_steady_frac,peak_waiting,latency_p50_s,latency_p90_s,latency_p99_s,ttft_p50_s,ttft_p90_s,ttft_p99_s,prefix_cache_hit_ratio
|
||||
8C,1,1214,1214,2994.218414353032,0.7174957362137578,0.3439702956225128,,,,,29,10.82550932947197,83.34998885790122,194.10265863158946,6.967104309005663,53.12018221841427,114.12611859919207,0.1937163528742694
|
||||
6P+2D,1,1214,1214,3419.065942236979,0.7726478112563957,0.42145750426378625,0.743272692817889,0.3082291074474133,0.9959636156907333,0.7434906196702672,128,44.48975181748392,91.82252187062406,147.70196208347772,40.95952733900049,86.68752026481089,142.84028979733685,0.0
|
||||
4P+4D,1,1214,1214,4170.666486939997,0.6997939169982945,0.45876918703808983,0.6438459351904491,0.28540363843092664,0.9753411028993746,0.5977686185332576,152,59.52004547297838,157.08703426021387,224.03997302683115,56.419772224500775,153.07864206891392,219.73412787001706,0.0
|
||||
2P+6D,1,1214,109,5761.816568834998,0.9698692438885731,0.9435119386014781,0.9969869243888573,0.9198408186469585,0.9620238772029562,0.9494504453287853,872,26.293884326005355,499.3484142678091,577.7122636228032,23.580788671970367,498.0334587502061,576.5306194114453,0.0
|
||||
|
Binary file not shown.
|
Before Width: | Height: | Size: 122 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 146 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 171 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 114 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 99 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 57 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 121 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 109 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 149 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 193 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 111 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 108 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 89 KiB |
@@ -1,178 +0,0 @@
|
||||
# Why KV-transfer is slow during migration under real load
|
||||
|
||||
**Question.** EAR's unified+A+B routing beats migration (v3) on agentic
|
||||
workloads. We wanted to know whether *layerwise* KV transfer would shrink
|
||||
migration's overhead enough to make it viable. Investigating that led to a
|
||||
sharper question: **in a real (loaded) cluster, when we migrate, the KV
|
||||
transfer is already slow — the effective bandwidth is far below the
|
||||
~10 GB/s wire rate. Why?**
|
||||
|
||||
This doc answers that with instrumented measurements.
|
||||
|
||||
**TL;DR.** Migration fires precisely when instances are *busy* (that's the
|
||||
trigger). But on a busy instance, KV transfer runs at **~3 GB/s instead of
|
||||
~10 GB/s**, because:
|
||||
|
||||
1. **The RDMA write itself slows ~2× under compute load** — GPU-direct RDMA
|
||||
(`batch_transfer_sync_write`) contends with the running attention/MLP
|
||||
kernels for **HBM and PCIe bandwidth**. (idle 7.6 GB/s → busy 4.0 GB/s)
|
||||
2. **The connector's Python control plane gets GIL-starved** — mooncake's
|
||||
ZMQ handshake + transfer orchestration run on asyncio threads inside the
|
||||
engine process; when the engine's main thread is doing a long forward
|
||||
pass (e.g. a 100k-token prefill), those threads stall for *seconds*.
|
||||
|
||||
Both are **inherent to upstream vLLM 0.18.1 + mooncake** (reproduced on a
|
||||
clean fresh venv; the transfer path is byte-identical to upstream — our
|
||||
patches did not cause this), and both get **worse**, not better, with
|
||||
layerwise transfer. So the bandwidth gap is not a layerwise problem; it is a
|
||||
*transfer-on-a-busy-GPU* problem.
|
||||
|
||||
---
|
||||
|
||||
## 1. Evidence chain
|
||||
|
||||
Three independent measurements, all on dash0 (8×H100, Qwen3-Coder-30B-A3B,
|
||||
TP=1), Mooncake `kv_both`.
|
||||
|
||||
### 1a. Instrumented v3 trace replay — where does migration time go?
|
||||
|
||||
Run `outputs/b3_v3_fullbreak_20260528_0338/`. Instruments:
|
||||
`instrument_dst_migration.py` (dst scheduler lifecycle) +
|
||||
`instrument_mooncake.py` (connector internals: `send_blocks` RDMA,
|
||||
`receive_kv` window, `ready_wait`).
|
||||
|
||||
25 migrations fired over the trace. Dst-side migration overhead
|
||||
(`T_kv_pull` = scheduler marks `WAITING_FOR_REMOTE_KVS` → `finished_recving`):
|
||||
|
||||
| component | share | what it is |
|
||||
|---|---:|---|
|
||||
| RDMA-actual (`batch_transfer_sync_write`) | **55%** (55.2 s) | the real RDMA write |
|
||||
| dst control-plane gap | **45%** (45.4 s) | scheduler↔receiver_loop dispatch + completion propagation |
|
||||
| `ready_wait` (src KV not committed) | 0% | 25/25 already committed — **ruled out** |
|
||||
|
||||
- Pure RDMA aggregate rate: **2.03 GB/s** (vs MB2 idle 9.7 GB/s).
|
||||
- RDMA rate **collapses with transfer size**: <3 GiB → 4–9.5 GB/s,
|
||||
>5 GiB → 0.9–2.6 GB/s.
|
||||
- The control-plane gap is **bimodal**: median 0.04 s, but a handful of
|
||||
requests stall ~10 s. Those are small-KV transfers (0.18 s of actual RDMA)
|
||||
whose `T_kv_pull` is 8–11 s — i.e. the dst's `receiver_loop` thread was
|
||||
GIL-starved for ~10 s while the engine did a big forward pass.
|
||||
|
||||
> Earlier (pre-instrumentation) we wrongly attributed ~90% of migration
|
||||
> overhead to "dst scheduler queueing" by estimating transfer at clean wire
|
||||
> speed. With real instrumentation, dst *scheduler admission* is ~0
|
||||
> (`T_admission_post_kv` = 0.003 s); the time is the transfer phase (RDMA +
|
||||
> connector control plane), both degraded by instance busy-ness.
|
||||
|
||||
### 1b. MB6 controlled microbench — does busy-ness cause it?
|
||||
|
||||
`microbench/fresh_setup/mb6_transfer_under_load.py` + `run_mb6.sh`: 2
|
||||
instances, transfer a fixed-size KV (prefill on A → migrate to B) while
|
||||
holding *N* background decode streams on both. Sweep N.
|
||||
|
||||
Effective transfer bandwidth (65k-token KV ≈ 6 GiB), main venv:
|
||||
|
||||
| background load | 65k transfer | eff bandwidth |
|
||||
|---|---:|---:|
|
||||
| **0 (idle)** | 747 ms | **8.76 GB/s** |
|
||||
| 8 (4/instance) | 2423 ms | 4.53 GB/s |
|
||||
| **24 (12/instance)** | 2015 ms | **3.33 GB/s** |
|
||||
|
||||
Monotonic degradation with load. **The busy level (3.3 GB/s) matches the
|
||||
v3 trace's 3.3 GB/s median exactly** — because agentic instances run
|
||||
~10+ concurrent requests, i.e. the bg=24 regime.
|
||||
|
||||
Decomposing the 65k transfer into RDMA-actual vs control-plane:
|
||||
|
||||
| bg | RDMA rate | control-plane share |
|
||||
|---|---:|---:|
|
||||
| 0 (idle) | 7.56 GB/s | 13% |
|
||||
| 8 | 4.07 GB/s | 11% |
|
||||
| 24 (busy) | 3.97 GB/s | 15% |
|
||||
|
||||
In the clean microbench the **RDMA write itself is the dominant degrading
|
||||
term** (7.6 → 4.0 GB/s). The ~10 s control-plane stalls seen in the trace
|
||||
(1a) don't reproduce here because steady decode forward passes are short;
|
||||
they require the long (100k-token) prefills that the real trace has.
|
||||
|
||||
### 1c. Fresh-venv comparison — is it our patch?
|
||||
|
||||
Same MB6 sweep on `agentic-kv-fresh/.venv` (clean upstream-style 0.18.1):
|
||||
|
||||
| bg | 65k eff (fresh) | 65k eff (main/patched) |
|
||||
|---|---:|---:|
|
||||
| 0 | 8.73 GB/s | 8.76 GB/s |
|
||||
| 8 | 4.52 GB/s | 4.53 GB/s |
|
||||
| 24 | 3.27 GB/s | 3.33 GB/s |
|
||||
|
||||
**Identical within noise.** Plus a static check: the v3 transfer path
|
||||
(`send_kv_to_decode`, `_send_blocks`/`batch_transfer_sync_write`,
|
||||
`_build_transfer_params`) is **byte-identical** to pristine upstream 0.18.1
|
||||
(commit `445e491`); `receive_kv_from_single_worker` differs only by a 4-line
|
||||
error branch. Our mooncake commits (`a7df84b` direct-read,
|
||||
`ea51497` partial-prefill, `e3a1d70` read→push) only touch a *separate*
|
||||
`direct_read` path that v3 does **not** use (v3 requests carry no
|
||||
`direct_read` flag → normal push path).
|
||||
|
||||
→ **The slowdown is upstream/hardware-inherent, not introduced by us.**
|
||||
|
||||
---
|
||||
|
||||
## 2. Root cause
|
||||
|
||||
Migration in agentic serving transfers KV **between instances that are
|
||||
concurrently busy with compute** — by construction, since v3 migrates *away
|
||||
from* a busy host. On a busy instance:
|
||||
|
||||
- **HBM/PCIe contention (the dominant, irreducible part).** Mooncake's
|
||||
transfer is GPU-direct RDMA: the NIC DMAs KV straight out of / into GPU
|
||||
HBM. While the GPU runs attention+MLP kernels, those kernels saturate HBM
|
||||
bandwidth, so the NIC's RDMA gets a smaller slice. Effective transfer
|
||||
bandwidth roughly halves (7.6 → 4.0 GB/s at our load), and degrades
|
||||
further for large multi-segment transfers.
|
||||
- **Control-plane GIL starvation (secondary, bursty).** The connector runs
|
||||
its ZMQ handshake + `send_kv_to_decode`/`receive_kv` orchestration on
|
||||
asyncio threads (`sender_loop`/`receiver_loop`) *inside the engine
|
||||
process*. A long forward pass (100k-token prefill) holds the GIL for
|
||||
seconds, stalling those threads → multi-second dispatch gaps even when the
|
||||
actual transfer is 0.2 s.
|
||||
|
||||
MB2 measured 9.7 GB/s precisely because both endpoints were **idle**. The
|
||||
real-workload gap is the difference between "idle benchmark" and "transfer
|
||||
while the GPU is doing the day job."
|
||||
|
||||
---
|
||||
|
||||
## 3. Implication: layerwise is the wrong lever; migration's tax is largely irreducible
|
||||
|
||||
| lever | effect on the gap |
|
||||
|---|---|
|
||||
| **Model-level layerwise transfer** (push each layer's KV during prefill) | **Worse.** Prefill is the most HBM-intensive phase, so per-layer transfers contend *harder* for HBM (Cause 1); and they multiply the control-plane round-trips (Cause 2). |
|
||||
| **Control-plane fix** (move mooncake orchestration off the GIL-contended threads / separate process) | Addresses only the bursty ~10 s stalls (~15% in the clean case, up to ~45% of the trace tail). Does **not** touch the HBM-contention half. |
|
||||
| **Reduce bytes** (cache-aware target so less KV moves) | Helps linearly; v3 Mechanism B already does some. Orthogonal. |
|
||||
| **Migrate to/from idle instances** | Would restore ~10 GB/s — but defeats the purpose (we migrate *because* the host is busy). |
|
||||
|
||||
The dominant cost (RDMA contending with compute for HBM on busy instances)
|
||||
is a **hardware reality**, not a software bug we can patch away, and not
|
||||
something layerwise improves. This reinforces
|
||||
[UNIFIED_ABLATION.md](UNIFIED_ABLATION.md): the unified no-migration path
|
||||
(A+B'+RaceFix) remains the right default; migration's transfer tax is
|
||||
structural on a loaded agentic cluster.
|
||||
|
||||
---
|
||||
|
||||
## 4. Repro / artifacts
|
||||
|
||||
- Instrumented v3 breakdown: `outputs/b3_v3_fullbreak_20260528_0338/unified_v3/`
|
||||
(`transfer_decomp.txt`, `dst_migration_breakdown.{csv,png}`,
|
||||
`transfer_rootcause.png`)
|
||||
- MB6 main: `outputs/mb6_agentic-kv_20260528_0552/mb6_result.json`
|
||||
- MB6 fresh: `outputs/mb6_fresh_20260528_0559/mb6_result.json`
|
||||
- Instruments: `microbench/fresh_setup/instrument_dst_migration.py`,
|
||||
`microbench/fresh_setup/instrument_mooncake.py`
|
||||
- Microbench: `microbench/fresh_setup/mb6_transfer_under_load.py` +
|
||||
`run_mb6.sh` (`VENV=… bash run_mb6.sh`)
|
||||
- Analyzers: `analyze_dst_migration.py`, `analyze_transfer_decomp.py`
|
||||
|
||||
All instruments apply/revert cleanly via `--apply`/`--revert`; both venvs
|
||||
were restored after the runs.
|
||||
@@ -1,431 +0,0 @@
|
||||
# Unified routing ablation: A (tighter affinity) + B (decode-aware LMetric)
|
||||
|
||||
Goal: judge whether `unified` (cache-aware hybrid affinity + LMetric fallback)
|
||||
has enough headroom to surpass v3 migration-based routing on agentic
|
||||
workloads, without invoking PD-sep migration.
|
||||
|
||||
## Workload / baseline
|
||||
|
||||
- Trace: `w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions)
|
||||
- Hardware: 8 × H100 (dash0), Qwen3-Coder-30B-A3B, TP=1, max_model_len=200000
|
||||
- Trace replay through `cache_aware_proxy.py` with policy `unified`
|
||||
- `b3_replay_20260527_0114/unified/` reference
|
||||
|
||||
| Metric (ms) | baseline (`overload_factor=2.0`) |
|
||||
|---|---:|
|
||||
| TTFT p50 | 520 |
|
||||
| TTFT p90 | **8781** |
|
||||
| TTFT p99 | 47647 |
|
||||
| TPOT p90 | 17.8 |
|
||||
| E2E p90 | 19989 |
|
||||
| E2E p99 | 85841 |
|
||||
|
||||
Reference points we're trying to beat / match:
|
||||
- v3 fixed rotation (cache-blind picker): TTFT p90 = 10828
|
||||
- v3 + Mechanism B (cache-rich picker): TTFT p90 = 9711
|
||||
- All v3 variants are +10–23% worse than `unified` baseline.
|
||||
|
||||
## Tail-source diagnostic on baseline
|
||||
|
||||
Decision split, baseline unified:
|
||||
|
||||
| Decision | n | TTFT mean | TTFT p90 | TTFT p99 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| affinity | 852 | 3183 | 7011 | 47432 |
|
||||
| lmetric_fallback | 362 | 4285 | 12083 | 46036 |
|
||||
|
||||
Long-tail (>20s, n=65):
|
||||
- 40 / 65 came from `affinity` decisions
|
||||
- 25 / 65 came from `lmetric_fallback`
|
||||
|
||||
For the 40 slow `affinity` reqs:
|
||||
- only 12 / 40 were actually overloaded at decision time (`aff_num_req > avg_num_req`)
|
||||
- overload ratio at decision: mean=0.93, p50=0.87
|
||||
- **most slow affinity reqs looked fine when the picker stuck — load piled
|
||||
on after dispatch**.
|
||||
|
||||
This is a snapshot-based-routing limitation. Tightening
|
||||
`overload_factor` only helps the genuine cases above the new threshold —
|
||||
expected to be a 5-10% improvement at best.
|
||||
|
||||
---
|
||||
|
||||
## Direction A — tighten affinity overflow
|
||||
|
||||
**Hypothesis.** `overload_factor=2.0` lets the picker stick to affinity
|
||||
even when `affinity.num_req` is up to 2× the cluster average. Reducing to
|
||||
1.3 forces earlier overflow to LMetric fallback, escaping busy affinity
|
||||
hosts before the tail blows up.
|
||||
|
||||
**Change.** Single CLI flag: `--overload-factor 1.3`. No code change.
|
||||
|
||||
**Run.** `unified_of13_20260527_1532/unified/`.
|
||||
|
||||
### A vs baseline
|
||||
|
||||
| Metric (ms) | baseline (of=2.0) | A (of=1.3) | Δ |
|
||||
|---|---:|---:|---:|
|
||||
| TTFT p50 | 520 | 495 | −5% |
|
||||
| TTFT p90 | 8781 | 8730 | ≈0 |
|
||||
| TTFT p99 | 47647 | 43059 | −10% |
|
||||
| TPOT p50 | 7.9 | 8.0 | ≈0 |
|
||||
| TPOT p90 | 17.8 | **15.5** | **−13%** |
|
||||
| E2E p50 | 1761 | 1824 | +4% |
|
||||
| E2E p90 | 19989 | 18407 | −8% |
|
||||
| E2E p99 | 85841 | **71396** | **−17%** |
|
||||
|
||||
TTFT p90 is essentially unchanged but the **deeper tail (p99) and
|
||||
TPOT both improved meaningfully**. Net: A alone gives roughly −10% to
|
||||
−17% on the long tail without hurting medians.
|
||||
|
||||
### Decision split, A vs baseline
|
||||
|
||||
| Decision | baseline n / p90 | A n / p90 | Δ p90 |
|
||||
|---|---|---|---|
|
||||
| affinity | 852 / 7011 | 817 / **5817** | **−17%** ✅ |
|
||||
| lmetric_fallback | 362 / 12083 | 397 / **15360** | **+27%** ⚠️ |
|
||||
|
||||
The picker now sticks to affinity 35 fewer times. The remaining affinity
|
||||
decisions are higher-quality (no longer "barely-fitting" cases), so their
|
||||
p90 drops 17%.
|
||||
|
||||
But the 35 extra reqs that got pushed into fallback **got slower**:
|
||||
fallback p90 went from 12083 → 15360. The LMetric scorer is selecting a
|
||||
worse instance for them.
|
||||
|
||||
### Per-worker TTFT under A (of=1.3)
|
||||
|
||||
```
|
||||
port 8000: n= 94 mean=4424 p90=12290 port 8004: n=192 mean=2597 p90=6968
|
||||
port 8001: n= 135 mean=2779 p90= 5553 port 8005: n=202 mean=3102 p90=6113
|
||||
port 8002: n= 88 mean=5827 p90=15804 port 8006: n=136 mean=4006 p90=10899
|
||||
port 8003: n= 217 mean=2674 p90= 4598 port 8007: n=150 mean=3648 p90= 7025
|
||||
```
|
||||
|
||||
Compared to baseline (88..217 reqs/port), A redistributes more evenly
|
||||
(88..217 still but distribution is fatter in the middle). port 8002
|
||||
remains slow (p90 15.8s) — its cache pool seems to keep getting cold
|
||||
work routed there by LMetric.
|
||||
|
||||
### Why A alone isn't enough
|
||||
|
||||
LMetric scorer (`unified_hybrid` fallback path):
|
||||
|
||||
```python
|
||||
score = (pending_prefill_tokens + new_uncached_tokens) * num_requests
|
||||
```
|
||||
|
||||
This **ignores `ongoing_decode_tokens`** entirely. An instance with no
|
||||
pending prefill but 200k tokens currently in decode looks "ideal"
|
||||
(score=0×num_req=0) — yet a new request landing there waits behind
|
||||
slow decode iters caused by the large batch KV reads.
|
||||
|
||||
A pushes more requests into fallback, but fallback can't tell which
|
||||
instance is actually free. → Direction B is mandatory companion.
|
||||
|
||||
---
|
||||
|
||||
## Direction B — decode-aware LMetric
|
||||
|
||||
**Hypothesis.** Adding a decode-load penalty to the LMetric score lets
|
||||
fallback distinguish "no prefill queued but heavy decode running" from
|
||||
"truly idle". Should restore fallback p90 ≤ 12s baseline level.
|
||||
|
||||
**Change.**
|
||||
```python
|
||||
score = (pending_prefill + new + lmetric_decode_weight * ongoing_decode_tokens) * num_requests
|
||||
```
|
||||
- `lmetric_decode_weight=0.0` ⇒ original LMetric (control)
|
||||
- `lmetric_decode_weight=0.01` ⇒ first experiment (rationale: 1 decode token
|
||||
in batch costs ~0.01 prefill-token-equivalent in scheduler iter time
|
||||
on H100 + Qwen3-30B-A3B)
|
||||
|
||||
CLI: `--lmetric-decode-weight 0.01`. Setting in code:
|
||||
`cache_aware_proxy.py:Settings.lmetric_decode_weight`.
|
||||
|
||||
**Run.** `unified_of13_lmw001_20260527_1628/unified/`.
|
||||
|
||||
### A+B vs baseline / A
|
||||
|
||||
| Metric (ms) | baseline | A (of=1.3) | A+B (of=1.3, lmw=0.01) | Δ vs baseline |
|
||||
|---|---:|---:|---:|---:|
|
||||
| TTFT p50 | 520 | 495 | 514 | −1% |
|
||||
| **TTFT p90** | 8781 | 8730 | **8421** | **−4%** ✅ |
|
||||
| TTFT p99 | 47647 | 43059 | 44800 | −6% |
|
||||
| TPOT p50 | 7.9 | 8.0 | 7.9 | ≈0 |
|
||||
| TPOT p90 | 17.8 | 15.5 | 15.7 | −12% |
|
||||
| E2E p50 | 1761 | 1824 | 1870 | +6% |
|
||||
| E2E p90 | 19989 | 18407 | **21064** | **+5%** ⚠️ |
|
||||
| E2E p99 | 85841 | 71396 | **64344** | **−25%** ✅ |
|
||||
|
||||
Long-tail counts:
|
||||
|
||||
```
|
||||
thresh baseline A A+B v3 MechB
|
||||
> 5000ms 170 173 170 177
|
||||
> 10000ms 105 109 109 119
|
||||
> 20000ms 65 64 59 78
|
||||
> 30000ms 41 40 37 50
|
||||
> 50000ms 8 5 6 14
|
||||
```
|
||||
|
||||
A+B is best on every long-tail-count threshold ≤30s, marginal worse at 50s.
|
||||
|
||||
### Decision split (A+B vs A)
|
||||
|
||||
| Decision | A (of=1.3) | A+B | Note |
|
||||
|---|---|---|---|
|
||||
| affinity p90 | 5817 | 5836 | ≈ same |
|
||||
| fallback p90 | **15360** | **13501** | B recovered some of A's fallback regression |
|
||||
|
||||
B partially fixed fallback's selection (−12% on fallback p90 vs A alone),
|
||||
but still worse than baseline (12083).
|
||||
|
||||
### Per-worker TTFT (A+B)
|
||||
|
||||
```
|
||||
port 8000: n=134 mean=3495 p90=10967 port 8004: n=136 mean=3102 p90= 7906
|
||||
port 8001: n=143 mean=2981 p90=10189 port 8005: n=179 mean=1624 p90= 2735
|
||||
port 8002: n=221 mean=2355 p90= 3502 port 8006: n=137 mean=5356 p90= 9628
|
||||
port 8003: n=146 mean=3932 p90=10729 port 8007: n=118 mean=5210 p90=26798 ← new hotspot
|
||||
```
|
||||
|
||||
A+B trades the baseline's 8002 hotspot (p90=35s) for a new 8007 hotspot
|
||||
(p90=26.8s). Lower amplitude but hotspot survives.
|
||||
|
||||
### Why 8007 became a hotspot under A+B — **found a bug in B**
|
||||
|
||||
8007 in A+B: 118 reqs, **53% affinity / 47% fallback** (vs other ports
|
||||
60–77% affinity), **cache_hit_mean=50.5% (lowest)**.
|
||||
|
||||
Top-10 slowest at 8007: all are big-prompt (100k+ tokens) fallback decisions
|
||||
with `cached_tokens=0` (cold prefill). LMetric is pushing many cold-prefill
|
||||
fallbacks to 8007.
|
||||
|
||||
Looking at the B formula:
|
||||
|
||||
```python
|
||||
decode_pen = lmetric_decode_weight * ongoing_decode_tokens
|
||||
score = (pending_prefill + new + decode_pen) * num_requests # ← BUG
|
||||
```
|
||||
|
||||
When `num_requests = 0`, the entire score (including decode penalty) zeros
|
||||
out. So an idle-but-decoding host (num_req=0 because its last prefill
|
||||
finished but decode is still running) looks like score=0, beating every
|
||||
busy host.
|
||||
|
||||
**Fix (B'):** multiply by `max(num_requests, 1)`:
|
||||
|
||||
```python
|
||||
score = (pending_prefill + new + decode_pen) * max(num_requests, 1)
|
||||
```
|
||||
|
||||
Now idle hosts with high decode load get score = decode_pen × 1 = real
|
||||
nonzero penalty, beating zero-load hosts only when decode is small.
|
||||
|
||||
### A+B' — re-run with the fix
|
||||
|
||||
**Run.** `unified_of13_lmw001_v2_20260527_1724/unified/`.
|
||||
|
||||
| Metric (ms) | baseline | A+B (BUG) | A+B' (fix) | Δ vs baseline |
|
||||
|---|---:|---:|---:|---:|
|
||||
| TTFT p50 | 520 | 514 | **485** | −7% |
|
||||
| **TTFT p90** | 8781 | 8421 | **8287** | **−5.6%** ✅ |
|
||||
| TTFT p99 | 47647 | 44800 | **41876** | **−12%** ✅ |
|
||||
| TPOT p90 | 17.8 | 15.7 | 17.5 | −2% |
|
||||
| E2E p90 | 19989 | 21064 | 20625 | +3% |
|
||||
| E2E p99 | 85841 | 64344 | 77827 | −9% |
|
||||
|
||||
A+B' **best of all variants on TTFT p90 (8287) and TTFT p99 (41876)**.
|
||||
Long-tail counts (>30s, >50s) also best across variants.
|
||||
|
||||
vs v3 reference points:
|
||||
| | TTFT p90 | TPOT p90 | E2E p99 |
|
||||
|---|---:|---:|---:|
|
||||
| **A+B'** | **8287** | 17.5 | 77827 |
|
||||
| v3 fixed (cache-blind) | 10828 | 21.0 | 47610 |
|
||||
| v3 + Mech B | 9711 | 18.3 | 84492 |
|
||||
|
||||
A+B' **beats v3 Mech B by 15% TTFT p90** with no migration overhead.
|
||||
|
||||
### Per-worker (A+B' fixed)
|
||||
|
||||
```
|
||||
8000: n=158 p90= 5688 8004: n=189 p90= 4249
|
||||
8001: n=159 p90= 7323 8005: n=116 p90=14598
|
||||
8002: n=114 p90= 8726 8006: n=180 p90= 6198
|
||||
8003: n=173 p90= 6715 8007: n=125 p90=22242 ← still hot
|
||||
```
|
||||
|
||||
A+B' redistributed load more evenly (114..189) but **8007 still has p90=22s**.
|
||||
|
||||
### 8007 deep-dive in A+B'
|
||||
|
||||
```
|
||||
8007: n=125, affinity=69 (55%), fallback=56 (45%), cache_hit_mean=lowest
|
||||
```
|
||||
|
||||
Top-15 slow at 8007:
|
||||
- 7 of them are session **1313181** turns 9–14 (130k+ tokens each, agentic
|
||||
long context, ~50% cache hit)
|
||||
- Several others are cold-start turn-1 of large-prompt sessions
|
||||
- First two slow reqs arrived **0.7 s apart** — strong hint of concurrent
|
||||
picker race
|
||||
|
||||
### Iteration 3: race-condition fix
|
||||
|
||||
**Diagnosis.** In `_handle_combined`:
|
||||
|
||||
```python
|
||||
chosen, best_idx, decision = pick_instance_unified_hybrid(...) # sync
|
||||
# ... sync breakdown updates ...
|
||||
return await _handle_local_request(...) # ← await yields here
|
||||
# THEN reservation happens
|
||||
```
|
||||
|
||||
`return await async_func(...)` evaluates the async call (creates coroutine)
|
||||
and yields to the event loop **before** the coroutine body executes. The
|
||||
reservation (`chosen.pending_prefill_tokens += new`, etc.) lives at the top
|
||||
of `_handle_local_request`, so between the picker and the reservation there
|
||||
is a **window where another coroutine can run and re-pick the same instance**.
|
||||
|
||||
When two big-prompt reqs arrive within milliseconds, both run pick →
|
||||
both pick the "free" 8007 → both yield → both reserve. Result: 8007 gets
|
||||
back-to-back 130k-token cold prefills, each waiting for the other.
|
||||
|
||||
**Fix.** Move the reservation **before** the await, inside `_handle_combined`:
|
||||
|
||||
```python
|
||||
# Race fix: reserve atomically with pick, before any await.
|
||||
chosen.ongoing_tokens += input_length
|
||||
chosen.pending_prefill_tokens += estimated_new
|
||||
chosen.num_requests += 1
|
||||
return await _handle_local_request(..., _pre_reserved=True)
|
||||
```
|
||||
|
||||
`_handle_local_request` skips its own reservation when `_pre_reserved=True`.
|
||||
PD-sep paths are unaffected (they have their own reservation).
|
||||
|
||||
**Run.** Pending — `unified_of13_lmw001_racefix_*`. Hypothesis: 8007 p90
|
||||
drops to within ±3s of cluster median, since concurrent picks for the
|
||||
same "free" instance no longer happen.
|
||||
|
||||
---
|
||||
|
||||
## A+B'+RaceFix — results
|
||||
|
||||
**Run.** `unified_of13_lmw001_racefix_20260527_1821/unified/`.
|
||||
|
||||
| Metric (ms) | baseline | A+B' | A+B'+RF | Δ vs baseline |
|
||||
|---|---:|---:|---:|---:|
|
||||
| TTFT p50 | 520 | 485 | **478** | −8% |
|
||||
| **TTFT p90** | 8781 | 8287 | **7770** | **−11.5%** ✅ |
|
||||
| TTFT p99 | 47647 | 41876 | **42447** | −11% |
|
||||
| TPOT p90 | 17.8 | 17.5 | 18.0 | +1% |
|
||||
| E2E p90 | 19989 | 20625 | **18418** | −8% |
|
||||
| E2E p99 | 85841 | 77827 | **71227** | −17% |
|
||||
|
||||
vs v3 reference:
|
||||
- **A+B'+RF TTFT p90 = 7770ms, vs v3 Mech B 9711ms → −20%** ✅
|
||||
|
||||
Long-tail counts (best across all variants):
|
||||
```
|
||||
> 5s: 170 → 158 > 30s: 41 → 33
|
||||
>10s: 105 → 103 > 50s: 8 → 4
|
||||
>20s: 65 → 57 >100s: 0 → 0
|
||||
```
|
||||
|
||||
### Decision split — race fix mainly helped affinity
|
||||
|
||||
| Decision | baseline | A+B'+RF |
|
||||
|---|---:|---:|
|
||||
| affinity p90 | 7011 | **5042** ✅ (−28%) |
|
||||
| fallback p90 | 12083 | 13944 (+15%) |
|
||||
|
||||
The race-condition was hurting affinity decisions the most. When two
|
||||
concurrent reqs both stuck to a "free-looking" affinity instance, they
|
||||
piled up and inflated affinity's tail. Fix removed this collision.
|
||||
|
||||
### Per-worker
|
||||
|
||||
```
|
||||
8000: n=86 p90=11541 8004: n=150 p90=11906
|
||||
8001: n=186 p90= 8307 8005: n=109 p90= 4798
|
||||
8002: n=105 p90=14540 8006: n=183 p90= 6258
|
||||
8003: n=264 p90= 3079 8007: n=131 p90=21850 ← still hot
|
||||
8000 spread now 86..264 — race fix did disperse routing
|
||||
```
|
||||
|
||||
### 8007 still hot — but it's **workload-inherent, not a routing bug**
|
||||
|
||||
Top sessions on 8007:
|
||||
```
|
||||
session 1279412: n=22 mean= 2208 max=18985 decisions: 91% affinity
|
||||
session 1313181: n=17 mean=17399 max=49089 decisions: 65% affinity
|
||||
session 1262354: n=15 mean= 622 max= 2325 decisions: 87% affinity
|
||||
session 1342921: n= 7 mean=17817 max=55589 decisions: 86% affinity
|
||||
session 1260327: n= 8 mean= 1636 max= 5382 decisions: 75% affinity
|
||||
session 1268831: n= 5 mean= 1443 max= 2673 decisions: 80% affinity
|
||||
```
|
||||
|
||||
Sessions 1313181 and 1342921 are **long agentic contexts**: 100k–130k tokens
|
||||
per turn with ~50% cache hit (i.e. 50k new tokens prefill per turn). Even
|
||||
on a perfectly load-balanced instance, each turn is 7–15s of pure compute.
|
||||
|
||||
Forcing these sessions to spread across instances would mean **cold prefill
|
||||
every turn (0% cache hit)** → each turn becomes 20–30s instead of 7–15s.
|
||||
Spreading is **net-negative**.
|
||||
|
||||
→ The 8007 p90=22s is the floor imposed by these sessions' structure,
|
||||
not by routing policy. Unified is at its ceiling for this workload.
|
||||
|
||||
---
|
||||
|
||||
## Final ranking and take-aways
|
||||
|
||||
| Policy | TTFT p90 (ms) | Δ vs baseline | Notes |
|
||||
|---|---:|---:|---|
|
||||
| baseline unified (of=2.0) | 8781 | — | reference |
|
||||
| A (of=1.3) | 8730 | ≈0 | affinity p90 -17%, fallback p90 +27% |
|
||||
| A+B (of=1.3, lmw=0.01, BUG) | 8421 | −4% | 8007 hotspot from `*num_req` zeroing bug |
|
||||
| A+B' (formula fix) | 8287 | −5.6% | Bug fixed, still 8007 mild hotspot |
|
||||
| **A+B'+RaceFix** | **7770** | **−11.5%** ✅ | **Best unified variant** |
|
||||
| v3 fixed | 10828 | +23% | PD-sep migration, cache-blind picker |
|
||||
| v3 + Mech B | 9711 | +11% | PD-sep + cache-rich target picker |
|
||||
|
||||
### Conclusions
|
||||
|
||||
1. **Unified path beats v3 PD-sep on this workload by 20%+ TTFT p90.**
|
||||
PD-sep migration's fixed cost (src prefill + dst first-token waiting on
|
||||
loaded scheduler) outweighs any decode-time savings for short-output
|
||||
agentic turns.
|
||||
|
||||
2. **Three orthogonal fixes compound for a 11.5% TTFT p90 win:**
|
||||
- A (`overload_factor=1.3`): tighter affinity overflow → −0.6% but
|
||||
much cleaner affinity decisions (p90 -17%)
|
||||
- B' (`lmetric_decode_weight=0.01` with `max(num_req,1)`): decode-aware
|
||||
fallback → −3.5%
|
||||
- RaceFix (atomic reserve before await): kills concurrent-pick
|
||||
collisions → −5.6%
|
||||
|
||||
3. **Race condition was the biggest single hidden bug.** `return await
|
||||
async_func(...)` yields to the event loop **before** the body of
|
||||
`async_func` runs, so reservations done in the body don't take effect
|
||||
in time to deter concurrent picks. This affects ANY async dispatch
|
||||
with separate pick/reserve steps — worth checking other routing
|
||||
policies.
|
||||
|
||||
4. **8007 p90=22s is workload-inherent.** Sessions with 100k+ token turns
|
||||
at 50% cache hit cannot finish faster than 7–15s per turn regardless
|
||||
of routing. Forcing spread would hurt rather than help.
|
||||
|
||||
5. **Migration (v3) is not necessary** when unified routing is tuned
|
||||
well. Save the PD-sep mechanism for cases where it can be proven
|
||||
net-positive (e.g. very-long-output sessions on extremely overloaded
|
||||
prefill hosts) and use unified A+B'+RaceFix as the default.
|
||||
|
||||
---
|
||||
|
||||
## Direction A+B — run pending
|
||||
|
||||
(Will be filled when `unified_of13_lmw001_*/unified/` finishes.)
|
||||
@@ -1,169 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""B3 5-policy re-test analyser.
|
||||
|
||||
Compute TTFT/TPOT/E2E mean/p50/p90/p99 for each policy from
|
||||
metrics.jsonl, compare against the historical b3_policy_comparison.json
|
||||
that drives fig_b3_latency_bars.png, and emit a side-by-side table
|
||||
plus a new figure with the same layout as the original.
|
||||
|
||||
Usage:
|
||||
python analyze_b3_replay.py --root <outroot> [--old-data <path>] [--figure <path>]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
POLICIES = ["lmetric", "load_only", "sticky", "unified", "unified_v2"]
|
||||
|
||||
|
||||
def pct(xs, p):
|
||||
if not xs:
|
||||
return None
|
||||
xs = sorted(xs)
|
||||
k = max(0, min(len(xs) - 1, int(p / 100.0 * (len(xs) - 1))))
|
||||
return xs[k]
|
||||
|
||||
|
||||
def summarise(path):
|
||||
rows = [json.loads(l) for l in open(path) if l.strip()]
|
||||
ok = [r for r in rows if not r.get("error")]
|
||||
ttft = [r["ttft_s"] * 1000 for r in ok if r.get("ttft_s") is not None]
|
||||
tpot = [r["tpot_s"] * 1000 for r in ok if r.get("tpot_s")]
|
||||
e2e = [r["latency_s"] * 1000 for r in ok if r.get("latency_s") is not None]
|
||||
return {
|
||||
"n_total": len(rows),
|
||||
"n_ok": len(ok),
|
||||
"ttft_mean_ms": statistics.mean(ttft) if ttft else None,
|
||||
"ttft_p50_ms": pct(ttft, 50),
|
||||
"ttft_p90_ms": pct(ttft, 90),
|
||||
"ttft_p99_ms": pct(ttft, 99),
|
||||
"tpot_mean_ms": statistics.mean(tpot) if tpot else None,
|
||||
"tpot_p50_ms": pct(tpot, 50),
|
||||
"tpot_p90_ms": pct(tpot, 90),
|
||||
"tpot_p99_ms": pct(tpot, 99),
|
||||
"e2e_mean_ms": statistics.mean(e2e) if e2e else None,
|
||||
"e2e_p50_ms": pct(e2e, 50),
|
||||
"e2e_p90_ms": pct(e2e, 90),
|
||||
"e2e_p99_ms": pct(e2e, 99),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--root", type=Path, required=True)
|
||||
ap.add_argument("--old-data", type=Path,
|
||||
default=Path("analysis/characterization/window_1_results/b3_policy_comparison.json"))
|
||||
ap.add_argument("--figure", type=Path, default=None)
|
||||
args = ap.parse_args()
|
||||
|
||||
new = {}
|
||||
for p in POLICIES:
|
||||
path = args.root / p / "metrics.jsonl"
|
||||
if not path.exists():
|
||||
print(f"MISSING: {path}")
|
||||
continue
|
||||
new[p] = summarise(path)
|
||||
|
||||
old = {}
|
||||
if args.old_data.exists():
|
||||
d = json.load(open(args.old_data))
|
||||
for r in d.get("rows", []):
|
||||
old[r["policy"]] = {
|
||||
"ttft_p50_ms": r["ttft_p50_s"] * 1000,
|
||||
"ttft_p90_ms": r["ttft_p90_s"] * 1000,
|
||||
"ttft_p99_ms": r["ttft_p99_s"] * 1000,
|
||||
"tpot_p90_ms": r["tpot_p90_s"] * 1000,
|
||||
"e2e_p90_ms": r.get("e2e_p90_s", 0) * 1000,
|
||||
}
|
||||
|
||||
def fmt(v): return f"{v:.0f}" if v is not None else "-"
|
||||
def pctd(a, b):
|
||||
if a is None or b is None or a == 0: return "-"
|
||||
return f"{(b/a-1)*100:+.1f}%"
|
||||
|
||||
# Headline table
|
||||
print(f"\n# NEW: today's re-test")
|
||||
print(f"{'policy':<14}{'n_ok':>6}{'TTFTp50':>10}{'TTFTp90':>10}{'TTFTp99':>10}{'TPOTp90':>10}{'E2Ep90':>10}")
|
||||
print("-" * 70)
|
||||
for p in POLICIES:
|
||||
if p not in new: continue
|
||||
r = new[p]
|
||||
print(f"{p:<14}{r['n_ok']:>6}{fmt(r['ttft_p50_ms']):>9}ms{fmt(r['ttft_p90_ms']):>9}ms{fmt(r['ttft_p99_ms']):>9}ms{fmt(r['tpot_p90_ms']):>9}ms{fmt(r['e2e_p90_ms']):>9}ms")
|
||||
|
||||
print(f"\n# OLD: window_1_results/b3_policy_comparison.json")
|
||||
print(f"{'policy':<14}{'TTFTp50':>10}{'TTFTp90':>10}{'TTFTp99':>10}{'TPOTp90':>10}{'E2Ep90':>10}")
|
||||
print("-" * 60)
|
||||
for p in POLICIES:
|
||||
if p not in old: continue
|
||||
r = old[p]
|
||||
print(f"{p:<14}{fmt(r['ttft_p50_ms']):>9}ms{fmt(r['ttft_p90_ms']):>9}ms{fmt(r['ttft_p99_ms']):>9}ms{fmt(r['tpot_p90_ms']):>9}ms{fmt(r['e2e_p90_ms']):>9}ms")
|
||||
|
||||
print(f"\n# DRIFT: today vs old (same policy)")
|
||||
print(f"{'policy':<14}{'ΔTTFTp50':>10}{'ΔTTFTp90':>10}{'ΔTTFTp99':>10}{'ΔTPOTp90':>10}{'ΔE2Ep90':>10}")
|
||||
print("-" * 60)
|
||||
for p in POLICIES:
|
||||
if p not in new or p not in old: continue
|
||||
n, o = new[p], old[p]
|
||||
print(f"{p:<14}{pctd(o['ttft_p50_ms'], n['ttft_p50_ms']):>10}"
|
||||
f"{pctd(o['ttft_p90_ms'], n['ttft_p90_ms']):>10}"
|
||||
f"{pctd(o['ttft_p99_ms'], n['ttft_p99_ms']):>10}"
|
||||
f"{pctd(o['tpot_p90_ms'], n['tpot_p90_ms']):>10}"
|
||||
f"{pctd(o['e2e_p90_ms'], n['e2e_p90_ms']):>10}")
|
||||
|
||||
# Relative ordering check
|
||||
def ranks(values_dict, key):
|
||||
items = [(p, r[key]) for p, r in values_dict.items() if r.get(key)]
|
||||
items.sort(key=lambda x: x[1])
|
||||
return [p for p, _ in items]
|
||||
|
||||
print(f"\n# TTFT p90 ranking (best → worst)")
|
||||
for label, src in [("OLD", old), ("NEW", new)]:
|
||||
if src:
|
||||
order = ranks(src, "ttft_p90_ms")
|
||||
print(f" {label}: {' < '.join(order)}")
|
||||
|
||||
out = {"new": new, "old": old}
|
||||
out_path = args.root / "b3_replay_summary.json"
|
||||
out_path.write_text(json.dumps(out, indent=2))
|
||||
print(f"\nWrote {out_path}")
|
||||
|
||||
# Bar plot (matplotlib)
|
||||
if not args.figure:
|
||||
args.figure = args.root / "fig_b3_latency_bars_new.png"
|
||||
try:
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
pols = [p for p in POLICIES if p in new]
|
||||
metrics = [("TTFT p90 (s)", "ttft_p90_ms", 1000),
|
||||
("TPOT p90 (ms)", "tpot_p90_ms", 1),
|
||||
("E2E p90 (s)", "e2e_p90_ms", 1000)]
|
||||
colors = {"lmetric": "tab:blue", "load_only": "tab:orange",
|
||||
"sticky": "tab:green", "unified": "tab:red",
|
||||
"unified_v2": "tab:purple"}
|
||||
fig, axes = plt.subplots(1, 3, figsize=(14, 4.5))
|
||||
for ax, (label, key, div) in zip(axes, metrics):
|
||||
vals = [new[p][key] / div for p in pols]
|
||||
bars = ax.bar(pols, vals,
|
||||
color=[colors.get(p, "gray") for p in pols],
|
||||
edgecolor="black", linewidth=0.5)
|
||||
ax.set_title(label)
|
||||
ax.tick_params(axis="x", rotation=20)
|
||||
for b, v in zip(bars, vals):
|
||||
ax.text(b.get_x() + b.get_width() / 2, v, f"{v:.1f}",
|
||||
ha="center", va="bottom", fontsize=9)
|
||||
ax.grid(alpha=0.3, axis="y")
|
||||
fig.suptitle(f"B3 5-policy re-test ({args.root.name})")
|
||||
fig.tight_layout()
|
||||
fig.savefig(args.figure, dpi=120)
|
||||
print(f"Wrote {args.figure}")
|
||||
except Exception as e:
|
||||
print(f"(figure skipped: {e})")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user