Gahow Wang
05592e6adc
Systematic study of prefill-decode disaggregation for agentic LLM workloads using production GLM-5.1 coder trace (2.1M requests, 71B input tokens). Key findings: - Cache-aware routing improves TPOT p90 by 15% and APC from 20.8% to 44.7% without PD separation, matching PD-Sep's decode isolation benefit - PD separation adds +72% TTFT overhead (KV transfer) with no TPOT gain when using the same cache-aware scheduler - Prefill remains compute-bound even at 95% KV cache reuse (AI >1000x vs decode AI <2), but absolute FLOPs drop 71% from cache hits - For agentic MoE workloads, cache-aware routing > PD separation Infrastructure: - Trace sampler preserving session structure + hash_ids for prefix sharing - Async trace replayer with streaming TTFT/TPOT/E2E measurement - Unified cache-aware + token-level load-balanced global scheduler proxy supporting both PD-colocated and PD-disaggregated (Mooncake/RDMA) modes - vLLM 0.18.1 scheduler patch for KV transfer abort race condition - Roofline analysis tool for prefill/decode compute characterization Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
290 lines
11 KiB
Markdown
290 lines
11 KiB
Markdown
# PD 分离在 Agentic Workload 下的系统分析
|
||
|
||
## 1. Trace 特征 (GLM-5.1 Agentic Coder, 2h, 2.1M requests)
|
||
|
||
```
|
||
Total requests: 2,114,220
|
||
Input tokens: 71.1B (avg 33.6k/req, p50=20k, p90=88k)
|
||
Output tokens: 940M (avg 445/req, p50=80, p90=811)
|
||
I/O ratio: 75.6x (aggregate), 217.8x (per-req median)
|
||
Prefill share: 98% of total tokens
|
||
Sessions: 1.3M (90% single-turn, 9% multi-turn)
|
||
```
|
||
|
||
**与传统 chatbot workload 的根本区别：**
|
||
|
||
| 特征 | Traditional Chatbot | Agentic Coder (GLM-5.1) |
|
||
|------|-------------------|------------------------|
|
||
| I/O ratio | 1-10x | **75.6x** |
|
||
| Input p50 | 500-2000 tokens | **20,030 tokens** |
|
||
| Output p50 | 200-500 tokens | **80 tokens** |
|
||
| Prefill token share | 50-80% | **98%** |
|
||
| >32k input | <5% | **38%** |
|
||
| Multi-turn | 50-80% | **9%** |
|
||
|
||
**KV Cache 复用特征：**
|
||
|
||
```
|
||
Unique hash blocks: 20,650,883
|
||
Shared blocks (ref>1): 9,749,379 (47%)
|
||
Highly shared (ref>10): 2,428,160
|
||
Intra-session reuse: 57%
|
||
Top-10 blocks ref count: 64,754 (system prompt blocks)
|
||
Theoretical cache hit: 71% (infinite cache, first 100k requests)
|
||
```
|
||
|
||
**Input length 分布与 token 占比：**
|
||
|
||
```
|
||
<1k: 202,396 reqs ( 9%) 89M tokens ( 0%)
|
||
1-8k: 380,009 reqs (17%) 1.6B tokens ( 2%)
|
||
8-32k: 720,871 reqs (34%) 12.7B tokens (17%)
|
||
32-65k: 405,371 reqs (19%) 19.4B tokens (27%)
|
||
65-131k: 394,014 reqs (18%) 35.7B tokens (50%)
|
||
>131k: 11,559 reqs ( 0%) 1.6B tokens ( 2%)
|
||
```
|
||
|
||
50% 的 token 计算量来自 65-131k 的长 context 请求。
|
||
|
||
## 2. DistServe 等 PD 分离的核心假设
|
||
|
||
DistServe (OSDI'24), Splitwise, TetriInfer 等 PD 分离工作基于以下假设：
|
||
|
||
### 假设 A: Prefill 和 Decode 有不同的计算特征
|
||
- **Prefill**: compute-bound, 高 GPU 利用率, batch 越大越好
|
||
- **Decode**: memory-bandwidth-bound, 低 GPU 利用率, latency-sensitive
|
||
|
||
**在 agentic workload 中的验证**: ✅ 成立，但需要细化
|
||
|
||
Roofline 分析显示（详见 Section 5）：
|
||
|
||
```
|
||
Arithmetic Intensity (FLOP/byte)
|
||
Decode: 1.0 - 1.9 (memory-bound, 始终远低于 ridge point)
|
||
Prefill 0% reuse: 23,000-72,000 (strongly compute-bound)
|
||
Prefill 70% reuse: 10,000-42,000 (仍然 compute-bound!)
|
||
Prefill 95% reuse: 1,900-10,800 (仍然 compute-bound!)
|
||
Ridge point (H20): 37
|
||
```
|
||
|
||
**即使 95% KV cache reuse，prefill 仍然是 compute-bound。** 但绝对计算量大幅减少。
|
||
|
||
### 假设 B: PD co-location 导致互相干扰
|
||
- Prefill 的大 batch 计算会抢占 GPU 资源，导致 decode 的 TPOT 升高
|
||
- Decode 的持续小计算会占用 GPU 调度槽位，影响 prefill 吞吐
|
||
|
||
**在 agentic workload 中的验证**: ⚠️ 干扰存在，但 **可被 cache-aware routing 消除**
|
||
|
||
```
|
||
同一 cache-aware scheduler, TP=1, 8 GPU:
|
||
Combined TP=1 DP=8: TPOT p90 = 0.073s
|
||
PD-Sep TP=1 4P+4D: TPOT p90 = 0.074s
|
||
→ 差异 <2%, 不显著
|
||
```
|
||
|
||
对比 round-robin routing:
|
||
```
|
||
Combined TP=1 DP=8 (RR): TPOT p90 = 0.086s
|
||
Combined TP=1 DP=8 (cache-aware): TPOT p90 = 0.073s → -15%
|
||
→ routing 改善 > PD 分离改善
|
||
```
|
||
|
||
**原因**: cache-aware routing 让 high-cache-hit 的请求集中到特定 instance，
|
||
每个 instance 的实际 prefill 新 token 数大幅减少（71% 被 cache），
|
||
prefill-decode 干扰因 prefill 工作量降低而自然缓解。
|
||
|
||
### 假设 C: KV Cache 传输开销可以忽略
|
||
- DistServe 假设 P→D 的 KV 传输延迟远小于 prefill 计算时间
|
||
- 在 InfiniBand/NVLink 等高带宽互联下成立
|
||
|
||
**在 agentic workload 中的验证**: ❌ 不成立
|
||
|
||
```
|
||
PD-Sep TTFT p50 = 1.261s vs Combined TTFT p50 = 0.731s (+72%)
|
||
```
|
||
|
||
原因：
|
||
1. Agentic workload 的 input 极长（p50=20k, p90=88k tokens），KV cache 很大
|
||
2. 单请求 KV cache = 20k tokens × 48 layers × 2(K+V) × 512 bytes ≈ 1GB
|
||
3. 更重要的是 await-prefill 链路的串行延迟：proxy → prefill → KV transfer → decode → first token
|
||
|
||
### 假设 D: 专用 prefill 节点可以提高 prefill 吞吐
|
||
- Prefill 节点不做 decode，GPU 利用率更高
|
||
- 可以用更大的 batch size
|
||
|
||
**在 agentic workload 中的验证**: ⚠️ 收益被 cache 稀释
|
||
|
||
```
|
||
理论 prefix cache hit (infinite cache): 71% of input tokens
|
||
实际 APC (Combined, cache-aware, 8 inst): 44.7%
|
||
```
|
||
|
||
71% cache hit → 只有 29% 的 input tokens 需要实际 prefill compute。
|
||
Nominal avg input 33.6k → Actual avg new prefill ~9.7k tokens。
|
||
专用 prefill 的 GPU 利用率优势因 prefill 工作量降低而缩小。
|
||
|
||
## 3. Roofline 分析：Prefill 在高 Cache Reuse 下的计算/访存特性
|
||
|
||
### 3.1 模型计算结构
|
||
|
||
```
|
||
Qwen3-Coder-30B-A3B (MoE 128E top-8):
|
||
48 layers, hidden=2048, heads=32, kv_heads=4 (GQA), head_dim=128
|
||
FFN: 6144 intermediate per expert, 8 experts active per token
|
||
Active params per token: ~3B
|
||
|
||
H20 GPU: 148 TFLOPS (BF16), 4.0 TB/s HBM → Ridge point: 37 FLOP/byte
|
||
```
|
||
|
||
### 3.2 Decode 永远 memory-bound
|
||
|
||
```
|
||
SeqLen FLOP Bytes AI (F/B) Bound
|
||
1,000 3.04e+10 3.01e+10 1.0 MEMORY
|
||
16,000 3.63e+10 3.16e+10 1.1 MEMORY
|
||
64,000 5.52e+10 3.63e+10 1.5 MEMORY
|
||
128,000 8.03e+10 4.26e+10 1.9 MEMORY
|
||
```
|
||
|
||
Decode 的 AI 始终 < 2，远低于 ridge point (37)。每个 decode step 只处理 1 个 token，
|
||
计算量极小，瓶颈在于读取模型权重和全量 KV cache。
|
||
|
||
### 3.3 Prefill 即使 95% reuse 仍然 compute-bound
|
||
|
||
```
|
||
SeqLen Reuse% NewTok AI (F/B) Bound vs Decode
|
||
32,000 0% 32,000 23,368 COMPUTE 18,190x
|
||
32,000 50% 16,000 14,899 COMPUTE 11,597x
|
||
32,000 70% 9,600 10,045 COMPUTE 7,819x
|
||
32,000 90% 3,200 3,821 COMPUTE 2,974x
|
||
32,000 95% 1,600 1,980 COMPUTE 1,542x
|
||
|
||
64,000 0% 64,000 40,758 COMPUTE 26,813x
|
||
64,000 70% 19,200 20,610 COMPUTE 13,559x
|
||
64,000 90% 6,400 8,544 COMPUTE 5,621x
|
||
64,000 95% 3,200 4,549 COMPUTE 2,993x
|
||
```
|
||
|
||
### 3.4 为什么高 reuse 不改变 compute-bound 性质
|
||
|
||
KV cache reuse 减少的：
|
||
- K/V projection 计算（只算 new tokens）
|
||
- KV 写入（只写 new tokens）
|
||
|
||
KV cache reuse **不减少**的：
|
||
- **Q×K^T attention**: 每个 new Q 都要和全部 seq_len 个 KV 做 attention
|
||
```
|
||
FLOPs = new_tokens × seq_len × head_dim × num_heads × 2 × num_layers
|
||
```
|
||
At 95% reuse, 32k seq: 1600 × 32000 × 128 × 32 × 2 × 48 ≈ 2×10^13
|
||
这个二次项在长 context 下主导总计算量
|
||
|
||
- **MoE FFN**: 每个 new token 激活 8 experts
|
||
```
|
||
FLOPs = new_tokens × 3 × D × D_ffn × 2 × K_experts × num_layers
|
||
```
|
||
|
||
**Prefill 只在接近 100% reuse (< 10 new tokens) 时才变成 memory-bound。**
|
||
|
||
### 3.5 Prefill 什么时候变 memory-bound
|
||
|
||
```
|
||
SeqLen=32,000: new_tokens ≈ 5-10 时 → AI ≈ 37 (ridge point)
|
||
SeqLen=64,000: new_tokens ≈ 5-10 时 → AI ≈ 37
|
||
```
|
||
|
||
在实际 agentic trace 中：
|
||
```
|
||
Compute-bound prefills: 961 (96%)
|
||
Memory-bound prefills: 37 (3%) ← 近 100% reuse 的极端 warm 请求
|
||
```
|
||
|
||
### 3.6 关键洞察："Compute-bound but lightweight"
|
||
|
||
高 cache reuse 下的 prefill 处于一种独特状态：
|
||
|
||
```
|
||
Prefill bound 类型: Compute-bound (不变)
|
||
Prefill 绝对工作量: 大幅降低 (71% cache → 只算 29% 的 tokens)
|
||
Prefill-Decode 干扰: 因绝对工作量降低而减轻 (不需要物理隔离)
|
||
```
|
||
|
||
这解释了为什么 PD 分离没有帮助：
|
||
- PD 分离解决的是 "prefill 太重干扰 decode" 的问题
|
||
- 但 cache-aware routing 已经把 prefill 的实际工作量降到足够轻
|
||
- 物理隔离（PD 分离）的收益被 KV 传输开销抵消
|
||
|
||
## 4. 实验结果
|
||
|
||
### 4.1 完整实验矩阵
|
||
|
||
所有实验使用统一的 cache-aware + token-level load-balanced global scheduler。
|
||
|
||
| Config | OK/N | TTFT p50 | TPOT p90 | E2E p50 | APC |
|
||
|--------|------|----------|----------|---------|-----|
|
||
| TP=8 DP=1 (single instance) | 998/1000 | 0.467s | 0.129s | 3.30s | 53.0% |
|
||
| TP=2 DP=4 (4 inst, RR) | 997/999 | 0.844s | 0.095s | 4.92s | 33.5% |
|
||
| TP=1 DP=8 (8 inst, RR) | 997/999 | 1.836s | 0.086s | 6.67s | 20.8% |
|
||
| **TP=1 DP=8 (cache-aware)** | **997/999** | **0.731s** | **0.073s** | **4.48s** | **44.7%** |
|
||
| TP=1 PD-Sep 4P+4D (cache-aware) | 509/564 | 1.261s | 0.074s | 5.61s | 40.2% |
|
||
|
||
### 4.2 Cache-Aware Routing 的效果
|
||
|
||
```
|
||
Round-robin → Cache-aware (Combined TP=1 DP=8):
|
||
TTFT p50: 1.836s → 0.731s (-60%)
|
||
TPOT p90: 0.086s → 0.073s (-15%)
|
||
E2E p50: 6.673s → 4.480s (-33%)
|
||
APC: 20.8% → 44.7% (+24pp)
|
||
```
|
||
|
||
Cache-aware routing 的提升远大于 PD 分离的提升。
|
||
|
||
### 4.3 修复工程问题的过程
|
||
|
||
实验过程中发现并修复了多个 PD 分离的工程问题：
|
||
|
||
| 问题 | 根因 | 修复 |
|
||
|------|------|------|
|
||
| Decode engine crash | vLLM scheduler assert: KV transfer 回调时 request 已 abort | Patch scheduler.py: assert → graceful skip |
|
||
| Head-of-line blocking | Proxy 按 request count 做 LB，不区分大小请求 | Token-level ongoing_tokens load balancing |
|
||
| "Timeout waiting for P side ready" | Proxy fire-and-forget prefill, decode 盲等 KV | Await-prefill + kv_load_failure_policy=recompute |
|
||
| Port collision on startup | 8 Mooncake instances 同时启动争抢 torch distributed port | Staggered startup + explicit MASTER_PORT |
|
||
| Cache routing "rich get richer" | score = ongoing - alpha*cached 导致流量集中到一个 instance | Normalized scoring: ongoing/avg_load - alpha*cache_ratio |
|
||
|
||
## 5. 结论
|
||
|
||
### 5.1 PD 分离为什么在 Agentic Workload 不生效
|
||
|
||
1. **Cache reuse 大幅降低 prefill 绝对工作量（71% cache hit → 只算 29%）**，使得 P-D 干扰不显著
|
||
2. **Prefill 仍然 compute-bound**（即使 95% reuse，AI 仍 >1000），但每个请求的总 FLOPs 因 new_tokens 减少而大幅降低
|
||
3. **Cache-aware routing 提供 "软 PD 隔离"**，效果等同于物理隔离但无 KV 传输开销
|
||
4. **KV 传输开销不可忽略**（TTFT +72%），抵消了隔离收益
|
||
5. **MoE 模型 active params 小**（3B），per-token compute 本身较轻
|
||
|
||
### 5.2 PD 分离在什么条件下有价值
|
||
|
||
| 条件 | Chatbot (有价值) | Agentic (无价值) |
|
||
|------|-----------------|-----------------|
|
||
| Cache hit rate | <10% | **71%** |
|
||
| Model active params | 70B (dense) | **3B (MoE)** |
|
||
| I/O ratio | 1-10x | **75.6x** |
|
||
| Per-request prefill FLOPs | Very high | **Low (after cache)** |
|
||
| KV transfer cost vs prefill cost | Negligible | **Significant** |
|
||
|
||
### 5.3 Agentic Workload 应该怎么优化
|
||
|
||
1. **Cache-aware routing** (已验证有效): 用 ongoing_tokens + prefix_cache_hit 做联合调度，
|
||
将 APC 从 20.8% (RR) 提升到 44.7%，TPOT p90 降低 15%
|
||
|
||
2. **Cross-instance KV cache sharing**: 让多个 instance 共享全局 KV pool，
|
||
进一步提升 cache hit 率接近理论 71%
|
||
|
||
3. **Prefix pre-warming**: 对 cold start 请求（55%，0% cache hit），
|
||
预计算 common prefix (system prompt blocks) 并分发到所有 instance
|
||
|
||
4. **不同 workload 类型的差异化处理**:
|
||
- Warm 请求 (22%, >90% cache hit, avg 1.3k new tokens): 几乎免费，任何 instance 都能处理
|
||
- Cold 请求 (55%, 0% cache hit, avg 17.7k new tokens): prefill-heavy，需要有足够 compute
|
||
- 可以用 request-type-aware routing 进一步优化
|