docs(kvc): correct reseed cost decomposition + flag D->P sync gap
After an independent Opus-agent forensic audit, the previous "(c) 增量 fetch (工程量较大,未实现)" line in V2_DEEP_ANALYSIS §4.2 was understating the gap. The audit confirmed: - No D->P KV transfer code exists in the framework at any layer (agentic_pd_hybrid orchestration, vendored SGLang disaggregation, or mooncake transport). - Mooncake MooncakeKVManager has a hard role split: PREFILL = sender, DECODE = receiver-only loop. `add_transfer_request` asserts the disaggregation_mode is PREFILL. - The BaseKVSender / BaseKVReceiver abstraction has no bidirectional slot. - session_aware_cache.release_session only calls kv_pool_allocator.free() on eviction -- no serialization, no outbound network call. - _commit_prefill_backup_residency is only called from the seed/reseed path (_invoke_kvcache_seeded_router). direct-to-D path never updates P-side backup state. - "capacity-backup" policy semantics: it only skips the close on P after reseed -- the backup is the seed-time static snapshot, never refreshed by D-side append-prefill activity. V2_DEEP_ANALYSIS §4.2: - Decomposed the 3-7s reseed cost into the P-side re-prefill segment (1.5-3s, dominant) and the P->D mooncake transfer segment (1.5-4s). - Quantified the realistic effect of enabling RDMA: only the transfer segment shrinks, reseed reduces to 1.7-3.2s, TTFT p99 ~0.7s, still loses to DP's 0.43s. - Replaced the throwaway "(c) incremental fetch" line with a full paragraph explaining what D->P sync would require, why it's the largest engineering gap, and that the blocker is SGLang's radix-tree single-producer assumption, not the network layer. KVC_ROUTER_ALGORITHM §9: - Refined Open Question 3 (RDMA) to clarify it only helps the transfer segment, not the re-prefill segment. - Added Open Question 4: D->P incremental KV sync as the central future-work contribution gap, with cited evidence for why it doesn't currently exist. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -274,12 +274,23 @@ Critic agent 对 v2 vs 4DP 的对等性做了 10 项审查。下面分两类:
|
||||
|
||||
按 session 分布:88% (52/59) 集中在 5 个超大输入 session(22080 / 44800 / 22400 / 58080 / 45280,input 60-90K)。
|
||||
|
||||
**机理**:reseed 必须把 session 整段 KV(50-90K tokens)通过 mooncake TCP loopback 从 P 推到 D。单次 transfer 实测 3-7s。DP 没有这条路径,每个请求在本地 worker 直接 prefill,相同 input 量做完只需 0.5-1s。
|
||||
**机理拆分**:reseed 路径的延迟由两段组成——
|
||||
1. **P 端 re-prefill 段**:用 trace 中带的完整 prompt 在 P 上重新算 prefill。**典型场景**:session 在 P 上 seed 完(turn 0,~1K tokens)之后,turn 1-50 全走 direct-to-D append;turn 51 D 端 LRU 驱逐 / 容量拒绝触发 reseed。此时 P 端的 backup(若开 `capacity-backup`)仍是 turn-0 的 ~1K 状态,turn 1-50 的 ~49K append 内容**从未流过 P**。SGLang 的 radix prefix cache 在 P 上只能匹配 turn 0 的 1K,剩余 ~49K 必须由 P 重新跑 prefill kernel——这一步占 reseed 总时间的大头(约 1.5-3s @ 1×H100,30B 模型)。
|
||||
2. **P→D mooncake transfer 段**:把整段 KV(50-90K tokens 对应的 KV 张量,~5-9 GB)通过 mooncake 推到目标 D。本次 benchmark 用的是 TCP loopback,实测 1.5-4s(取决于 session 大小)。生产用 IB RDMA(节点实际有 mlx5_0/_1 @ 200 Gb/s × 2 active)应可压到 200-400ms。
|
||||
|
||||
**这是 KVC 机制本身的代价,不是 measurement bug。** 生产部署的缓解策略:
|
||||
- (a) **真 RDMA 替换 mooncake TCP loopback**——本次 benchmark 用的是单机 TCP 模拟,生产用 IB/RoCE 后预期 transfer 从 3-7s 压到 0.3-0.7s(10×),slow path 长尾可能消失
|
||||
- (b) **容量规划**:sessions × peak context ≤ 总 D KV pool × 0.7,让 LRU/reseed 几乎不触发
|
||||
- (c) **增量 fetch**:reseed 时只 transfer overlap 之外的 delta(工程量较大,未实现)
|
||||
**两段相加**:当前 reseed 中位 ~2.5s、p99 ~7.7s。
|
||||
|
||||
### 缓解策略的真实效果
|
||||
|
||||
- (a) **真 RDMA 替换 mooncake TCP loopback**——救的是 transfer 段(~1.5-4s → ~200-400ms),不动 re-prefill 段。预期 reseed 总延迟从 3-7s 压到 **1.7-3.2s**,TTFT p99 从 1.28s 降到 ~0.7s 量级(**仍输 DP 0.43s**)。**当前 sweep 未启用**(缺 `--force-rdma --ib-device mlx5_0`)。
|
||||
- (b) **容量规划**:sessions × peak context ≤ 总 D KV pool × 0.7,让 LRU/reseed 几乎不触发。对生产部署而言最可靠,但对本 trace 不适用——sessions 已固定。
|
||||
- (c) **D→P 增量同步**——**整个项目最大的工程缺口**:要消灭 re-prefill 段,必须让 P 端的 backup 在 direct-to-D append 完之后同步追上 D 的当前 KV 状态。这样 reseed 时 P 端已经有最新整段 KV,可以直接 P→D transfer,无需 re-prefill。**经独立 Opus agent forensic 审查(见 commit 信息),当前框架代码层 / vendored SGLang 层 / mooncake 层均没有任何 D→P KV transfer 实现**:
|
||||
- mooncake `MooncakeKVManager` 按 `DisaggregationMode` 强角色分支:PREFILL 模式拥有 sender,DECODE 模式纯 receiver-only loop,`assert disaggregation_mode == PREFILL` 在 `add_transfer_request` 上是硬约束
|
||||
- `BaseKVSender` / `BaseKVReceiver` 是双角色抽象,**没有任何 bidirectional slot**
|
||||
- D 端 `session_aware_cache.release_session` 只调 `kv_pool_allocator.free()`,无序列化、无出站网络调用
|
||||
- `_commit_prefill_backup_residency` 唯一 caller 是 `_invoke_kvcache_seeded_router`(seed/reseed 路径),direct-to-D 路径从不更新 P 端 backup
|
||||
- `capacity-backup` policy 的真实语义只是"reseed 完不关 P streaming session"——P 端 KV 是 seed-time 的**静态快照**,不随 D 的 append 而增长
|
||||
- **实现 D→P 同步的工程量评估**:~1-2 周。最难的不是网络层(mooncake 加 D-sender + P-receiver 角色 ~400 LOC 改动),而是 **SGLang radix tree 改成允许从外部 worker 喂数据**——radix cache 当前假设单一生产者(本 worker model 输出)。这是论文里 §future-work 的核心 contribution 缺口。
|
||||
|
||||
### 4.3 Error 统计口径已修复;abort 数双方都比之前发现的多
|
||||
|
||||
|
||||
Reference in New Issue
Block a user