feat(snapshot): D→P RDMA link Phase 1 — minimal byte transport
A thin wrapper around mooncake.engine.TransferEngine that does one-sided RDMA writes between two SnapshotPeer endpoints. Bypasses SGLang's MooncakeKVManager (which is hard-gated to PREFILL/DECODE roles via add_transfer_request assertion at conn.py:1563) so the D→P direction doesn't require invasive role-axis changes upstream. Smoke test (two subprocess.Popen processes, mlx5_60, 127.0.0.1): 1 KB 9.0 ms (one-time openSegment handshake) 16 KB 0.04 ms 3.5 Gbps 1 MB 0.10 ms 82 Gbps 16 MB 0.58 ms 232 Gbps 64 MB 1.70 ms 316 Gbps (~80% of NDR 400G line rate) All 5 sizes pass SHA256 verification end-to-end. Files: src/agentic_pd_hybrid/snapshot_link.py — SnapshotPeer, SnapshotEndpoint scripts/snapshot_link_receiver.py — child-process receiver scripts/smoke_snapshot_link.py — sender + verifier docs/D_TO_P_PHASE1_LINK_ZH.md — phase 1 acceptance doc Next: Phase 2 (D-side scheduler commit hook), Phase 3 (P-side prefill bypass with snapshot KV). See docs/D_TO_P_SYNC_DESIGN_ZH.md §5.
This commit is contained in:
Claude Code Agent
140
docs/D_TO_P_PHASE1_LINK_ZH.md
Normal file
140
docs/D_TO_P_PHASE1_LINK_ZH.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# D→P Phase 1：底层 RDMA 链路（已验收）
|
||||
|
||||
**日期**：2026-05-13
|
||||
**状态**：底层链路通过 smoke test 验收
|
||||
**前置**：`docs/D_TO_P_SYNC_DESIGN_ZH.md`
|
||||
**对应 commit**：`feat(snapshot): D→P snapshot link over mooncake RDMA`
|
||||
|
||||
---
|
||||
|
||||
## 0. 一句话
|
||||
|
||||
实现一个独立于 SGLang `MooncakeKVManager` 的**最小 RDMA 字节传输模块**（`src/agentic_pd_hybrid/snapshot_link.py`），双进程 smoke test 跑通 1 KB → 64 MB 一共 5 个 size，全部 SHA 校验通过，64 MB 单次 RDMA write 实测 315 Gbps（mlx5_60 NDR 400 Gb 的约 80%）。
|
||||
|
||||
## 1. 设计动机
|
||||
|
||||
`docs/D_TO_P_SYNC_DESIGN_ZH.md` 选定 Option C（D→P snapshot push + P SessionSlot + prefill bypass）。这个方案的最底层依赖是"D 进程能把字节通过 RDMA 推到 P 进程的预注册缓冲区"。
|
||||
|
||||
直接复用 SGLang 的 `MooncakeKVManager` 不可行：
|
||||
- `add_transfer_request` 在 `conn.py:1563` 硬 assert `disaggregation_mode == PREFILL`
|
||||
- PD pipeline 的发送 / 接收 thread / queue / staging 紧耦合 PD 角色
|
||||
- 改 PD 路径风险大（影响现有 E1/E2/E3 配置）
|
||||
|
||||
因此把 D→P link 单独写成一个轻量模块，直接调 `mooncake.engine.TransferEngine` 的 `transfer_sync_write` / `batch_transfer_sync_write`，不经过 PD pipeline。
|
||||
|
||||
## 2. 实现
|
||||
|
||||
### 2.1 `snapshot_link.SnapshotPeer`
|
||||
|
||||
```python
|
||||
peer = SnapshotPeer(host, port, ib_device, receive_capacity_bytes)
|
||||
endpoint = peer.endpoint # SnapshotEndpoint(session_id, base_ptr, capacity_bytes)
|
||||
peer.register_send_buffer(ptr, length)
|
||||
peer.push(target_endpoint, local_ptr, local_off, length, remote_off=0)
|
||||
peer.batch_push(target, local_addrs, remote_addrs, lengths)
|
||||
peer.read_bytes(offset, length) -> bytes
|
||||
peer.close()
|
||||
```
|
||||
|
||||
- 每个 `SnapshotPeer` 拥有自己的 `TransferEngine`，绑定 `host:port`
|
||||
- `receive_capacity_bytes > 0` 时分配一段 ctypes `c_ubyte` 数组 + `register_memory`
|
||||
- `push` 直接走 `engine.transfer_sync_write(peer_session_id, local_ptr, remote_ptr, length)`
|
||||
- 角色完全对称——任何 `SnapshotPeer` 既可以发送也可以接收，由 caller 决定
|
||||
|
||||
### 2.2 Smoke test 双进程结构
|
||||
|
||||
```
|
||||
父进程 (sender) 子进程 (receiver, subprocess.Popen)
|
||||
│ │
|
||||
│ spawn → ──────────────────────────────►│
|
||||
│ │ SnapshotPeer(recv_capacity=64MB)
|
||||
│ │ write endpoint.json
|
||||
│ read endpoint.json ◄───────────────────│
|
||||
│ │
|
||||
│ SnapshotPeer(no recv buf) │
|
||||
│ register_send_buffer(64MB) │
|
||||
│ │
|
||||
│ for size in [1K, 16K, 1M, 16M, 64M]: │
|
||||
│ fill_pattern(send_buf, seed) │
|
||||
│ peer.push(endpoint, 0, size) ─RDMA──►│
|
||||
│ │ wait signal
|
||||
│ write endpoint.do{size} ────────────►│ read signal seed
|
||||
│ │ compute expected SHA
|
||||
│ │ recv_bytes = peer.read_bytes
|
||||
│ wait endpoint.ack{size} │ compare SHA → emit JSON event
|
||||
│ │ write endpoint.ack{size}
|
||||
│ ... │
|
||||
│ │
|
||||
│ drain child stdout, parse JSON │ exit
|
||||
│ verify each event has ok=true │
|
||||
```
|
||||
|
||||
### 2.3 性能（首次 smoke run）
|
||||
|
||||
| Size | Push duration | Throughput |
|
||||
|---:|---:|---:|
|
||||
| 1 KB | 9.0 ms | 0.001 Gbps |
|
||||
| 16 KB | 0.037 ms | 3.5 Gbps |
|
||||
| 1 MB | 0.102 ms | 82 Gbps |
|
||||
| 16 MB | 0.577 ms | 232 Gbps |
|
||||
| **64 MB** | **1.70 ms** | **316 Gbps** |
|
||||
|
||||
- 1 KB 第一次有 ~9 ms 的 mooncake p2p handshake/openSegment overhead（一次性）
|
||||
- 16 KB 之后是稳态，吞吐随 size 增长接近线速
|
||||
- mlx5_60 是 mlx5 ConnectX-7 NDR 400 Gb（4× 100Gb lanes）；64 MB 测到 316 Gbps 是 79% 的链路利用率，对单次 RDMA write 来说正常（剩余空间留给 verb dispatch / completion handling overhead）
|
||||
|
||||
## 3. 验收
|
||||
|
||||
- ✅ 5/5 size SHA 校验全部通过
|
||||
- ✅ 64 MB 一次 RDMA 1.7 ms
|
||||
- ✅ 双进程独立，不耦合 SGLang PD pipeline
|
||||
- ✅ Smoke test 脚本 `scripts/smoke_snapshot_link.py` 可重跑
|
||||
|
||||
## 4. 当前覆盖范围（清单）
|
||||
|
||||
- ✅ Host CPU 内存的 D→P RDMA byte transfer
|
||||
- ✅ 单 IB device (mlx5_60)
|
||||
- ✅ 同节点 loopback（127.0.0.1）
|
||||
- ⏳ GPU 内存（设备指针 + `batch_transfer_write_on_cuda`）—— 现有 `push()` 走 `transfer_sync_write`，对 GPU 指针支持取决于 mooncake 的 protocol；下一步验证
|
||||
- ⏳ 跨节点（远端 IP）—— 设计上一致，未验证
|
||||
- ⏳ 多 D → 单 P（多 sender → 共享 recv buffer 的 offset 协调）—— 留给 Phase 3 整合时设计
|
||||
- ⏳ ZeroCopy 入 SGLang kv_pool slot —— 留给 Phase 2
|
||||
|
||||
## 5. 下一步（Phase 2 / Phase 3）
|
||||
|
||||
详见 `docs/D_TO_P_SYNC_DESIGN_ZH.md` §5。本 phase 1 解锁后，整个 D→P 同步可以正式开始整合到 SGLang scheduler：
|
||||
|
||||
| Phase | 描述 | 风险 |
|
||||
|---|---|---|
|
||||
| 2 | D-side commit hook：`cache_finished_req` 完成后 enqueue snapshot push | 中。需要在 scheduler 后台线程跑 push，不能阻塞 schedule loop |
|
||||
| 3 | P-side snapshot store + prefill bypass：P scheduler 收到 use-snapshot 请求时跳过 `model.forward()`，直接用 snapshot KV 触发 P→D' transfer | **最高**。需要深入 SGLang prefill 流程 |
|
||||
| 4 | agentic-pd-hybrid hook：`_invoke_kvcache_seeded_router` 先 probe P → 决定走 bypass 还是 fallback | 低 |
|
||||
| 5 | CLI flag + structural log | 低 |
|
||||
| 6 | 端到端 smoke + E4 sweep | 中 |
|
||||
|
||||
## 6. 知识沉淀
|
||||
|
||||
### 易踩坑
|
||||
|
||||
| 坑 | 原因 | 修法 |
|
||||
|---|---|---|
|
||||
| 多进程 `multiprocessing.Process` 子进程崩溃信息丢失 | spawn context 下 child 没有继承 parent 的 stderr | 改用 `subprocess.Popen` + stderr 重定向到文件 |
|
||||
| `bytes(ctypes.c_byte * N)` 失败 `ValueError: bytes must be in range(0, 256)` | `c_byte` 是 **signed**，>= 128 的 byte 在 Python 看就是负数 | 用 `c_ubyte` 或 `ctypes.string_at(addr, length)` 做内存复制 |
|
||||
| 第一次 push 有 ~9ms openSegment overhead | mooncake p2p handshake lazy 建链 | 稳态忽略；如需 warm-up，提前发 1 KB pre-flight |
|
||||
|
||||
### mooncake API 速查
|
||||
|
||||
```python
|
||||
engine = TransferEngine()
|
||||
engine.initialize(f"{host}:{port}", "P2PHANDSHAKE", "rdma", ib_device)
|
||||
engine.register_memory(ptr, length) # mr 注册
|
||||
engine.transfer_sync_write(peer_session_id, local_ptr, remote_ptr, length) # RDMA write
|
||||
engine.batch_transfer_sync_write(peer_session_id, [local_ptrs], [remote_ptrs], [lengths])
|
||||
engine.unregister_memory(ptr)
|
||||
```
|
||||
|
||||
`peer_session_id` 是 `"host:rpc_port"`，其中 `rpc_port = peer_engine.get_rpc_port()`。
|
||||
|
||||
---
|
||||
|
||||
**核心句**：D→P 底层 RDMA 链路独立模块跑通，64 MB 1.7 ms / 316 Gbps，与 SGLang PD pipeline 完全解耦。Phase 2/3 可以放心在这上面叠加。
|
||||
Reference in New Issue
Block a user