feat(snapshot): D→P RDMA Phase 1b — GPU pointer path verified

Confirms snapshot_link works for cuda device pointers, not just host
memory. Sender on cuda:0 pushes to receiver on cuda:1 via RDMA over
mlx5_60. All 5 sizes (16K, 1M, 16M, 64M, 256M) pass SHA verification.

  16 KB     8.3 ms   0.016 Gbps  (cold openSegment)
  1 MB      0.10 ms  87.6 Gbps
  16 MB     0.84 ms  159 Gbps
  64 MB     2.52 ms  213 Gbps
  256 MB    8.54 ms  251 Gbps    (~60% NDR400 line rate)

For Inferact-scale sessions (~50K tokens × ~80 KB layer-per-token =
~4 GB), this projects D→P transfer time at ~130 ms — within the
"reseed-savings" envelope sketched in design doc §3.2.

Files:
  scripts/snapshot_link_receiver_gpu.py
  scripts/smoke_snapshot_link_gpu.py

Next: SGLang scheduler integration for D-side dump + P-side ingest.
This commit is contained in:
Claude Code Agent
2026-05-13 00:59:43 +08:00
parent dc4867c270
commit 7216507773
3 changed files with 375 additions and 3 deletions

View File

@@ -92,13 +92,25 @@ peer.close()
## 4. 当前覆盖范围(清单)
- ✅ Host CPU 内存的 D→P RDMA byte transfer
- ✅ Host CPU 内存的 D→P RDMA byte transfer (`scripts/smoke_snapshot_link.py`)
-**GPU 内存** cuda:0 → cuda:1 的 D→P RDMA`scripts/smoke_snapshot_link_gpu.py`5/5 size 全 SHA 校验通过256 MB 8.5 ms / 251 Gbps
- ✅ 单 IB device (mlx5_60)
- ✅ 同节点 loopback127.0.0.1
- ⏳ GPU 内存(设备指针 + `batch_transfer_write_on_cuda`)—— 现有 `push()``transfer_sync_write`,对 GPU 指针支持取决于 mooncake 的 protocol下一步验证
- ⏳ 跨节点(远端 IP—— 设计上一致,未验证
- ⏳ 多 D → 单 P多 sender → 共享 recv buffer 的 offset 协调)—— 留给 Phase 3 整合时设计
- ⏳ ZeroCopy 入 SGLang kv_pool slot —— 留给 Phase 2
- ⏳ ZeroCopy 入 SGLang kv_pool slot —— 留给 Phase 2/3
### GPU smoke 性能
| Size | Push duration | Throughput |
|---:|---:|---:|
| 16 KB | 8.27 ms (cold) | 0.016 Gbps |
| 1 MB | 0.096 ms | 87.6 Gbps |
| 16 MB | 0.844 ms | 159 Gbps |
| 64 MB | 2.52 ms | 213 Gbps |
| **256 MB** | **8.54 ms** | **251 Gbps** |
GPU↔GPU 比 host↔host 慢一些251 vs 316 Gbps for 64MB但仍接近 mlx5_60 NDR 400Gb 的 60% 线率。对 KVC 单 session ~50K tokens × ~80 KB/token ≈ 4 GB 量级的 transfer对应 D→P 时间约 130 ms。
## 5. 下一步Phase 2 / Phase 3