c9d350b372
Document the iterative debugging from v1 (broken KVC) through v4
(routing fixed + session cap raised), with code-level analysis of
the two main bugs encountered:
1. v2 root cause (mis-diagnosed previously as `allow_local_prefill`):
`--policy default` for KVC mechanism caused replay's round-robin
policy and the PD router's round-robin to diverge, sending requests
with `session_params` to a D worker that did not have the session
open. Resulted in 56-61% truncation with finish_reason
"session id X does not exist".
Fix: use `--policy kv-aware` (sweep_tp1_v3_kvaware.sh) so replay
emits `x-smg-target-worker` and PD router uses consistent_hashing.
2. v3 new bottleneck: `pd-router-fallback-large-append-session-cap`
dominated 52-65% of requests. Root cause was hardcoded
`min(4, ...)` in `_decode_session_soft_cap`. With 7 D workers x 4
sessions = 28 slots for 52 trace sessions, ~24 sessions starved
permanently (bimodal direct-to-D rate of 0% or 99%).
Fix: raise the cap to 16 (replay.py).
Also includes the v3 finding that direct-to-d-session path P50=0.495s
and TTFT P50=0.043s already beats the 8-way DP baseline (0.65s/0.093s)
- the KVC core mechanism works when fallback paths are avoided.
Files:
- docs/KVC_DEBUG_JOURNEY_V1_TO_V4.md: full journey + code location index
- docs/SWEBENCH_EXPERIMENT_{PROGRESS,RESULTS}.md: prior session notes
- scripts/sweep_tp1_v{2,3,4}*.sh: experiment driver scripts
- src/agentic_pd_hybrid/replay.py: cap 4 -> 16, audit fields
- src/agentic_pd_hybrid/pd_router.py: strip session_params from prefill
- src/agentic_pd_hybrid/metrics.py: truncated_request_count
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
96 lines
4.8 KiB
Markdown
96 lines
4.8 KiB
Markdown
# SWE-Bench PD Hybrid Experiment Progress
|
||
|
||
## 实验目标
|
||
|
||
在单节点 8xH100 上复现 agentic-pd-hybrid 三种 serving mechanism，对比 Qwen3.5-35B-A3B 在 SWE-Bench 500 instance agentic trajectory 上的性能。
|
||
|
||
## 硬件环境
|
||
|
||
- 8x H100 80GB (NVLink 互联, 2 NUMA nodes: GPU 0-3 / GPU 4-7)
|
||
- 无 RDMA/IB 设备
|
||
- Transfer backend: **mooncake TCP** (nixl UCX 因 pip 包缺少 CUDA 支持导致 segfault，已放弃)
|
||
|
||
## 实验矩阵
|
||
|
||
| 实验 | Mechanism | Workers | GPU 分配 | Router | Policy |
|
||
|------|-----------|---------|----------|--------|--------|
|
||
| A | pd-disaggregation | 1P + 1D (TP4 each) | P: 0-3, D: 4-7 | Yes | default |
|
||
| B | pd-colo | 2 direct (TP4 each) | D0: 0-3, D1: 4-7 | No | default |
|
||
| C | kvcache-centric | 1P + 1D (TP4 each) | P: 0-3, D: 4-7 | Yes | default |
|
||
|
||
## 测试负载
|
||
|
||
- 源数据: `simm-swe-bench/outputs/20260416-205833-hicache-qwen35-verified-0-500/audit.jsonl`
|
||
- 39,417 lines (turns), 497 unique instances (sessions)
|
||
- 每个 instance 8-150 turns (均值 79.3)
|
||
- 转换为 agentic-pd-hybrid trace 格式: `outputs/qwen35-swebench-500.jsonl`
|
||
|
||
## 关键发现
|
||
|
||
### Transfer Backend 选择
|
||
|
||
- **nixl (UCX)**: pip 安装的 nixl_cu12 包自带的 UCX 库没有 CUDA 支持，导致 GPU memory registration 时 segfault。系统 UCX (/opt/hpcx/ucx) 有 CUDA 支持但因 RPATH 无法被 NIXL 使用。
|
||
- **mooncake (TCP)**: 可用。需要两处修改:
|
||
1. `third_party/sglang/.../mooncake_transfer_engine.py`: 从环境变量 `MOONCAKE_PROTOCOL` 读取协议，而非硬编码 `"rdma"`
|
||
2. `src/agentic_pd_hybrid/stack.py`: 当 `transfer_backend == "mooncake"` 且非 `force_rdma` 时，自动设置 `MOONCAKE_PROTOCOL=tcp`
|
||
|
||
### 代码修改记录
|
||
|
||
1. **`third_party/sglang/python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py`**
|
||
- 将 `"rdma"` 硬编码改为 `os.environ.get("MOONCAKE_PROTOCOL", "rdma")`
|
||
|
||
2. **`src/agentic_pd_hybrid/stack.py`**
|
||
- 在 `_build_process_env()` 中添加: mooncake 非 force_rdma 时默认设置 `MOONCAKE_PROTOCOL=tcp`
|
||
|
||
3. **`scripts/convert_audit_to_trace.py`** (新建)
|
||
- 将 sibench audit.jsonl 转换为 agentic-pd-hybrid trace 格式
|
||
|
||
## 实验进度
|
||
|
||
- [x] Step 0: 环境准备 (uv sync, nixl/mooncake 安装)
|
||
- [x] Step 1: Trace 格式转换 (39,417 lines 验证通过)
|
||
- [x] Step 2: Smoke test (pd-disaggregation, mooncake TCP, 100 requests) — **通过**
|
||
- 100/100 requests, 0 errors
|
||
- Mean latency: 1.53s, P50: 0.77s, P90: 2.82s
|
||
- TTFT: mean 0.49s, P50 0.29s; TPOT: mean 4.7ms
|
||
- 91/100 cache hits
|
||
- [x] Step 3a: 实验 A 全量尝试 (39K reqs, 497 sessions) — **中止**
|
||
- Run dir: `outputs/swebench-exps/pd-disaggregation-default-20260426T171113Z` (无metrics,被kill)
|
||
- 前 90% 完成 ~80min (~8-10 req/s), 但尾部 D 侧 KV cache 98% 饱和
|
||
- 497 并发 session 争抢 D 侧 token 空间, mamba 80-93 sessions 无法 drain
|
||
- **教训**: 1P+1D (TP4) 无法支撑 497 并发 session, 需减少 session 数量或降低 concurrency
|
||
- [x] Step 3b: 实验 A — pd-disaggregation (52 sessions, 4449 reqs, concurrency=32) — **完成**
|
||
- Run dir: `outputs/swebench-exps/pd-disaggregation-default-20260426T202540Z`
|
||
- Trace: `outputs/qwen35-swebench-50sess.jsonl` (10% sample, 52 sessions)
|
||
- **结果**: 4449/4449 成功, 0 errors
|
||
- Latency: mean=1.66s, P50=0.97s, P90=3.64s, P99=7.68s
|
||
- TTFT: mean=0.45s, P50=0.34s, P90=0.88s
|
||
- TPOT: mean=5.2ms, P50=5.2ms
|
||
- Cache hit: 4199/4449 (94.4%)
|
||
- [x] Step 4: 实验 B — pd-colo — **失败: SGLang bug**
|
||
- Run dir: `outputs/swebench-exps/pd-colo-default-20260426T210129Z`
|
||
- **Bug**: `--disaggregation-mode null` (colocation) 下 Qwen3.5-35B-A3B 模型触发 token_to_kv_pool_allocator 内存泄漏
|
||
- 错误: `ValueError: token_to_kv_pool_allocator memory leak detected!`
|
||
- 两个 direct worker 在处理 ~5 个请求后均 crash (Scheduler exception)
|
||
- **结论**: 当前 vendored SGLang v0.5.10 不支持 Qwen3.5-35B-A3B 的 colocation 模式
|
||
- [x] Step 5: 实验 C — kvcache-centric — **完成 (高错误率)**
|
||
- Run dir: `outputs/swebench-exps/kvcache-centric-default-worker-admission-20260426T210800Z`
|
||
- 4390/4449 errors (98.7%) — admission control 过于保守
|
||
- 59 成功请求: mean latency 1.24s (比 pd-disagg 快 25%), TTFT 0.18s (快 60%)
|
||
- 详细分析见 `docs/SWEBENCH_EXPERIMENT_RESULTS.md`
|
||
- [x] Step 6: 结果对比分析 — **完成**
|
||
- 完整报告: `docs/SWEBENCH_EXPERIMENT_RESULTS.md`
|
||
|
||
## 启动脚本
|
||
|
||
- `scripts/run_exp_a_pd_disagg.sh` — 实验 A
|
||
- `scripts/run_exp_b_pd_colo.sh` — 实验 B
|
||
- `scripts/run_exp_c_kvcache_centric.sh` — 实验 C
|
||
- `scripts/convert_audit_to_trace.py` — Trace 转换
|
||
|
||
## 已知风险
|
||
|
||
1. Qwen3.5-35B-A3B TP4 可用 mem ~12GB/GPU (after model + CUDA graph)，长 session (150 turns) 可能 OOM
|
||
2. mooncake TCP loopback 延迟远低于真实跨机，结果偏乐观
|
||
3. 原始 trace 时间跨度 ~6000s，全量回放非常耗时
|