gahow/agentic-pd-hybrid
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
506d360160082f2160a3f9a983fad3d5e90459be
agentic-pd-hybrid/docs/SWEBENCH_EXPERIMENT_PROGRESS.md
kzlin c9d350b372 docs: KVC v1-v4 debug journey + raise session soft_cap to 16 
Document the iterative debugging from v1 (broken KVC) through v4
(routing fixed + session cap raised), with code-level analysis of
the two main bugs encountered:

1. v2 root cause (mis-diagnosed previously as `allow_local_prefill`):
   `--policy default` for KVC mechanism caused replay's round-robin
   policy and the PD router's round-robin to diverge, sending requests
   with `session_params` to a D worker that did not have the session
   open. Resulted in 56-61% truncation with finish_reason
   "session id X does not exist".
   Fix: use `--policy kv-aware` (sweep_tp1_v3_kvaware.sh) so replay
   emits `x-smg-target-worker` and PD router uses consistent_hashing.

2. v3 new bottleneck: `pd-router-fallback-large-append-session-cap`
   dominated 52-65% of requests. Root cause was hardcoded
   `min(4, ...)` in `_decode_session_soft_cap`. With 7 D workers x 4
   sessions = 28 slots for 52 trace sessions, ~24 sessions starved
   permanently (bimodal direct-to-D rate of 0% or 99%).
   Fix: raise the cap to 16 (replay.py).

Also includes the v3 finding that direct-to-d-session path P50=0.495s
and TTFT P50=0.043s already beats the 8-way DP baseline (0.65s/0.093s)
- the KVC core mechanism works when fallback paths are avoided.

Files:
- docs/KVC_DEBUG_JOURNEY_V1_TO_V4.md: full journey + code location index
- docs/SWEBENCH_EXPERIMENT_{PROGRESS,RESULTS}.md: prior session notes
- scripts/sweep_tp1_v{2,3,4}*.sh: experiment driver scripts
- src/agentic_pd_hybrid/replay.py: cap 4 -> 16, audit fields
- src/agentic_pd_hybrid/pd_router.py: strip session_params from prefill
- src/agentic_pd_hybrid/metrics.py: truncated_request_count

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 21:10:41 +08:00

4.8 KiB
Raw Blame History

SWE-Bench PD Hybrid Experiment Progress

实验目标

在单节点 8xH100 上复现 agentic-pd-hybrid 三种 serving mechanism，对比 Qwen3.5-35B-A3B 在 SWE-Bench 500 instance agentic trajectory 上的性能。

硬件环境

  • 8x H100 80GB (NVLink 互联, 2 NUMA nodes: GPU 0-3 / GPU 4-7)
  • 无 RDMA/IB 设备
  • Transfer backend: mooncake TCP (nixl UCX 因 pip 包缺少 CUDA 支持导致 segfault，已放弃)

实验矩阵

实验 Mechanism Workers GPU 分配 Router Policy
A pd-disaggregation 1P + 1D (TP4 each) P: 0-3, D: 4-7 Yes default
B pd-colo 2 direct (TP4 each) D0: 0-3, D1: 4-7 No default
C kvcache-centric 1P + 1D (TP4 each) P: 0-3, D: 4-7 Yes default

测试负载

  • 源数据: simm-swe-bench/outputs/20260416-205833-hicache-qwen35-verified-0-500/audit.jsonl
  • 39,417 lines (turns), 497 unique instances (sessions)
  • 每个 instance 8-150 turns (均值 79.3)
  • 转换为 agentic-pd-hybrid trace 格式: outputs/qwen35-swebench-500.jsonl

关键发现

Transfer Backend 选择

  • nixl (UCX): pip 安装的 nixl_cu12 包自带的 UCX 库没有 CUDA 支持，导致 GPU memory registration 时 segfault。系统 UCX (/opt/hpcx/ucx) 有 CUDA 支持但因 RPATH 无法被 NIXL 使用。
  • mooncake (TCP): 可用。需要两处修改:
    1. third_party/sglang/.../mooncake_transfer_engine.py: 从环境变量 MOONCAKE_PROTOCOL 读取协议，而非硬编码 "rdma"
    2. src/agentic_pd_hybrid/stack.py: 当 transfer_backend == "mooncake" 且非 force_rdma 时，自动设置 MOONCAKE_PROTOCOL=tcp

代码修改记录

  1. third_party/sglang/python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py

    • 将 "rdma" 硬编码改为 os.environ.get("MOONCAKE_PROTOCOL", "rdma")

  2. src/agentic_pd_hybrid/stack.py

    • 在 _build_process_env() 中添加: mooncake 非 force_rdma 时默认设置 MOONCAKE_PROTOCOL=tcp

  3. scripts/convert_audit_to_trace.py (新建)

    • 将 sibench audit.jsonl 转换为 agentic-pd-hybrid trace 格式

实验进度

  • Step 0: 环境准备 (uv sync, nixl/mooncake 安装)
  • Step 1: Trace 格式转换 (39,417 lines 验证通过)
  • Step 2: Smoke test (pd-disaggregation, mooncake TCP, 100 requests) — 通过
    • 100/100 requests, 0 errors
    • Mean latency: 1.53s, P50: 0.77s, P90: 2.82s
    • TTFT: mean 0.49s, P50 0.29s; TPOT: mean 4.7ms
    • 91/100 cache hits
  • Step 3a: 实验 A 全量尝试 (39K reqs, 497 sessions) — 中止
    • Run dir: outputs/swebench-exps/pd-disaggregation-default-20260426T171113Z (无metrics,被kill)
    • 前 90% 完成 ~80min (~8-10 req/s), 但尾部 D 侧 KV cache 98% 饱和
    • 497 并发 session 争抢 D 侧 token 空间, mamba 80-93 sessions 无法 drain
    • 教训: 1P+1D (TP4) 无法支撑 497 并发 session, 需减少 session 数量或降低 concurrency
  • Step 3b: 实验 A — pd-disaggregation (52 sessions, 4449 reqs, concurrency=32) — 完成
    • Run dir: outputs/swebench-exps/pd-disaggregation-default-20260426T202540Z
    • Trace: outputs/qwen35-swebench-50sess.jsonl (10% sample, 52 sessions)
    • 结果: 4449/4449 成功, 0 errors
    • Latency: mean=1.66s, P50=0.97s, P90=3.64s, P99=7.68s
    • TTFT: mean=0.45s, P50=0.34s, P90=0.88s
    • TPOT: mean=5.2ms, P50=5.2ms
    • Cache hit: 4199/4449 (94.4%)
  • Step 4: 实验 B — pd-colo — 失败: SGLang bug
    • Run dir: outputs/swebench-exps/pd-colo-default-20260426T210129Z
    • Bug: --disaggregation-mode null (colocation) 下 Qwen3.5-35B-A3B 模型触发 token_to_kv_pool_allocator 内存泄漏
    • 错误: ValueError: token_to_kv_pool_allocator memory leak detected!
    • 两个 direct worker 在处理 ~5 个请求后均 crash (Scheduler exception)
    • 结论: 当前 vendored SGLang v0.5.10 不支持 Qwen3.5-35B-A3B 的 colocation 模式
  • Step 5: 实验 C — kvcache-centric — 完成 (高错误率)
    • Run dir: outputs/swebench-exps/kvcache-centric-default-worker-admission-20260426T210800Z
    • 4390/4449 errors (98.7%) — admission control 过于保守
    • 59 成功请求: mean latency 1.24s (比 pd-disagg 快 25%), TTFT 0.18s (快 60%)
    • 详细分析见 docs/SWEBENCH_EXPERIMENT_RESULTS.md
  • Step 6: 结果对比分析 — 完成
    • 完整报告: docs/SWEBENCH_EXPERIMENT_RESULTS.md

启动脚本

  • scripts/run_exp_a_pd_disagg.sh — 实验 A
  • scripts/run_exp_b_pd_colo.sh — 实验 B
  • scripts/run_exp_c_kvcache_centric.sh — 实验 C
  • scripts/convert_audit_to_trace.py — Trace 转换

已知风险

  1. Qwen3.5-35B-A3B TP4 可用 mem ~12GB/GPU (after model + CUDA graph)，长 session (150 turns) 可能 OOM
  2. mooncake TCP loopback 延迟远低于真实跨机，结果偏乐观
  3. 原始 trace 时间跨度 ~6000s，全量回放非常耗时