Gahow Wang
7f93d36970
Evidence-backed analysis with per-request matched comparison: 1. KV CACHE MEMORY WALL (Evidence 3) Combined: 12% KV cache per instance (comfortable) PD-Sep 6P+2D: 48-97% on decode instances (saturation -> 100s waits) 2. KV TRANSFER OVERHEAD (Evidence 4, matched requests) Mean 1.79s extra TTFT per request, 3.3x slower overall Small requests (<5k) hit 8.0x ratio (transfer dominates prefill) Large requests (>50k) hit 1.3x ratio (prefill dominates) 3. SESSION AFFINITY BROKEN (Evidence 5) Combined: turn N+1 hits same GPU -> 80% multi-turn APC PD-Sep: turn N+1 prefill on P has NO prior KV (sent to D) -> 0% APC on P Must re-prefill + re-transfer on every turn 4. GPU UNDERUTILIZATION (Evidence 2) PD-Sep: 12-17% GPU util (decode is memory-bound, wastes GPU compute) Combined: 28-54% GPU util (flexible P+D on same GPU) Root cause: agentic workloads break PD-Sep's assumptions (short input, no prefix sharing, compute-heavy prefill) with long context, 91% intra-session KV reuse, and lightweight MoE compute. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>