From baf7ffb08cde58500ab89a5ac9c934dc7ba6f582 Mon Sep 17 00:00:00 2001 From: Gahow Wang Date: Sat, 23 May 2026 05:51:47 +0800 Subject: [PATCH] 16-session contention: TPOT +45% from prefill-decode interference Key finding: at 16 concurrent sessions (2 per GPU), TPOT p90 degrades from 0.073 to 0.106 (+45%), with MEDIUM TPOT at 0.197 (+149%). This is the first time we've reproduced real prefill-decode interference in controlled experiments. Elastic RDMA at 16 sessions doesn't help: only 13/500 offloaded (cache-gate correct for cold turn-1), kv_both adds ~16% TPOT overhead at high concurrency. Load scaling: 1000req_ts20, 200req_ts10, 200req_ts5, 500req_ts10 all show ~30% GPU util at 8 sessions. The bottleneck is max_inflight_sessions, not arrival rate. Updated elastic_hypotheses.md with H8, H9, and comprehensive final analysis. The real bottleneck is vLLM's chunked prefill scheduling, not routing or PD disaggregation. Co-Authored-By: Claude Opus 4.6 (1M context) --- analysis/elastic_hypotheses.md | 89 ++++++++++++++++++++++++++++++++-- 1 file changed, 84 insertions(+), 5 deletions(-) diff --git a/analysis/elastic_hypotheses.md b/analysis/elastic_hypotheses.md index ddbd828..1960d4e 100644 --- a/analysis/elastic_hypotheses.md +++ b/analysis/elastic_hypotheses.md @@ -185,9 +185,88 @@ OK/N: 198/200 198/200 ← same reliability 5. **GPU balance improvement → HEAVY TTFT -10.5%**: validated (H4 HEAVY_COLO data) 6. **The bottleneck is at time_scale=20 with 200 req**: system is only 30% loaded. Higher load may reveal more optimization opportunities. -## Next directions +--- -- **Higher load (time_scale=10, 500+ req)**: increase contention to amplify routing differences -- **1000 req at time_scale=20**: reduce statistical noise (±7% → ±3%) -- **Session-count-aware placement**: balance number of active heavy sessions per instance, not just ongoing tokens -- **Layerwise KV transfer**: modify Mooncake to pipeline KV transfer with compute (requires deep vLLM change) +## H8: Higher concurrency reveals prefill-decode interference + +**Claim**: At 8 sessions / 8 GPUs, the system is underloaded (30% GPU util). Increasing to 16 sessions should reveal prefill-decode interference. + +**Experiments**: +- 8 sessions, ts=20, 1000 req: TPOT90=0.073, GPU=30%, imbal=1.5x +- 16 sessions, ts=10, 500 req: TPOT90=0.106, GPU=~25%, imbal=~3.5x +- 32 sessions, ts=10, 500 req: (not run yet) + +**Result**: +``` + 8 sessions 16 sessions Delta +TPOT p90: 0.0729 0.1058 +45%! +WARM TPOT90: 0.0640 0.1301 +103%! +MEDIUM TPOT90: 0.0750 0.1970 +149%! +HEAVY TTFT50: (varies) 3.399 — +E2E p50: 4.516 5.830 +29% +``` + +**Verdict**: **VALIDATED**. 16 sessions creates real prefill-decode interference. MEDIUM TPOT degrades 2.5x because HEAVY prefills (via chunked prefill) block decode steps on the same GPU. This is the scenario where PD disaggregation should theoretically help. + +--- + +## H9: Elastic RDMA offload at 16 sessions reduces interference + +**Claim**: At 16 sessions where interference is severe, elastic V2 (C_s prefill + flexible D decode via RDMA) should reduce TPOT by isolating heavy prefill from decode. + +**Experiment**: 16 sessions, 500 req, elastic (kv_both + H4 cache-gate) + +**Result**: +``` + Baseline 16s Elastic 16s Delta +TPOT p90: 0.1058 0.1231 +16% (WORSE) +MEDIUM TPOT90: 0.1970 0.2056 +4% (same) +TTFT p50: 0.828 0.937 +13% (WORSE) +E2E p50: 5.830 6.528 +12% (WORSE) +OK/N: 498/500 498/500 same +Offloaded: — 13/500 (2.6%) too few to matter +``` + +**Verdict**: **REJECTED**. Elastic at 16 sessions is WORSE, not better. Root causes: +1. Cache-gate correctly blocks 89% of HEAVY (cold turn-1, cache_ratio=0) → only 13 offloads +2. kv_both runtime overhead at high concurrency adds ~16% TPOT vs plain baseline +3. The 13 offloaded requests have TTFT p50=17.5s (RDMA overhead), much worse than colocated 3.5s + +**Key learning**: The RDMA transfer approach cannot solve prefill-decode interference because: +- Most HEAVY are cold (no cache to benefit from offload) +- Mooncake lacks layerwise transfer (RDMA is pure sequential overhead after prefill) +- kv_both has non-zero overhead at high concurrency (contradicts Phase 0 at low concurrency) + +--- + +## Current Understanding (final) + +### What DOESN'T work for agentic workloads: + +1. **PD-Sep**: net negative — KV cache memory wall on decode instances +2. **LMetric (OSDI'26)**: ≈ linear routing — session affinity limits routing freedom +3. **Elastic P2P RDMA offload**: net negative — Mooncake transfer overhead, no layerwise pipeline +4. **OVERLOAD_FACTOR tuning**: no effect — imbalance from workload skew, not routing +5. **Dedicated Prefill Service (PS)**: cannot win cost comparison without KV pull, PS is always slower than cached C +6. **Cache-gate offload (H4)**: correct but only 10-12% of HEAVY have cache → limited activation + +### What DOES work: + +1. **Cache-aware session-sticky routing**: +24pp APC, -60% TTFT vs round-robin (the dominant optimization) +2. **GPU balance from offload routing**: HEAVY_COLO -10.5% TTFT when imbalance reduced (H4 data) + +### The real bottleneck: + +At production-level concurrency (>1 session/GPU), the dominant bottleneck is **chunked prefill interference**: large HEAVY prefill chunks block decode steps on the same GPU, causing TPOT to degrade 45-149%. + +Neither routing nor RDMA-based PD disaggregation solves this. The root cause is vLLM's scheduler design: +- Chunked prefill chunk size (`max_num_batched_tokens`, default 8192) is fixed +- Large prefill chunks monopolize the GPU for tens of ms, stalling decode +- Reducing chunk size would improve decode responsiveness but increase prefill overhead + +### Next direction: Adaptive chunked prefill scheduling + +Instead of fixed chunk size, dynamically adjust based on decode pressure: +- When decode queue is deep: smaller chunks → more decode slots → better TPOT +- When decode queue is empty: larger chunks → faster prefill → better TTFT +- This is a vLLM scheduler modification, not a routing change