16-session contention: TPOT +45% from prefill-decode interference

Key finding: at 16 concurrent sessions (2 per GPU), TPOT p90 degrades
from 0.073 to 0.106 (+45%), with MEDIUM TPOT at 0.197 (+149%).
This is the first time we've reproduced real prefill-decode interference
in controlled experiments.

Elastic RDMA at 16 sessions doesn't help: only 13/500 offloaded (cache-gate
correct for cold turn-1), kv_both adds ~16% TPOT overhead at high concurrency.

Load scaling: 1000req_ts20, 200req_ts10, 200req_ts5, 500req_ts10 all show
~30% GPU util at 8 sessions. The bottleneck is max_inflight_sessions, not
arrival rate.

Updated elastic_hypotheses.md with H8, H9, and comprehensive final analysis.
The real bottleneck is vLLM's chunked prefill scheduling, not routing or
PD disaggregation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

analysis/elastic_hypotheses.md

@@ -185,9 +185,88 @@ OK/N:              198/200         198/200        ← same reliability
 5. **GPU balance improvement → HEAVY TTFT -10.5%**: validated (H4 HEAVY_COLO data)
 6. **The bottleneck is at time_scale=20 with 200 req**: system is only 30% loaded. Higher load may reveal more optimization opportunities.
 
-## Next directions
+---
 
-- **Higher load (time_scale=10, 500+ req)**: increase contention to amplify routing differences
-- **1000 req at time_scale=20**: reduce statistical noise (±7% → ±3%)
-- **Session-count-aware placement**: balance number of active heavy sessions per instance, not just ongoing tokens
-- **Layerwise KV transfer**: modify Mooncake to pipeline KV transfer with compute (requires deep vLLM change)
+## H8: Higher concurrency reveals prefill-decode interference
+
+**Claim**: At 8 sessions / 8 GPUs, the system is underloaded (30% GPU util). Increasing to 16 sessions should reveal prefill-decode interference.
+
+**Experiments**:
+- 8 sessions, ts=20, 1000 req: TPOT90=0.073, GPU=30%, imbal=1.5x
+- 16 sessions, ts=10, 500 req: TPOT90=0.106, GPU=~25%, imbal=~3.5x
+- 32 sessions, ts=10, 500 req: (not run yet)
+
+**Result**:
+```
+                8 sessions      16 sessions     Delta
+TPOT p90:       0.0729          0.1058          +45%!
+WARM TPOT90:    0.0640          0.1301          +103%!
+MEDIUM TPOT90:  0.0750          0.1970          +149%!
+HEAVY TTFT50:   (varies)        3.399           —
+E2E p50:        4.516           5.830           +29%
+```
+
+**Verdict**: **VALIDATED**. 16 sessions creates real prefill-decode interference. MEDIUM TPOT degrades 2.5x because HEAVY prefills (via chunked prefill) block decode steps on the same GPU. This is the scenario where PD disaggregation should theoretically help.
+
+---
+
+## H9: Elastic RDMA offload at 16 sessions reduces interference
+
+**Claim**: At 16 sessions where interference is severe, elastic V2 (C_s prefill + flexible D decode via RDMA) should reduce TPOT by isolating heavy prefill from decode.
+
+**Experiment**: 16 sessions, 500 req, elastic (kv_both + H4 cache-gate)
+
+**Result**:
+```
+                Baseline 16s    Elastic 16s     Delta
+TPOT p90:       0.1058          0.1231          +16% (WORSE)
+MEDIUM TPOT90:  0.1970          0.2056          +4% (same)
+TTFT p50:       0.828           0.937           +13% (WORSE)
+E2E p50:        5.830           6.528           +12% (WORSE)
+OK/N:           498/500         498/500         same
+Offloaded:      —               13/500 (2.6%)   too few to matter
+```
+
+**Verdict**: **REJECTED**. Elastic at 16 sessions is WORSE, not better. Root causes:
+1. Cache-gate correctly blocks 89% of HEAVY (cold turn-1, cache_ratio=0) → only 13 offloads
+2. kv_both runtime overhead at high concurrency adds ~16% TPOT vs plain baseline
+3. The 13 offloaded requests have TTFT p50=17.5s (RDMA overhead), much worse than colocated 3.5s
+
+**Key learning**: The RDMA transfer approach cannot solve prefill-decode interference because:
+- Most HEAVY are cold (no cache to benefit from offload)
+- Mooncake lacks layerwise transfer (RDMA is pure sequential overhead after prefill)
+- kv_both has non-zero overhead at high concurrency (contradicts Phase 0 at low concurrency)
+
+---
+
+## Current Understanding (final)
+
+### What DOESN'T work for agentic workloads:
+
+1. **PD-Sep**: net negative — KV cache memory wall on decode instances
+2. **LMetric (OSDI'26)**: ≈ linear routing — session affinity limits routing freedom
+3. **Elastic P2P RDMA offload**: net negative — Mooncake transfer overhead, no layerwise pipeline
+4. **OVERLOAD_FACTOR tuning**: no effect — imbalance from workload skew, not routing
+5. **Dedicated Prefill Service (PS)**: cannot win cost comparison without KV pull, PS is always slower than cached C
+6. **Cache-gate offload (H4)**: correct but only 10-12% of HEAVY have cache → limited activation
+
+### What DOES work:
+
+1. **Cache-aware session-sticky routing**: +24pp APC, -60% TTFT vs round-robin (the dominant optimization)
+2. **GPU balance from offload routing**: HEAVY_COLO -10.5% TTFT when imbalance reduced (H4 data)
+
+### The real bottleneck:
+
+At production-level concurrency (>1 session/GPU), the dominant bottleneck is **chunked prefill interference**: large HEAVY prefill chunks block decode steps on the same GPU, causing TPOT to degrade 45-149%.
+
+Neither routing nor RDMA-based PD disaggregation solves this. The root cause is vLLM's scheduler design:
+- Chunked prefill chunk size (`max_num_batched_tokens`, default 8192) is fixed
+- Large prefill chunks monopolize the GPU for tens of ms, stalling decode
+- Reducing chunk size would improve decode responsiveness but increase prefill overhead
+
+### Next direction: Adaptive chunked prefill scheduling
+
+Instead of fixed chunk size, dynamically adjust based on decode pressure:
+- When decode queue is deep: smaller chunks → more decode slots → better TPOT
+- When decode queue is empty: larger chunks → faster prefill → better TTFT
+- This is a vLLM scheduler modification, not a routing change