agentic-kvc/analysis/research_findings.md

Research Findings: KV Cache Optimization for Agentic LLM Workloads

Date: 2026-05-23 Author: Gahow Wang

---

1. Agentic Workload Characteristics (vs Chatbot/API)

Property	Chatbot/API	Agentic (this work)
Input length	1-5k tokens	avg 33.6k, p90 88k
Output length	100-500 tokens	avg 445 (similar)
I/O ratio	1-10x	75.6x
Prefill token share	50-70%	98%
KV reuse	Low (independent requests)	71% theoretical, 91% intra-session
Session structure	Mostly single-turn	Multi-turn chains (parent_chat_id)
Request weight distribution	Uniform	Bimodal: 49% WARM/MEDIUM, 51% HEAVY

These characteristics fundamentally change what optimizations matter.

2. What Doesn't Work (and Why)

2.1 PD Disaggregation (DistServe/Splitwise approach)

Setup: 4P + 4D instances (Mooncake RDMA KV transfer) Result: TTFT +72%, TPOT +1%, APC -5pp vs combined

Root cause: KV cache memory wall on decode instances. With avg 33.6k input and dedicated decode instances:

• Decode KV cache fills to 97.1%
• GPU idle (Running: 0), but new requests queue for KV cache memory
• 87.7% of TTFT is spent waiting for KV cache space

Why it's different from chatbot: Chatbot has short context (1-5k), so decode KV cache rarely fills. Agentic has 33k+ context, requiring 4-8GB KV per request → 2-3 concurrent requests saturate a single GPU's KV cache.

2.2 LMetric (OSDI'26, P_tokens × BS multiplication routing)

Setup: 8 instances, LMetric vs linear routing Result: TTFT +2.2%, TPOT -4.4%, E2E +2.6% — all within noise (±7% run-to-run)

Root cause (updated): LMetric is not "neutralized by affinity constraints" — pure --policy lmetric runs without session affinity at all. The actual reason the LMetric vs linear comparison sits within noise is that P_tokens already includes new_uncached_tokens = input_length - cache_hit, which means later turns of a session naturally score lowest on the instance that cached their prefix. This gives LMetric an implicit soft affinity that competes with linear's explicit sticky affinity. The two arrive at similar placements through different mechanisms.

This is also why explicit migration buys little on top of LMetric: the first-order signal driving placement is already cache-derived. See docs/migration-policy-design.md for how the current hybrid policy uses this insight (LMetric base + explicit affinity only when cache_ratio > 0.5).

Previous framing (incorrect): an earlier draft of this section attributed the result to session affinity constraining LMetric's routing freedom. That framing assumed --policy lmetric inherited the linear-mode session-sticky behavior, which it does not (verified in tests/test_proxy_pick.py).

2.3 Elastic P2P RDMA Offload (Heavy prefill on different instance)

Setup: 8 instances (kv_both), HEAVY requests prefilled on different instance, KV transferred via Mooncake RDMA Result: E2E +37%, TPOT +11.6% — significantly worse

Root causes:

1. Mooncake lacks layerwise KV transfer: All blocks transferred after prefill completes (sequential, not pipelined). Transfer p50=1.1s for 40k tokens, highly variable (R²=0.095 vs input length).
2. 92% of HEAVY are turn-1 cold: No cache to exploit on the P instance → full cold prefill is always slower than co-located cached prefill.
3. kv_both has non-zero overhead at high concurrency: Zero overhead at idle (Phase 0), but +16% TPOT at 16 sessions (background RDMA threads compete for resources).

2.4 Dedicated Prefill Service

Setup: 1 PS (no sessions) + 7 C (session-sticky) Result: PS either gets 0 offloads (cost model correctly identifies it's more expensive) or gets too many (cascading timeouts)

Root cause: Without KV pull from C, PS does cold prefill (full input) which is always slower than C's cached prefill. With KV pull, double RDMA transfer overhead negates the benefit.

2.5 Chunk Size Tuning (max_num_batched_tokens)

Setup: 2048/4096/8192/16384 at 16 sessions Result: Default 8192 is optimal; smaller chunks add scheduler overhead, larger chunks help HEAVY but hurt overall

2.6 OVERLOAD_FACTOR Tuning

Setup: 2.0/1.5/1.3/1.0 session affinity breaking threshold Result: No effect — imbalance from workload skew, not routing

3. What DOES Work

3.1 Cache-Aware Session-Sticky Routing (the dominant optimization)

Setup: score = ongoing_tokens - α × cache_hit_tokens, session affinity for turn 2+ Result vs round-robin:

Metric	Round-Robin	Cache-Aware	Delta
TTFT p50	1.836s	0.731s	-60%
TPOT p90	0.086s	0.073s	-15%
APC	20.8%	44.7%	+24pp

Why it works for agentic: 91% of KV reuse is intra-session. Session-sticky routing ensures subsequent turns find their KV cache on the same instance. Cache-aware scoring steers turn-1 requests to instances with matching system prompt blocks (47% of blocks are shared across sessions).

3.2 GPU Balance → Latency Improvement (H4 evidence)

When GPU imbalance was reduced from 4.0x to 2.0x (via H4 cache-gate routing):

• HEAVY_COLO TTFT: 7.02s → 6.28s (-10.5%)
• No TPOT regression

Mechanism: Hot GPU (63.3% util) causes queuing delays for co-located requests. Spreading load more evenly eliminates the queuing bottleneck.

Limitation: Only demonstrated for the 52/60 HEAVY requests that stayed co-located. The 8 offloaded requests had RDMA overhead. A routing-only approach to achieve balance (without RDMA) would be ideal.

4. System-Level Insights

4.1 Prefill-Decode Interference Threshold

Concurrency	TPOT p90	MEDIUM TPOT p90	GPU Util
8 sessions (1/GPU)	0.073	0.075	30%
16 sessions (2/GPU)	0.106 (+45%)	0.197 (+163%)	25%

At >1 session per GPU, chunked prefill interference becomes significant. MEDIUM requests' TPOT degrades 2.5x because HEAVY prefill chunks block their decode steps.

4.2 Mooncake Transfer Engine Limitations

• No layerwise transfer: All KV blocks transferred after full prefill → pure sequential overhead
• High variance: R²=0.095 (transfer time uncorrelated with data size), bimodal distribution (0.6s or 18-30s)
• Zero idle overhead: kv_both mode has no cost when not transferring (Phase 0 validated)
• Non-zero overhead at high concurrency: +16% TPOT at 16 sessions (background threads)

4.3 KV Cache Reuse Structure

Total trace: 2.1M requests, 71% theoretical APC
Reuse breakdown:
  91% intra-session (same session, subsequent turns)
   4.8% cross-session (shared system prompts)
   4.2% unique (no reuse)

Effective APC achieved:
  Round-robin: 20.8% (destroys session locality)
  Cache-aware: 44.7% (preserves session locality)
  Theoretical max: 71% (infinite cache, single instance)
  Gap: 26pp from eviction + routing imperfection

4.4 Request Weight After Cache

A critical insight: the "weight" of a request for scheduling should be new_tokens = input - cached, not total_input.

Request	Total Input	After 70% Cache	Effective Weight
80k HEAVY	80k tokens	24k tokens	MEDIUM
30k MEDIUM	30k tokens	9k tokens	WARM
5k WARM	5k tokens	5k tokens	WARM

This changes the scheduling picture: most "HEAVY" requests in agentic workloads are effectively MEDIUM after cache — PD separation's premise (heavy prefill needs dedicated resources) doesn't apply.

5. Paper-Ready Summary

Agentic Workload Characteristics (vs prior LLM serving work):

1. Extreme I/O ratio (75x) → 98% of compute is prefill
2. High intra-session KV reuse (91%) → session affinity is critical
3. Bimodal request weight (51% HEAVY by input, but only ~15% HEAVY by new_tokens after cache)
4. Multi-turn session structure → routing decisions have long-term consequences (session migration destroys cache)

Why existing approaches don't work:

1. PD-Sep assumes decode needs dedicated resources → agentic has memory wall on decode
2. LMetric matches linear within noise because cache-hit appears in P_tokens itself, so it already routes later turns back to the cached instance via implicit soft affinity — explicit affinity buys little
3. Elastic RDMA assumes KV transfer is cheap → Mooncake lacks layerwise pipelining
4. Size-based classification assumes HEAVY = needs special handling → after cache, most HEAVY is MEDIUM

Our insights:

1. Cache-aware session-sticky routing is the dominant optimization: -60% TTFT, +24pp APC
2. Routing quality > PD separation > eviction policy (simulation verified: routing gives 24pp, eviction gives 1.8pp)
3. Effective request weight (new_tokens, not total_input) should drive scheduling
4. Prefill-decode interference only matters at >1 session/GPU (rarely reached in production clusters)
5. GPU balance improvement directly improves HEAVY TTFT (-10.5% demonstrated)

Quantitative improvements (this work vs vLLM default):

• TTFT p50: -60% (0.731s vs 1.836s) from cache-aware session-sticky routing
• APC: +24pp (44.7% vs 20.8%)
• TPOT p90: -15% (0.073s vs 0.086s) from reduced prefill-decode interference via better cache utilization