agentic-kvc

Go to file

Gahow Wang 10636b1ab1 KV cache lifecycle design + eviction loss analysis

Root cause of 10.1pp APC gap: multi-turn sessions' KV evicted between
turns by cold-start prefills (66% of loss). Inter-turn gap is only 2
requests p50, but LRU cache (550 blocks) can't protect 93 blocks/session
across 14-21 concurrent sessions.

Three approaches designed:
  A. Session-sticky routing with KV reservation (proxy-only, no vLLM change)
  B. Two-tier KV cache: GPU + DRAM offload via Mooncake
  C. Prefill-aware eviction (LFU/ARC instead of LRU, vLLM patch)

Next: simulate LRU vs LFU vs "infinite-for-MT" to quantify upper bounds,
then implement Approach A (lowest effort, immediate benchmark).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-22 01:27:22 +08:00

analysis

KV cache lifecycle design + eviction loss analysis

2026-05-22 01:27:22 +08:00

patches

Add vLLM patches directory for version-controlled patch management

2026-05-22 00:26:14 +08:00

replayer

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

scripts

KV cache lifecycle design + eviction loss analysis