agentic-kvc/analysis/kv_lifecycle_design.md

KV Cache Lifecycle Management for Agentic Workloads

Date: 2026-05-22 Status: Design, pending implementation + experiments Context: PD separation's real issue is not P-D compute interference (cache-aware routing solves that), but KV cache eviction destroying multi-turn session state. This design addresses the root cause directly.

---

1. Problem: Multi-Turn KV Eviction

Our eviction analysis on 1000 sampled requests shows:

  Infinite cache APC:   53.4%
  LRU cache APC:        43.3%
  Gap:                  10.1 pp (3.2M tokens lost)
  
  Loss breakdown:
    Multi-turn (prior turn evicted): 66% of loss (6.7pp)
    Cross-session (shared prefix):   34% of loss (3.4pp)

The mechanism: a multi-turn session completes turn N (KV = ~93 blocks p50, ~47k tokens). Before turn N+1 arrives (gap = 2 requests p50, 12 requests p90), cold-start requests fill the LRU cache and evict turn N's KV. Turn N+1 arrives, finds zero cache hit, must re-prefill the entire context.

Key workload parameters:

Metric	Value
Multi-turn sessions	9% of sessions, but 66% of eviction loss
Inter-turn gap	p50=2 req, p90=12 req (very short)
KV to protect per session	p50=93 blocks (47k tokens)
Concurrent sessions needing protection	p50=14, max=21
Total protection budget needed	p50=3515 blocks (6.4x single-instance capacity)
Per-instance capacity	550 blocks

The challenge: protecting all concurrent multi-turn sessions' KV requires 3515 blocks, but each instance only has 550. Even spreading across 8 instances (4400 total blocks), it's tight at peak (4927 blocks needed).

2. Three Design Approaches

Approach A: Session-Sticky Routing with KV Reservation

Idea: Route all turns of a multi-turn session to the same instance. Reserve a fraction of each instance's KV cache for "protected" multi-turn sessions.

Instance KV layout (550 blocks):
  ┌──────────────────────────────────────────┐
  │  Protected zone (200 blocks)             │  ← Multi-turn session KV
  │  LRU eviction disabled here              │  ← Pinned by session affinity
  ├──────────────────────────────────────────┤
  │  Evictable zone (350 blocks)             │  ← Cold-start + overflow
  │  Normal LRU eviction                     │
  └──────────────────────────────────────────┘

Routing: Cache-aware + session-sticky. Multi-turn turn 2+ goes to the instance that served turn 1. Load-balance new sessions across instances.

KV protection: Not a vLLM change — implemented at the routing level. By concentrating a session's turns on one instance and ensuring the instance has enough cache headroom, the session's KV stays warm naturally (inter-turn gap is only 2 requests p50).

Budget: 21 concurrent multi-turn sessions / 8 instances ≈ 3 sessions per instance. At 93 blocks/session, that's ~280 blocks protected, leaving 270 blocks for cold starts.

Pros: No vLLM modification. Pure routing optimization. Cons: Instance load imbalance if multi-turn sessions cluster. Protected blocks may waste cache if session ends unexpectedly.

Experiment: Compare combined cache-aware (current) vs combined with aggressive session-sticky routing where multi-turn sessions are balanced across instances by their KV size.

Approach B: Two-Tier KV Cache (GPU + DRAM Offload)

Idea: When a multi-turn session's turn completes, offload its KV from GPU to DRAM. When the next turn arrives, reload from DRAM (faster than re-prefill). GPU cache is freed for cold starts.

  Turn N completes:
    GPU KV (hot) ──offload──> DRAM KV pool (warm)
    GPU cache freed for cold-start requests
  
  Turn N+1 arrives:
    DRAM KV pool ──reload──> GPU KV (hot)
    Skip prefill, go directly to decode
    
  Latency: DRAM reload ~1-10ms (PCIe/RDMA) vs re-prefill ~3-10s (compute)

Implementation: Use Mooncake's DRAM pool as a KV cache extension. Each instance runs with kv_role=kv_both. When the scheduler detects a turn completion for a multi-turn session, it triggers KV offload to DRAM. On next turn arrival, the scheduler triggers KV reload.

Budget: DRAM is much larger than GPU HBM. Each H20 has ~512GB system DRAM. 21 sessions × 93 blocks × 512 tokens × 48 layers × 2(K+V) × 128 dim × 2 bytes ≈ 24GB in DRAM — easily fits.

Pros: Decouples KV cache capacity from GPU HBM. DRAM reload is 100-1000x faster than re-prefill. Cons: Requires Mooncake integration. Offload/reload adds latency (but much less than re-prefill). vLLM changes needed for proactive offload trigger.

Experiment: Hard to implement quickly in vLLM. Can simulate the benefit by comparing: (a) current APC with eviction vs (b) APC if multi-turn sessions always hit cache (simulated infinite cache for multi-turn only).

Approach C: Prefill-Aware Eviction Policy

Idea: Replace LRU with a policy that considers session lifecycle. Blocks belonging to active multi-turn sessions get eviction priority boost.

  Standard LRU: evict oldest accessed block
  Session-aware: evict oldest accessed block THAT IS NOT part of an active session
  
  Active session: session with turn completed in last T seconds (or N requests)

Implementation: Modify vLLM's prefix cache eviction in third_party/vllm/. The eviction policy checks if a block's hash belongs to a known active session before evicting it.

The problem: vLLM's prefix cache uses block hashes, not session IDs. There's no direct mapping from block → session. We'd need to maintain a mapping at the scheduler level.

Alternative: Simpler proxy — just use block access frequency instead of pure LRU. Blocks that belong to system prompts (accessed by many requests) and multi-turn sessions (accessed repeatedly) naturally have higher frequency and survive eviction. This is LFU (Least Frequently Used) or ARC (Adaptive Replacement Cache).

Pros: Directly solves eviction at the cache layer. No routing changes needed. Cons: Requires vLLM source modification. Cache policy changes are subtle and may have side effects.

Experiment: Simulate LFU vs LRU on the trace to estimate APC improvement before implementing.

3. Feasibility and Experiment Priority

Approach	Implementation Effort	vLLM Changes	Expected APC Gain	Experiment
A: Session-sticky	Low (proxy only)	None	+3-5pp (multi-turn stays warm)	Run immediately
B: DRAM offload	High (Mooncake)	Medium	+6-7pp (all multi-turn recovered)	Simulate first
C: Eviction policy	Medium (vLLM patch)	Yes	+5-10pp (both MT and cross-session)	Simulate LFU vs LRU first

Recommended experiment order:

1. Simulate: LRU vs LFU vs "infinite-for-MT" on the trace → quantify upper bound
2. Approach A: Session-sticky routing with KV-size-balanced placement → real benchmark
3. Approach C: If simulation shows LFU helps, patch vLLM eviction policy → real benchmark
4. Approach B: If DRAM offload shows large benefit in simulation, implement with Mooncake

4. Relationship to PD Separation

These approaches are orthogonal to PD separation. They address KV cache lifecycle, not P-D compute interference:

• Approach A works in combined mode (no PD-Sep needed)
• Approach B could complement PD-Sep (offload from D to DRAM between turns)
• Approach C works in any mode

The key insight: for agentic workloads, KV cache management is a more impactful optimization axis than P-D compute separation. The 10.1pp APC gap from eviction translates to ~3.2M extra tokens of re-prefill per 1000 requests — far more overhead than P-D interference.

5. Combined Architecture Vision

The endgame combines all insights:

  ┌──────────────────────────────────────────────┐
  │            Global Scheduler                   │
  │  - Cache-aware + token-level LB               │
  │  - Session-sticky for multi-turn              │
  │  - KV-size-aware placement                    │
  └──────────────┬───────────────────────────────┘
                 │
  ┌──────────────┴───────────────────────────────┐
  │  8× PD-Combined Instances (TP=1)              │
  │                                               │
  │  Per-instance KV cache:                       │
  │    [Session-protected zone] [LFU evictable]   │
  │                                               │
  │  DRAM KV pool (Mooncake):                     │
  │    - Offloaded between-turn KV                │
  │    - Shared prefix blocks (system prompt)     │
  │    - Overflow buffer                          │
  └───────────────────────────────────────────────┘

All 8 GPUs do both P and D. The scheduler, cache policy, and DRAM pool work together to maximize APC and minimize prefill work — which is the real bottleneck for agentic workloads.