agentic-kvc

Go to file

Gahow Wang efe984477a Balanced routing result: APC +4pp but latency +23% (cache-load tradeoff)

Balanced session-sticky routing improves APC from 44.7% to 48.7% (+4pp,
close to simulated 49.2%) but TTFT worsens by 30% and E2E by 23%.

Root cause: session-sticky creates load hotspots — some instances get
multiple heavy concurrent sessions, causing queue delays, despite higher
per-instance APC.

Key finding: APC optimization and latency optimization are in tension.
  - Cache affinity (sticky) -> higher APC, worse load balance -> worse latency
  - Load-based routing (old) -> lower APC, better load balance -> better latency

The optimal design must balance both dimensions, not optimize one at
the expense of the other.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-22 02:13:15 +08:00

analysis

KV cache lifecycle design + eviction loss analysis

2026-05-22 01:27:22 +08:00

patches

Add vLLM patches directory for version-controlled patch management

2026-05-22 00:26:14 +08:00

replayer

Balanced session-sticky routing + agentic workload pattern analysis

2026-05-22 01:50:27 +08:00

scripts

Balanced routing result: APC +4pp but latency +23% (cache-load tradeoff)