agentic-kvc

Go to file

Gahow Wang be273f7f27 Replace static offload gate with runtime cost model

Old gate: cache_ratio >= 0.3 (static, only 14% of HEAVY triggered)
New gate: offload when offload_cost < colocated_cost, where:
  colocated_cost = queue(C_s) + prefill(new_tokens)
  offload_cost = queue(P_idle) + prefill(P_tokens) + RDMA_overhead

Key changes:
- P is now least-loaded instance (not session-sticky C_s)
- Gate considers C_s queue depth dynamically
- Crossover: offload wins when C_s queue >= 38k tokens (~5.4s)
- Cold HEAVY requests CAN be offloaded if C_s is busy enough
- P accounting uses P's actual cache hit, not C_s's

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-23 19:42:33 +08:00

analysis

Add comprehensive research findings document

2026-05-23 07:16:31 +08:00

experiments

Add elastic PS evaluation plan for production-realistic trace

2026-05-23 15:56:05 +08:00

patches

Add vLLM patches directory for version-controlled patch management

2026-05-22 00:26:14 +08:00

replayer

Fix replay methodology: trace-driven dispatch, no artificial limits