gahow/kvcache-simulator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
8 Commits 1 Branch 0 Tags
a3f386c8586b049269875aa48066f03986b6c5ed
Commit Graph

3 Commits

Author SHA1 Message Date
Gahow Wang
365ceac3be chore: update ablation and clean configs 2026-04-15 14:48:59 +08:00
Gahow Wang
8d41123418 Update README with full feature documentation 
Cover all 11 routing policies (including new prefix_affinity, cache_load,
cache_score, estimated_ttft, least_tokens), HuggingFace config.json
auto-parsing, GPU hardware presets, architecture-aware compute model
(MoE/MLA/DSA/GQA), router parameter tuning, bundled model configs and
config files, and available traces.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-14 01:21:28 +08:00
Gahow Wang
ec73a95e05 KVCache simulator for LLM serving cluster routing research 
Discrete-event simulator for evaluating KV cache-aware routing policies
in prefill-disaggregated LLM serving clusters. Models a two-tier KV cache
hierarchy (L0 GPU HBM + L1 CPU DRAM) with RDMA/PCIe link contention,
architecture-derived roofline compute (MoE, MLA, DSA), and a cluster-wide
meta-store for prefix-aware routing decisions.

Includes 11 routing policies (random, round_robin, least_loaded,
least_tokens, ttl_aware, precise, min_pd, cache_load, cache_score,
estimated_ttft, prefix_affinity), HuggingFace config.json auto-parsing,
built-in GPU hardware presets (H100/H800/H20/A100/B200), and ablation
tooling for systematic policy comparison across real Alibaba serving traces.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-14 01:16:02 +08:00