kvcache-simulator

gahow/kvcache-simulator

Fork 0

Commit Graph

Author	SHA1	Message	Date
Gahow Wang	84696604e8	Add B300 GPU preset and GLM-5-NVFP4 on 8xB300 config Add NVIDIA B300 (Blackwell Ultra) to hardware presets: same GB202 die as B200 (2.25 PFLOPS BF16 dense) but with HBM3e 12-Hi stacks (288 GB, 12 TB/s — 50% more capacity and bandwidth than B200). Add nvidia/GLM-5-NVFP4 HuggingFace config.json and a matching simulation config for 8xB300: FP4 weights (~372 GB) leave ~1.9 TB for KV cache, yielding 82k blocks per instance (3.8x more than the BF16-on-B200 setup). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-04-14 11:37:20 +08:00
Gahow Wang	ec73a95e05	KVCache simulator for LLM serving cluster routing research Discrete-event simulator for evaluating KV cache-aware routing policies in prefill-disaggregated LLM serving clusters. Models a two-tier KV cache hierarchy (L0 GPU HBM + L1 CPU DRAM) with RDMA/PCIe link contention, architecture-derived roofline compute (MoE, MLA, DSA), and a cluster-wide meta-store for prefix-aware routing decisions. Includes 11 routing policies (random, round_robin, least_loaded, least_tokens, ttl_aware, precise, min_pd, cache_load, cache_score, estimated_ttft, prefix_affinity), HuggingFace config.json auto-parsing, built-in GPU hardware presets (H100/H800/H20/A100/B200), and ablation tooling for systematic policy comparison across real Alibaba serving traces. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-04-14 01:16:02 +08:00

Author

SHA1

Message

Date

Gahow Wang

84696604e8

Add B300 GPU preset and GLM-5-NVFP4 on 8xB300 config

Add NVIDIA B300 (Blackwell Ultra) to hardware presets: same GB202 die as
B200 (2.25 PFLOPS BF16 dense) but with HBM3e 12-Hi stacks (288 GB,
12 TB/s — 50% more capacity and bandwidth than B200).

Add nvidia/GLM-5-NVFP4 HuggingFace config.json and a matching simulation
config for 8xB300: FP4 weights (~372 GB) leave ~1.9 TB for KV cache,
yielding 82k blocks per instance (3.8x more than the BF16-on-B200 setup).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-04-14 11:37:20 +08:00

Gahow Wang

ec73a95e05

KVCache simulator for LLM serving cluster routing research

Discrete-event simulator for evaluating KV cache-aware routing policies
in prefill-disaggregated LLM serving clusters. Models a two-tier KV cache
hierarchy (L0 GPU HBM + L1 CPU DRAM) with RDMA/PCIe link contention,
architecture-derived roofline compute (MoE, MLA, DSA), and a cluster-wide
meta-store for prefix-aware routing decisions.

Includes 11 routing policies (random, round_robin, least_loaded,
least_tokens, ttl_aware, precise, min_pd, cache_load, cache_score,
estimated_ttft, prefix_affinity), HuggingFace config.json auto-parsing,
built-in GPU hardware presets (H100/H800/H20/A100/B200), and ablation
tooling for systematic policy comparison across real Alibaba serving traces.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-04-14 01:16:02 +08:00

2 Commits