2 Commits

Author	SHA1	Message	Date
Gahow Wang	663ca9c5b9	Support compute_dtype for FP4/FP8 tensor core FLOPS selection ... Add `compute_dtype` field to ModelConfig ("bf16", "fp8", "fp4") which controls two things: - GPU FLOPS tier: auto-selects from preset FP4/FP8/BF16 TFLOPS - Weight bytes: uses 0.5/1.0/2.0 bytes per param for memory-bound check Hardware presets now include per-GPU FP8 and FP4 dense FLOPS for all GPUs that support them (H100/H800/H20: FP8, B200/B300: FP8+FP4). Config resolution auto-selects the right FLOPS when compute_dtype is set and the user hasn't explicitly overridden gpu_flops. GLM-5-NVFP4 on 8xB300 now correctly uses 13.5 PFLOPS/GPU FP4 (6x faster prefill) and 0.5 bytes/param weights (halved memory footprint). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-04-14 11:54:10 +08:00
Gahow Wang	ec73a95e05	KVCache simulator for LLM serving cluster routing research ... Discrete-event simulator for evaluating KV cache-aware routing policies in prefill-disaggregated LLM serving clusters. Models a two-tier KV cache hierarchy (L0 GPU HBM + L1 CPU DRAM) with RDMA/PCIe link contention, architecture-derived roofline compute (MoE, MLA, DSA), and a cluster-wide meta-store for prefix-aware routing decisions. Includes 11 routing policies (random, round_robin, least_loaded, least_tokens, ttl_aware, precise, min_pd, cache_load, cache_score, estimated_ttft, prefix_affinity), HuggingFace config.json auto-parsing, built-in GPU hardware presets (H100/H800/H20/A100/B200), and ablation tooling for systematic policy comparison across real Alibaba serving traces. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-04-14 01:16:02 +08:00