Gahow Wang
ec73a95e05
Discrete-event simulator for evaluating KV cache-aware routing policies in prefill-disaggregated LLM serving clusters. Models a two-tier KV cache hierarchy (L0 GPU HBM + L1 CPU DRAM) with RDMA/PCIe link contention, architecture-derived roofline compute (MoE, MLA, DSA), and a cluster-wide meta-store for prefix-aware routing decisions. Includes 11 routing policies (random, round_robin, least_loaded, least_tokens, ttl_aware, precise, min_pd, cache_load, cache_score, estimated_ttft, prefix_affinity), HuggingFace config.json auto-parsing, built-in GPU hardware presets (H100/H800/H20/A100/B200), and ablation tooling for systematic policy comparison across real Alibaba serving traces. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
30 lines
692 B
Rust
30 lines
692 B
Rust
use anyhow::Result;
|
|
use std::fs::File;
|
|
use std::io::{BufWriter, Write};
|
|
use std::path::Path;
|
|
|
|
use crate::router::RouteDecision;
|
|
|
|
pub struct RoutingLogWriter {
|
|
inner: BufWriter<File>,
|
|
}
|
|
|
|
impl RoutingLogWriter {
|
|
pub fn create<P: AsRef<Path>>(path: P) -> Result<Self> {
|
|
let f = File::create(path)?;
|
|
Ok(Self { inner: BufWriter::new(f) })
|
|
}
|
|
|
|
pub fn write(&mut self, decision: &RouteDecision) -> Result<()> {
|
|
let line = serde_json::to_string(decision)?;
|
|
self.inner.write_all(line.as_bytes())?;
|
|
self.inner.write_all(b"\n")?;
|
|
Ok(())
|
|
}
|
|
|
|
pub fn finish(mut self) -> Result<()> {
|
|
self.inner.flush()?;
|
|
Ok(())
|
|
}
|
|
}
|