Gahow Wang
32f7f55990
Follow-up to the LMetric sweep: rerun with --policy linear (cache-aware load + sticky session affinity, the cache_aware_proxy default) and cap each PD-disagg arm at 2x the colo bench wall (SIGTERM bench.sh once cap is exceeded; the cleanup trap clears vLLM and proxy; capped runs lack metrics.summary.json so the analysis script computes from raw metrics.jsonl). Headline: the success-rate ceiling is policy-invariant. arm linear (capped at 2x) lmetric (uncapped) colo 807/807 = 100%, 964s 807/807 = 100%, 1021s pd6 (6:2) 472/807 = 58%, 2280s ⊗ 474/807 = 59%, 3325s pd4 (4:4) 349/807 = 43%, 2281s ⊗ 348/807 = 43%, 6850s pd2 (2:6) 176/807 = 22%, 2280s ⊗ 180/807 = 22%, 19275s Routing affects only how much wall is wasted timing out unreachable requests at 600s each. Linear hits the same ceiling in 2280s as LMetric does in 3300-19000s. This *strengthens* the §5 D-pool capacity-ceiling thesis -- the cap is structural, not a routing artifact. Artifacts: analysis/v2/fig4r_linear.json -- 4-arm linear summary analysis/v2/PD_DISAGG_LMETRIC.md -- extended with wall-cap section figs/v2/fig4_linear_vs_lmetric.png -- 3-panel side-by-side comparison microbench/fresh_setup/plot_fig4_linear_vs_lmetric.py
agentic-kv
Serving agentic LLM workloads by keeping the KV working set in GPU HBM
(GPU-hit-first). Research outline: PAPER_OUTLINE.md.
Evidence + experiments: v2/.
⚠️ Benchmarking methodology — read this first
Replay agentic traces with
--dispatch-mode thinktime, not the defaulttracets. It is the faithful, more realistic load — and the dispatch mode materially changes the performance you measure.
The replayer offers two ways to time each turn:
| mode | turn-k dispatched at | what it models |
|---|---|---|
tracets (default) |
max(prev_turn_finished, trace_ts) |
absolute production schedule |
thinktime (use this) |
prev_turn_finished + time_to_parent_chat |
real closed-loop agent pacing |
Why it matters. tracets collapses the inter-turn think-time to ~0 whenever
the system falls behind (it fires the next turn immediately because the trace
timestamp is already in the past). That manufactures artificial request
bursts — spiking instantaneous concurrency → KV-pool pressure → preemption →
inflated tail latency and wasted throughput. thinktime keeps each turn's real
gap (tool-exec + agent think), so the offered load is what a real agent produces.
Measured (w600 first-300s window, 8×H20, round-robin, 100% completion):
| metric (N=8) | tracets (Mode 1) |
thinktime (Mode 2) |
Δ |
|---|---|---|---|
| E2E p90 | 102.8 s | 73.5 s | −28% |
| E2E p99 | 245 s | 227 s | −7% |
| TTFT p90 | 56.1 s | 39.7 s | −29% |
| system TPS | 111.8 | 119.3 | +7% |
| wall-clock | 967 s | 787 s | −19% |
| TPOT p90 | 0.174 s | 0.188 s | ~flat |
So under realistic capacity, tracets makes the system look ~30% worse on
tail latency than it actually is. Tell-tale: scaling 6→8 instances barely helped
tracets (975→967 s — its bursts re-saturate regardless of capacity) but helped
thinktime a lot (1125→787 s). Under heavy saturation (N=6) the two converge
(E2E p90 ≈ 118–120 s), since there is no slack for bursts to harm. Decode (TPOT)
is dispatch-independent everywhere.
Recommendation: benchmark with --dispatch-mode thinktime; use tracets
only as an explicit bursty stress case. Full ablation:
v2/exp_c_dispatch_ablation/.
How to use it
# 1. annotate a trace with the real per-turn gap (one-time; scans the raw trace)
python scripts/add_time_to_parent.py traces/w600_r0.0015_st30.jsonl traces/w600_ttp.jsonl
# 2. replay closed-loop with faithful think-time
python -m replayer --trace traces/w600_ttp.jsonl --endpoint <eps> \
--model <model> --dispatch-mode thinktime
time_to_parent_chat = this_turn.request_ready_time_ms − parent_turn.request_end_time_ms,
computed from the raw trace and stored per request; turn-1 has none (fires at its
trace arrival). Traces without the field fall back to tracets.
Project map
PAPER_OUTLINE.md— GPU-hit-first paper outline (the thesis).v2/— evidence experiments:exp_a_tier_latency/— KV-hit cost by tier (GPU < CPU-local < remote-RDMA < miss).exp_b_capacity_knee/— realized APC / latency knee vs GPU capacity.exp_c_dispatch_ablation/— the replay-mode study above.
replayer/— trace replayer (--dispatch-mode, closed-loop think-time).scripts/add_time_to_parent.py— trace annotation forthinktime.microbench/,analysis/— PD-disagg, routing, workload characterization.