Gahow Wang
9c105cf05a
Reuse and concurrency axes redone with proper controlled variables, plus
the orchestration used to run them on dash0:
- run_reuse_fixed.sh: hold REAL prefill work (delta) constant, vary only
cached prefix -> reuse = C/(C+U). Supersedes old fig1 (which held
input=8192 and sliced prefix out, confounding "more reuse" with "less
prefill").
- run_conc.sh: agentic-corner config (in=32768, delta=512, reuse=0.984,
out=128) that exposes PD's structural KV-transfer tax. Supersedes old fig3.
- run_campaign{,2,3}.sh, backfill_d2048o128.sh: serial campaign drivers
(strictly one driver at a time), out=128 sweeps, PD wall-cap for
collapse-draining high-reuse arms, and flaked-arm backfill.
- mb5_run_gpu.sh: per-config bring-up / replay / teardown orchestrator.
- plot_pd_crossover.py: render the reuse_compare figures from fig_agg dumps.
- fig_agg.py: tolerate null stats from fully-collapsed arms (0 successes
write the stat keys as null; `dict.get(k, {})` returns null, not {}).
Data: fig1_reuse_fixed.json, fig1_reuse_d{1024,2048}_o128.json
Figs: reuse_compare_AB.png, reuse_compare_ABC.png
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
agentic-kv
Serving agentic LLM workloads by keeping the KV working set in GPU HBM
(GPU-hit-first). Research outline: PAPER_OUTLINE.md.
Evidence + experiments: v2/.
⚠️ Benchmarking methodology — read this first
Replay agentic traces with
--dispatch-mode thinktime, not the defaulttracets. It is the faithful, more realistic load — and the dispatch mode materially changes the performance you measure.
The replayer offers two ways to time each turn:
| mode | turn-k dispatched at | what it models |
|---|---|---|
tracets (default) |
max(prev_turn_finished, trace_ts) |
absolute production schedule |
thinktime (use this) |
prev_turn_finished + time_to_parent_chat |
real closed-loop agent pacing |
Why it matters. tracets collapses the inter-turn think-time to ~0 whenever
the system falls behind (it fires the next turn immediately because the trace
timestamp is already in the past). That manufactures artificial request
bursts — spiking instantaneous concurrency → KV-pool pressure → preemption →
inflated tail latency and wasted throughput. thinktime keeps each turn's real
gap (tool-exec + agent think), so the offered load is what a real agent produces.
Measured (w600 first-300s window, 8×H20, round-robin, 100% completion):
| metric (N=8) | tracets (Mode 1) |
thinktime (Mode 2) |
Δ |
|---|---|---|---|
| E2E p90 | 102.8 s | 73.5 s | −28% |
| E2E p99 | 245 s | 227 s | −7% |
| TTFT p90 | 56.1 s | 39.7 s | −29% |
| system TPS | 111.8 | 119.3 | +7% |
| wall-clock | 967 s | 787 s | −19% |
| TPOT p90 | 0.174 s | 0.188 s | ~flat |
So under realistic capacity, tracets makes the system look ~30% worse on
tail latency than it actually is. Tell-tale: scaling 6→8 instances barely helped
tracets (975→967 s — its bursts re-saturate regardless of capacity) but helped
thinktime a lot (1125→787 s). Under heavy saturation (N=6) the two converge
(E2E p90 ≈ 118–120 s), since there is no slack for bursts to harm. Decode (TPOT)
is dispatch-independent everywhere.
Recommendation: benchmark with --dispatch-mode thinktime; use tracets
only as an explicit bursty stress case. Full ablation:
v2/exp_c_dispatch_ablation/.
How to use it
# 1. annotate a trace with the real per-turn gap (one-time; scans the raw trace)
python scripts/add_time_to_parent.py traces/w600_r0.0015_st30.jsonl traces/w600_ttp.jsonl
# 2. replay closed-loop with faithful think-time
python -m replayer --trace traces/w600_ttp.jsonl --endpoint <eps> \
--model <model> --dispatch-mode thinktime
time_to_parent_chat = this_turn.request_ready_time_ms − parent_turn.request_end_time_ms,
computed from the raw trace and stored per request; turn-1 has none (fires at its
trace arrival). Traces without the field fall back to tracets.
Project map
PAPER_OUTLINE.md— GPU-hit-first paper outline (the thesis).v2/— evidence experiments:exp_a_tier_latency/— KV-hit cost by tier (GPU < CPU-local < remote-RDMA < miss).exp_b_capacity_knee/— realized APC / latency knee vs GPU capacity.exp_c_dispatch_ablation/— the replay-mode study above.
replayer/— trace replayer (--dispatch-mode, closed-loop think-time).scripts/add_time_to_parent.py— trace annotation forthinktime.microbench/,analysis/— PD-disagg, routing, workload characterization.