Replayer think-time dispatch mode + benchmarking guidance
Adds `--dispatch-mode {tracets,thinktime}` to the replayer and documents that
agentic serving should be benchmarked with `thinktime` (the faithful load).
- `tracets` (old default): turn-k at the absolute trace timestamp, i.e.
max(prev_finished, trace_ts) -- collapses inter-turn think-time to ~0 when the
system is behind, manufacturing request bursts.
- `thinktime`: turn-1 at trace arrival; turn-k at prev_finished +
time_to_parent_chat (real production gap). scripts/add_time_to_parent.py
annotates a trace with that gap from the raw trace's request_ready/end_ms.
exp(c) ablation (v2/exp_c_dispatch_ablation/): at N=8 (capacity slack) thinktime
beats tracets -- E2E p90 -28% (73.5 vs 102.8s), TTFT p90 -29%, TPS +7%, because
tracets' bursts spike concurrency -> KV pressure -> preemption. At N=6
(saturated) they converge. So tracets makes the system look ~30% worse on tail
latency than realistic agent pacing. Root README.md carries the headline
guidance; raw per-request metrics gitignored (perf_summary.json kept).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2
v2/.gitignore
vendored
2
v2/.gitignore
vendored
@@ -1,3 +1,3 @@
|
||||
# raw per-request replay dumps (~0.6 MB each) — regenerable; keep summary/m0/m1
|
||||
*/results/metrics_blk*.jsonl
|
||||
*/results/metrics_*.jsonl
|
||||
*/results/vllm_*.log
|
||||
|
||||
61
v2/exp_c_dispatch_ablation/README.md
Normal file
61
v2/exp_c_dispatch_ablation/README.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# exp (c) — Replay dispatch mode: `tracets` vs `thinktime`
|
||||
|
||||
Which replay mode should we benchmark agentic serving with, and how much does it
|
||||
change the measured performance?
|
||||
|
||||
**Two dispatch modes** (`replayer --dispatch-mode`):
|
||||
- **`tracets`** (default): turn-k at the absolute trace timestamp ⇒ effectively
|
||||
`max(prev_finished, trace_ts)`.
|
||||
- **`thinktime`**: turn-1 at trace arrival; turn-k at
|
||||
`prev_finished + time_to_parent_chat` (the REAL production gap; annotated by
|
||||
`scripts/add_time_to_parent.py` from the raw trace's
|
||||
`request_ready_time_ms`/`request_end_time_ms`).
|
||||
|
||||
Setup: w600 windowed to first 300 s (366 reqs, 223 multi-turn), round-robin across
|
||||
N H20 instances, both modes on the same instances, 100% completion throughout.
|
||||
|
||||
## Performance result
|
||||
|
||||
| metric | N6 tracets | N6 thinktime | N8 tracets | **N8 thinktime** |
|
||||
|---|---:|---:|---:|---:|
|
||||
| system TPS | 110.9 | 96.1 | 111.8 | **119.3** |
|
||||
| wall (s) | 975 | 1125 | 967 | **787** |
|
||||
| TTFT p50 / p90 / p99 | 4.4 / 61.8 / 135 | 4.5 / 83.7 / 130 | 2.9 / 56.1 / 115 | 3.1 / **39.7** / **83.5** |
|
||||
| TPOT p50 / p90 / p99 | .039 / .242 / .96 | .037 / .264 / .69 | .037 / .174 / .89 | .037 / .188 / .85 |
|
||||
| E2E p50 / p90 / p99 | 17.1 / 118 / 298 | 15.0 / 120 / 338 | 11.9 / 102.8 / 245 | 12.3 / **73.5** / **227** |
|
||||
|
||||
**At N=8 (capacity slack), `thinktime` is clearly better**: E2E p90 −28%, TTFT p90
|
||||
−29%, TPS +7%, wall −19%. **At N=6 (saturated) they converge** (E2E p90 ≈ 118–120 s).
|
||||
TPOT (decode) is dispatch-independent everywhere.
|
||||
|
||||
## Why — the mechanism (`figs/exp_c_dispatch_ablation.png`)
|
||||
|
||||
`tracets` collapses the realized inter-turn gap to ~0 under load (p50 0.00 s, 75%
|
||||
< 0.5 s) — it fires the next turn immediately because the trace timestamp is in the
|
||||
past. `thinktime` preserves the real gap (p50 1.22 s = the trace). The figure shows
|
||||
both realized-gap CDFs against the real `time_to_parent_chat`.
|
||||
|
||||
That gap-collapse manufactures **bursts** → peak concurrency spikes → KV-pool
|
||||
pressure → preemption → inflated tail latency + wasted throughput. The bursts
|
||||
re-saturate the system regardless of capacity, which is why scaling 6→8 instances
|
||||
barely helped `tracets` (975→967 s) but helped `thinktime` a lot (1125→787 s).
|
||||
Under saturation (N=6) there is no slack for bursts to harm, so the modes converge.
|
||||
|
||||
## Conclusion
|
||||
|
||||
Benchmark agentic serving with **`--dispatch-mode thinktime`** — it is the faithful
|
||||
closed-loop agent load and avoids the `tracets` burst artifact that makes the system
|
||||
look ~30% worse on tail latency than it is. Use `tracets` only as an explicit bursty
|
||||
stress case. (See the repo [`README.md`](../../README.md) for the headline guidance.)
|
||||
|
||||
Caveat: round-robin pays full prefill every turn (no cache reuse), so absolute
|
||||
latencies here are high; a cache-aware policy (LPWL) would lower them and likely
|
||||
widen the `thinktime` advantage. The raw window is also heavy (E2E in tens of
|
||||
seconds); a lighter load shows a healthier operating point.
|
||||
|
||||
## Repro
|
||||
```bash
|
||||
N=8 TRACE=traces/w600_ttp_win.jsonl bash v2/exp_c_dispatch_ablation/run_ablation.sh
|
||||
python v2/exp_c_dispatch_ablation/analyze.py traces/w600_ttp_win.jsonl \
|
||||
v2/exp_c_dispatch_ablation/results/metrics_{tracets,thinktime}.jsonl
|
||||
```
|
||||
115
v2/exp_c_dispatch_ablation/analyze.py
Normal file
115
v2/exp_c_dispatch_ablation/analyze.py
Normal file
@@ -0,0 +1,115 @@
|
||||
"""Mode 1 (tracets) vs Mode 2 (thinktime): wall-clock amplification + the
|
||||
think-collapse mechanism.
|
||||
|
||||
Mechanism (the smoking gun): realized inter-turn gap = this.t_dispatch_unix -
|
||||
prev.t_finish_unix, per session, consecutive turns. Mode 1 collapses it toward 0
|
||||
when the system is behind; Mode 2 holds it at the real time_to_parent_chat.
|
||||
|
||||
Amplification: wall / ideal_span. Mode 1 ideal = trace_span; Mode 2 ideal = the
|
||||
think-chain span under instant serving (analytic). If Mode 1 amplifies (>>1) with
|
||||
collapsed gaps while Mode 2 stays ~1 with preserved gaps, the 8x was a Mode-1
|
||||
artifact, not a workload property.
|
||||
|
||||
Usage: analyze.py <trace_ttp.jsonl> <m1.metrics.jsonl> <m2.metrics.jsonl> [fig.png]
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
|
||||
def pct(v, q):
|
||||
v = sorted(v)
|
||||
return v[min(int(q * len(v)), len(v) - 1)] if v else 0.0
|
||||
|
||||
|
||||
def load_trace(trace):
|
||||
rows = [json.loads(l) for l in open(trace)]
|
||||
span = max(r["timestamp"] for r in rows) - min(r["timestamp"] for r in rows)
|
||||
by = defaultdict(list)
|
||||
for r in rows:
|
||||
by[r["session_id"]].append(r)
|
||||
starts, ends, ttps = [], [], []
|
||||
for sid, t in by.items():
|
||||
t.sort(key=lambda r: (r["turn"], r["timestamp"]))
|
||||
s = t[0]["timestamp"]
|
||||
chain = sum((r.get("time_to_parent_chat") or 0.0) for r in t[1:])
|
||||
starts.append(s); ends.append(s + chain)
|
||||
ttps += [r["time_to_parent_chat"] for r in t[1:]
|
||||
if r.get("time_to_parent_chat") is not None]
|
||||
return span, (max(ends) - min(starts)), sorted(ttps)
|
||||
|
||||
|
||||
def realized_gaps(metrics):
|
||||
"""this.t_dispatch - prev.t_finish, per session, consecutive turns."""
|
||||
rows = [json.loads(l) for l in open(metrics)]
|
||||
ok = [r for r in rows if not r.get("error") and r.get("t_dispatch_unix")]
|
||||
by = defaultdict(list)
|
||||
for r in ok:
|
||||
by[r["session_id"]].append(r)
|
||||
gaps = []
|
||||
for sid, t in by.items():
|
||||
t.sort(key=lambda r: r["turn_id"])
|
||||
for a, b in zip(t, t[1:]):
|
||||
if a.get("t_finish_unix") and b.get("t_dispatch_unix"):
|
||||
gaps.append(max(0.0, b["t_dispatch_unix"] - a["t_finish_unix"]))
|
||||
return sorted(gaps), rows
|
||||
|
||||
|
||||
def wallclock(metrics):
|
||||
rows = [json.loads(l) for l in open(metrics)]
|
||||
ds = [r["t_dispatch_unix"] for r in rows if r.get("t_dispatch_unix")]
|
||||
fs = [r["t_finish_unix"] for r in rows if r.get("t_finish_unix")]
|
||||
return (max(fs) - min(ds)) if ds and fs else 0.0
|
||||
|
||||
|
||||
def e2e(metrics):
|
||||
ok = [json.loads(l) for l in open(metrics)]
|
||||
ok = [r for r in ok if not r.get("error")]
|
||||
return pct([r["latency_s"] for r in ok if r.get("latency_s")], .9), len(ok)
|
||||
|
||||
|
||||
def main():
|
||||
trace, m1, m2 = sys.argv[1:4]
|
||||
fig = sys.argv[4] if len(sys.argv) > 4 else "v2/figs/exp_c_dispatch_ablation.png"
|
||||
span, ideal2, ttps = load_trace(trace)
|
||||
g1, _ = realized_gaps(m1)
|
||||
g2, _ = realized_gaps(m2)
|
||||
w1, w2 = wallclock(m1), wallclock(m2)
|
||||
e1, n1 = e2e(m1)
|
||||
e2_, n2 = e2e(m2)
|
||||
|
||||
def collapsed(g): # fraction of realized gaps << the trace think-time median
|
||||
return sum(1 for x in g if x < 0.5) / len(g) if g else 0
|
||||
|
||||
print(f"trace_span={span:.0f}s mode2_ideal_span={ideal2:.0f}s "
|
||||
f"trace_ttp: p50={pct(ttps,.5):.2f}s frac<1s={sum(1 for x in ttps if x<1)/len(ttps):.0%}\n")
|
||||
print(f"{'mode':<16}{'wall_s':>8}{'amp/ideal':>10}{'e2e_p90':>9}"
|
||||
f"{'realgap_p50':>12}{'frac<0.5s':>10}")
|
||||
for name, w, ideal, e, n, g in [
|
||||
("Mode1 tracets", w1, span, e1, n1, g1),
|
||||
("Mode2 thinktime", w2, ideal2, e2_, n2, g2)]:
|
||||
print(f"{name:<16}{w:>8.0f}{w/ideal:>10.2f}{e:>9.1f}"
|
||||
f"{pct(g,.5):>12.2f}{collapsed(g):>10.0%}")
|
||||
|
||||
# figure: realized inter-turn gap CDF, trace vs Mode1 vs Mode2
|
||||
plt.figure(figsize=(7.2, 4.8))
|
||||
for data, lab, c in [(ttps, "trace time_to_parent_chat (real)", "#1f77b4"),
|
||||
(g2, "Mode2 thinktime: realized gap", "#2ca02c"),
|
||||
(g1, "Mode1 tracets: realized gap", "#d62728")]:
|
||||
if data:
|
||||
d = sorted(x for x in data if x >= 0)
|
||||
ys = [(i + 1) / len(d) for i in range(len(d))]
|
||||
plt.plot([max(x, 1e-3) for x in d], ys, label=lab, c=c, lw=2)
|
||||
plt.xscale("log"); plt.xlabel("inter-turn gap (s, log)"); plt.ylabel("CDF")
|
||||
plt.title("Mode 1 collapses the inter-turn gap under load; Mode 2 preserves it")
|
||||
plt.legend(); plt.grid(alpha=.3, which="both"); plt.tight_layout()
|
||||
plt.savefig(fig, dpi=140)
|
||||
print("wrote", fig)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
87
v2/exp_c_dispatch_ablation/results/perf_summary.json
Normal file
87
v2/exp_c_dispatch_ablation/results/perf_summary.json
Normal file
@@ -0,0 +1,87 @@
|
||||
{
|
||||
"setup": "w600 first-300s window (366 req, 223 multi-turn), round-robin x N H20, Qwen3-Coder-30B-A3B",
|
||||
"N6_tracets": {
|
||||
"n": 366,
|
||||
"ok": 366,
|
||||
"wall_s": 974.8,
|
||||
"tps": 110.9,
|
||||
"ttft": {
|
||||
"p50": 4.414,
|
||||
"p90": 61.791,
|
||||
"p99": 135.243
|
||||
},
|
||||
"tpot": {
|
||||
"p50": 0.039,
|
||||
"p90": 0.242,
|
||||
"p99": 0.958
|
||||
},
|
||||
"e2e": {
|
||||
"p50": 17.074,
|
||||
"p90": 118.02,
|
||||
"p99": 297.572
|
||||
}
|
||||
},
|
||||
"N6_thinktime": {
|
||||
"n": 366,
|
||||
"ok": 366,
|
||||
"wall_s": 1125.1,
|
||||
"tps": 96.1,
|
||||
"ttft": {
|
||||
"p50": 4.52,
|
||||
"p90": 83.662,
|
||||
"p99": 130.373
|
||||
},
|
||||
"tpot": {
|
||||
"p50": 0.037,
|
||||
"p90": 0.264,
|
||||
"p99": 0.694
|
||||
},
|
||||
"e2e": {
|
||||
"p50": 15.029,
|
||||
"p90": 119.68,
|
||||
"p99": 338.466
|
||||
}
|
||||
},
|
||||
"N8_tracets": {
|
||||
"n": 366,
|
||||
"ok": 366,
|
||||
"wall_s": 967.2,
|
||||
"tps": 111.8,
|
||||
"ttft": {
|
||||
"p50": 2.869,
|
||||
"p90": 56.128,
|
||||
"p99": 115.189
|
||||
},
|
||||
"tpot": {
|
||||
"p50": 0.037,
|
||||
"p90": 0.174,
|
||||
"p99": 0.89
|
||||
},
|
||||
"e2e": {
|
||||
"p50": 11.879,
|
||||
"p90": 102.849,
|
||||
"p99": 245.492
|
||||
}
|
||||
},
|
||||
"N8_thinktime": {
|
||||
"n": 365,
|
||||
"ok": 365,
|
||||
"wall_s": 787.0,
|
||||
"tps": 119.3,
|
||||
"ttft": {
|
||||
"p50": 3.099,
|
||||
"p90": 39.663,
|
||||
"p99": 83.524
|
||||
},
|
||||
"tpot": {
|
||||
"p50": 0.037,
|
||||
"p90": 0.188,
|
||||
"p99": 0.853
|
||||
},
|
||||
"e2e": {
|
||||
"p50": 12.256,
|
||||
"p90": 73.525,
|
||||
"p99": 227.295
|
||||
}
|
||||
}
|
||||
}
|
||||
53
v2/exp_c_dispatch_ablation/run_ablation.sh
Normal file
53
v2/exp_c_dispatch_ablation/run_ablation.sh
Normal file
@@ -0,0 +1,53 @@
|
||||
#!/bin/bash
|
||||
# Exp (c): does wall-clock amplification survive Mode 2 (real think-time)?
|
||||
# Launch N vLLM instances; replayer round-robins across them; replay the SAME
|
||||
# annotated trace under Mode 1 (tracets) and Mode 2 (thinktime).
|
||||
set -uo pipefail
|
||||
cd /home/admin/cpfs/wjh/agentic-kv
|
||||
PY=.venv/bin/python
|
||||
MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct
|
||||
N=${N:-4}
|
||||
TRACE=${TRACE:-traces/w600_ttp_win.jsonl}
|
||||
REQLIMIT=${REQLIMIT:-}
|
||||
OUT=v2/exp_c_dispatch_ablation/results
|
||||
mkdir -p "$OUT"
|
||||
PIDS=()
|
||||
EPS=""
|
||||
|
||||
launch() { # $1 gpu, $2 port
|
||||
CUDA_VISIBLE_DEVICES=$1 VLLM_LOGGING_LEVEL=WARNING \
|
||||
$PY -m vllm.entrypoints.openai.api_server --model "$MODEL" \
|
||||
--host 0.0.0.0 --port $2 --tensor-parallel-size 1 --trust-remote-code \
|
||||
--enable-prefix-caching --enforce-eager --dtype auto --max-model-len 200000 \
|
||||
--gpu-memory-utilization 0.9 > "$OUT/vllm_$2.log" 2>&1 &
|
||||
PIDS+=($!)
|
||||
}
|
||||
teardown() {
|
||||
for p in "${PIDS[@]:-}"; do kill -TERM "$p" 2>/dev/null; done
|
||||
sleep 6
|
||||
for p in $(pgrep -f "VLLM::EngineCore"); do kill -9 "$p" 2>/dev/null; done
|
||||
sleep 3
|
||||
}
|
||||
trap teardown EXIT
|
||||
|
||||
echo ">>> launch $N instances"
|
||||
for i in $(seq 0 $((N-1))); do
|
||||
launch "$i" $((8000+i))
|
||||
EPS="$EPS,http://127.0.0.1:$((8000+i))"
|
||||
done
|
||||
EPS="${EPS#,}"
|
||||
for i in $(seq 0 $((N-1))); do
|
||||
echo -n " wait health $((8000+i))..."
|
||||
timeout 900 bash -c "until curl -sf http://127.0.0.1:$((8000+i))/health >/dev/null 2>&1; do sleep 5; done" \
|
||||
&& echo ok || { echo FAIL; exit 1; }
|
||||
done
|
||||
|
||||
LIM=""; [ -n "$REQLIMIT" ] && LIM="--request-limit $REQLIMIT"
|
||||
for MODE in tracets thinktime; do
|
||||
echo "=== replay dispatch-mode=$MODE ==="
|
||||
$PY -m replayer --trace "$TRACE" --output "$OUT/metrics_$MODE.jsonl" \
|
||||
--endpoint "$EPS" --model "$MODEL" --dispatch-mode "$MODE" $LIM
|
||||
cp "$OUT/metrics_$MODE.summary.json" "$OUT/summary_$MODE.json" 2>/dev/null || true
|
||||
done
|
||||
teardown
|
||||
echo "=== exp (c) DONE ==="
|
||||
BIN
v2/figs/exp_c_dispatch_ablation.png
Normal file
BIN
v2/figs/exp_c_dispatch_ablation.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 68 KiB |
Reference in New Issue
Block a user