Replayer think-time dispatch mode + benchmarking guidance
Adds `--dispatch-mode {tracets,thinktime}` to the replayer and documents that
agentic serving should be benchmarked with `thinktime` (the faithful load).
- `tracets` (old default): turn-k at the absolute trace timestamp, i.e.
max(prev_finished, trace_ts) -- collapses inter-turn think-time to ~0 when the
system is behind, manufacturing request bursts.
- `thinktime`: turn-1 at trace arrival; turn-k at prev_finished +
time_to_parent_chat (real production gap). scripts/add_time_to_parent.py
annotates a trace with that gap from the raw trace's request_ready/end_ms.
exp(c) ablation (v2/exp_c_dispatch_ablation/): at N=8 (capacity slack) thinktime
beats tracets -- E2E p90 -28% (73.5 vs 102.8s), TTFT p90 -29%, TPS +7%, because
tracets' bursts spike concurrency -> KV pressure -> preemption. At N=6
(saturated) they converge. So tracets makes the system look ~30% worse on tail
latency than realistic agent pacing. Root README.md carries the headline
guidance; raw per-request metrics gitignored (perf_summary.json kept).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -28,6 +28,9 @@ class TraceRequest:
|
||||
request_type: str
|
||||
turn_id: int
|
||||
hash_ids: tuple[int, ...]
|
||||
# real production gap (s) from parent turn finishing to this turn arriving;
|
||||
# None for turn-1 / unannotated traces. Used by --dispatch-mode thinktime.
|
||||
time_to_parent_chat_s: float | None = None
|
||||
|
||||
|
||||
def load_trace(
|
||||
@@ -66,6 +69,9 @@ def load_trace(
|
||||
request_type=str(row["type"]),
|
||||
turn_id=int(row["turn"]),
|
||||
hash_ids=tuple(int(h) for h in row.get("hash_ids", [])),
|
||||
time_to_parent_chat_s=(
|
||||
float(row["time_to_parent_chat"])
|
||||
if row.get("time_to_parent_chat") is not None else None),
|
||||
))
|
||||
|
||||
return requests
|
||||
|
||||
Reference in New Issue
Block a user