Replayer think-time dispatch mode + benchmarking guidance

Adds `--dispatch-mode {tracets,thinktime}` to the replayer and documents that
agentic serving should be benchmarked with `thinktime` (the faithful load).

- `tracets` (old default): turn-k at the absolute trace timestamp, i.e.
  max(prev_finished, trace_ts) -- collapses inter-turn think-time to ~0 when the
  system is behind, manufacturing request bursts.
- `thinktime`: turn-1 at trace arrival; turn-k at prev_finished +
  time_to_parent_chat (real production gap). scripts/add_time_to_parent.py
  annotates a trace with that gap from the raw trace's request_ready/end_ms.

exp(c) ablation (v2/exp_c_dispatch_ablation/): at N=8 (capacity slack) thinktime
beats tracets -- E2E p90 -28% (73.5 vs 102.8s), TTFT p90 -29%, TPS +7%, because
tracets' bursts spike concurrency -> KV pressure -> preemption. At N=6
(saturated) they converge. So tracets makes the system look ~30% worse on tail
latency than realistic agent pacing. Root README.md carries the headline
guidance; raw per-request metrics gitignored (perf_summary.json kept).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-05-30 16:25:33 +08:00
parent f0d085ceda
commit 8a6b22c11c
11 changed files with 547 additions and 3 deletions

View File

@@ -30,6 +30,11 @@ def main() -> None:
default=float(_env_think) if _env_think else None,
help="Closed-loop think-time (s) after each turn completes; "
"ignore absolute trace schedule. Env: REPLAY_INTER_TURN_THINK_S")
p.add_argument("--dispatch-mode", choices=["tracets", "thinktime"],
default=os.environ.get("REPLAY_DISPATCH_MODE", "tracets"),
help="tracets (Mode 1): absolute trace ts = max(prev_finished, ts). "
"thinktime (Mode 2): turn-k at prev_finished + "
"time_to_parent_chat. Env: REPLAY_DISPATCH_MODE")
p.add_argument("--request-timeout", type=float, default=600.0)
p.add_argument("--request-limit", type=int, default=None,
help="Limit number of requests to replay")
@@ -51,6 +56,7 @@ def main() -> None:
request_limit=args.request_limit,
max_inflight_sessions=args.max_inflight_sessions,
inter_turn_think_s=args.inter_turn_think,
dispatch_mode=args.dispatch_mode,
)
results = asyncio.run(replay_trace(config))