gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replayer think-time dispatch mode + benchmarking guidance

Browse Source 
Adds `--dispatch-mode {tracets,thinktime}` to the replayer and documents that
agentic serving should be benchmarked with `thinktime` (the faithful load).

- `tracets` (old default): turn-k at the absolute trace timestamp, i.e.
  max(prev_finished, trace_ts) -- collapses inter-turn think-time to ~0 when the
  system is behind, manufacturing request bursts.
- `thinktime`: turn-1 at trace arrival; turn-k at prev_finished +
  time_to_parent_chat (real production gap). scripts/add_time_to_parent.py
  annotates a trace with that gap from the raw trace's request_ready/end_ms.

exp(c) ablation (v2/exp_c_dispatch_ablation/): at N=8 (capacity slack) thinktime
beats tracets -- E2E p90 -28% (73.5 vs 102.8s), TTFT p90 -29%, TPS +7%, because
tracets' bursts spike concurrency -> KV pressure -> preemption. At N=6
(saturated) they converge. So tracets makes the system look ~30% worse on tail
latency than realistic agent pacing. Root README.md carries the headline
guidance; raw per-request metrics gitignored (perf_summary.json kept).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-30 16:25:33 +08:00
parent f0d085ceda
commit 8a6b22c11c
11 changed files with 547 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
replayer/replay.py
View File

@@ -66,6 +66,13 @@ class ReplayConfig:
# max_inflight_sessions=N this is a stable N-user closed-loop (no open-loop
# runaway), so it removes the "immediate retrigger under load" artifact.
inter_turn_think_s: float | None = None
# Dispatch timing for intra-session turns:
# "tracets" (Mode 1): fire at absolute trace timestamp -> effectively
# max(prev_finished, trace_ts); collapses think-time to 0 when
# the system is behind (the amplification-inflation suspect).
# "thinktime" (Mode 2): turn-1 at trace arrival; turn-k at
# prev_finished + time_to_parent_chat (real production gap).
dispatch_mode: str = "tracets"
def _build_prompt_token_ids(req: TraceRequest) -> list[int]:
@@ -286,14 +293,26 @@ async def _run_session(
realized_context: list[int] = []
try:
for turn_idx, req in enumerate(state.turns):
if config.inter_turn_think_s is not None:
if config.dispatch_mode == "thinktime":
# Mode 2: turn-1 at absolute trace arrival (preserve session
# schedule); later turns wait the REAL per-record think-time after
# the previous turn completed -> no think-collapse under load.
if turn_idx == 0:
target_wall = (req.timestamp_s - earliest_ts)
elapsed = time.perf_counter() - sweep_start
if elapsed < target_wall:
await asyncio.sleep(target_wall - elapsed)
else:
think = req.time_to_parent_chat_s
await asyncio.sleep(think if think is not None else 0.0)
elif config.inter_turn_think_s is not None:
# Closed-loop: turn 1 fires on admission; later turns wait a fixed
# think-time AFTER the previous turn completed (no absolute schedule,
# so no "fire immediately because timestamp is in the past").
if turn_idx > 0:
await asyncio.sleep(config.inter_turn_think_s)
else:
# Original: dispatch at the request's absolute trace timestamp.
# Mode 1: dispatch at the request's absolute trace timestamp.
target_wall = (req.timestamp_s - earliest_ts)
elapsed = time.perf_counter() - sweep_start
if elapsed < target_wall: