gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix replay methodology: trace-driven dispatch, no artificial limits

Browse Source 
The replayer was artificially limiting concurrency with --max-inflight-sessions
(semaphore) and --time-scale (time compression), producing unrealistically low
1 req/GPU load that masked prefill-decode interference.

Replayer changes:
- Remove session_sem and time_scale entirely
- Each request dispatched at its trace timestamp exactly
- Sessions still sequential (turn N+1 waits for turn N completion)
- If turn completes late, next turn fires immediately

Sampler changes:
- Add --sample-ratio for GPU-proportional session sampling
- Keep --target-requests for backwards compat
- No time compression (preserve original arrival pattern)

bench.sh: remove --time-scale and --max-inflight-sessions args

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-23 12:43:41 +08:00
parent c8ba666517
commit 4089ffd63f
4 changed files with 84 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
scripts/bench.sh
View File

@@ -30,9 +30,7 @@ POLICY="linear" # linear | lmetric
N_INSTANCES=8
BASE_PORT=8000
PROXY_PORT=9090
REQUESTS=200
TIME_SCALE=20
MAX_SESSIONS=8
REQUESTS="" # empty = all requests in trace
HEAVY_THRESHOLD=20000
NO_OFFLOAD=false
OVERLOAD_FACTOR_ARG=""
@@ -46,8 +44,6 @@ while [[ $# -gt 0 ]]; do
--policy) POLICY="$2"; shift 2 ;;
--instances) N_INSTANCES="$2"; shift 2 ;;
--requests) REQUESTS="$2"; shift 2 ;;
--time-scale) TIME_SCALE="$2"; shift 2 ;;
--sessions) MAX_SESSIONS="$2"; shift 2 ;;
--heavy-threshold) HEAVY_THRESHOLD="$2"; shift 2 ;;
--no-offload) NO_OFFLOAD=true; shift ;;
--overload-factor) OVERLOAD_FACTOR_ARG="$2"; shift 2 ;;
@@ -58,6 +54,7 @@ done
if [ -z "$TAG" ]; then
echo "Usage: bench.sh --tag NAME --mode {baseline|elastic} [--instances N] [--policy {linear|lmetric}] [--requests N]"
echo " Trace QPS is controlled by sample_trace.py --sample-ratio, not by bench.sh."
exit 1
fi
@@ -76,9 +73,7 @@ cat > "$OUTDIR/config.json" << CONF
"policy": "$POLICY",
"model": "$MODEL",
"n_instances": $N_INSTANCES,
"requests": $REQUESTS,
"time_scale": $TIME_SCALE,
"max_sessions": $MAX_SESSIONS,
"requests": "${REQUESTS:-all}",
"heavy_threshold": $HEAVY_THRESHOLD,
"no_offload": "$NO_OFFLOAD",
"overload_factor": "${OVERLOAD_FACTOR_ARG:-2.0}",
@@ -245,7 +240,13 @@ launch_proxy() {
# ─── Run benchmark ─────────────────────────────────────────────────────────
run_benchmark() {
echo "[bench] Running $REQUESTS requests (time_scale=$TIME_SCALE, sessions=$MAX_SESSIONS)..."
local request_args=""
if [ -n "$REQUESTS" ]; then
request_args="--request-limit $REQUESTS"
echo "[bench] Running $REQUESTS requests (trace-driven timing)..."
else
echo "[bench] Running all requests in trace (trace-driven timing)..."
fi
# Start GPU monitor in background
bash "$PROJECT_DIR/scripts/gpu_monitor.sh" "$OUTDIR/gpu_util.csv" 5 &
@@ -256,9 +257,7 @@ run_benchmark() {
--output "$OUTDIR/metrics.jsonl" \
--endpoint "http://localhost:$PROXY_PORT" \
--model "$MODEL" \
--time-scale "$TIME_SCALE" \
--max-inflight-sessions "$MAX_SESSIONS" \
--request-limit "$REQUESTS" \
$request_args \
-v 2>&1 | tee "$OUTDIR/replayer.log"
# Stop GPU monitor
@@ -324,7 +323,7 @@ print('=' * 70)
echo "================================================================"
echo " bench.sh: $TAG"
echo " mode=$MODE policy=$POLICY requests=$REQUESTS overload_factor=${OVERLOAD_FACTOR_ARG:-2.0} max_batched_tokens=${MAX_BATCHED_TOKENS:-default}"
echo " mode=$MODE policy=$POLICY requests=${REQUESTS:-all} overload_factor=${OVERLOAD_FACTOR_ARG:-2.0}"
echo " $(date)"
echo "================================================================"

92
scripts/sample_trace.py
View File

@@ -2,22 +2,28 @@
Preserves:
- Complete session structure (all turns within a session kept together)
- Original arrival timing (inter-session and intra-session gaps)
- Original arrival timing (re-zeroed to t=0 but NOT compressed)
- hash_ids for KV cache reuse patterns
- Request type distribution
Sampling strategy:
1. Group requests by session (derived from parent_chat_id chains)
2. Randomly sample N sessions (or until target request count reached)
2. Randomly sample a fraction of sessions (--sample-ratio)
OR sample until target request count (--target-requests)
3. Re-zero timestamps so first event starts at t=0
4. Optionally compress time axis to increase load density
4. The resulting QPS is proportional to the sample ratio,
preserving the production arrival pattern
Usage:
# Sample 1.6% of sessions (e.g., 8 GPUs / 500 cluster GPUs)
python scripts/sample_trace.py \\
--input ~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl \\
--output traces/sampled.jsonl \\
--target-requests 5000 \\
--seed 42
--output traces/sampled_ratio016.jsonl \\
--sample-ratio 0.016 --seed 42
# Sample by request count (legacy)
python scripts/sample_trace.py \\
--input ... --output ... --target-requests 1000 --seed 42
"""
from __future__ import annotations
@@ -58,39 +64,37 @@ def load_raw_rows(path: Path) -> dict[str, list[dict]]:
def sample_sessions(
rows_by_session: dict[str, list[dict]],
*,
target_requests: int,
sample_ratio: float | None = None,
target_requests: int | None = None,
seed: int,
strategy: str = "random",
) -> list[str]:
"""Select sessions until target request count is reached."""
"""Select sessions by ratio or until target request count."""
all_sids = list(rows_by_session.keys())
rng = random.Random(seed)
rng.shuffle(all_sids)
if strategy == "random":
rng.shuffle(all_sids)
elif strategy == "sequential":
pass # keep file order
else:
raise ValueError(f"Unknown strategy: {strategy}")
if sample_ratio is not None:
n_select = max(1, int(len(all_sids) * sample_ratio))
return all_sids[:n_select]
selected = []
total = 0
for sid in all_sids:
selected.append(sid)
total += len(rows_by_session[sid])
if total >= target_requests:
break
if target_requests is not None:
selected = []
total = 0
for sid in all_sids:
selected.append(sid)
total += len(rows_by_session[sid])
if total >= target_requests:
break
return selected
return selected
raise ValueError("Must specify --sample-ratio or --target-requests")
def build_output(
rows_by_session: dict[str, list[dict]],
selected: list[str],
*,
time_scale: float = 1.0,
) -> list[dict]:
"""Build output rows with re-zeroed timestamps."""
"""Build output rows with re-zeroed timestamps (no time compression)."""
out_rows = []
for sid in selected:
for row in rows_by_session[sid]:
@@ -103,10 +107,9 @@ def build_output(
if not out_rows:
return out_rows
# Re-zero: subtract earliest timestamp
t0 = float(out_rows[0]["timestamp"])
for row in out_rows:
row["timestamp"] = (float(row["timestamp"]) - t0) / time_scale
row["timestamp"] = float(row["timestamp"]) - t0
return out_rows
@@ -125,17 +128,16 @@ def print_summary(
output_lens = [r["output_length"] for r in out_rows]
span_s = float(out_rows[-1]["timestamp"]) if out_rows else 0
qps = n_requests / span_s if span_s > 0 else 0
session_starts = {}
for r in out_rows:
sid = r["session_id"]
ts = float(r["timestamp"])
if sid not in session_starts:
session_starts[sid] = ts
starts_sorted = sorted(session_starts.values())
deltas = [starts_sorted[i+1] - starts_sorted[i]
for i in range(len(starts_sorted) - 1)]
# hash_ids overlap: count unique hash_ids across all requests
# hash_ids overlap
all_hashes = set()
for r in out_rows:
all_hashes.update(r.get("hash_ids", []))
@@ -149,12 +151,8 @@ def print_summary(
print(f" Output length: min={min(output_lens)} max={max(output_lens)} "
f"avg={sum(output_lens)/len(output_lens):.0f}")
print(f" Trace span: {span_s:.1f}s ({span_s/60:.1f} min)")
print(f" QPS: {qps:.2f} req/s")
print(f" Unique hash blocks: {len(all_hashes)}")
if deltas:
deltas.sort()
p = lambda q: deltas[min(int(q * len(deltas)), len(deltas) - 1)]
print(f" Session arrival deltas (s): p10={p(0.1):.2f} p50={p(0.5):.2f} "
f"p90={p(0.9):.2f} max={max(deltas):.2f}")
def main() -> None:
@@ -164,15 +162,16 @@ def main() -> None:
help="Path to the full trace JSONL file")
p.add_argument("--output", type=Path, required=True,
help="Path to write sampled trace JSONL")
p.add_argument("--target-requests", type=int, default=5000,
help="Target number of requests (stops after session that crosses it)")
p.add_argument("--strategy", choices=["random", "sequential"], default="random",
help="Session selection strategy")
p.add_argument("--time-scale", type=float, default=1.0,
help="Compress time axis by this factor (>1 = faster arrival)")
p.add_argument("--sample-ratio", type=float, default=None,
help="Fraction of sessions to sample (e.g. 0.016 for 8/500 GPU ratio)")
p.add_argument("--target-requests", type=int, default=None,
help="Target number of requests (legacy, stops after session that crosses it)")
p.add_argument("--seed", type=int, default=42)
args = p.parse_args()
if args.sample_ratio is None and args.target_requests is None:
p.error("Must specify --sample-ratio or --target-requests")
print(f"Loading trace from {args.input} ...")
rows_by_session = load_raw_rows(args.input)
total_sessions = len(rows_by_session)
@@ -181,15 +180,12 @@ def main() -> None:
selected = sample_sessions(
rows_by_session,
sample_ratio=args.sample_ratio,
target_requests=args.target_requests,
seed=args.seed,
strategy=args.strategy,
)
out_rows = build_output(
rows_by_session, selected,
time_scale=args.time_scale,
)
out_rows = build_output(rows_by_session, selected)
print_summary(rows_by_session, selected, out_rows)