v7 with --decode-mem-fraction-static=0.8 + SGLANG_SNAPSHOT_LINK_BUF_BYTES=16GB
silently fell back to 1 GB snapshot_buf because Prefill (mem-fraction
default 0.88) left only 10.8 GB free on GPU 0. Reducing prefill
mem-fraction lets 16 GB snapshot_buf fit.
Background nvidia-smi poller runs at 1 Hz for all 4 GPUs throughout
the sweep, writing CSV to $OUTPUT/gpu_util.csv. Captures:
timestamp_iso, gpu_index, util_pct, mem_used_MiB, mem_total_MiB,
sm_clock_MHz, power_W, temperature_C
Sampler is started before benchmark-live and torn down via trap on
EXIT/INT/TERM so it always cleans up even if the run is killed.
This data lets us plot time-windowed wall-clock GPU utilization
(per-card) so we can answer "is concurrency the bottleneck or is
each D's per-session decode the bottleneck" — a question that
came up during E4-v3 / v5 analysis.
The third_party SWE-Bench trace uses real wall-clock timestamps
(5.44h span, p50 inter-turn 2.53s). With --time-scale 1 the sweep
mirrors the original timeline, taking 5.44h. TIME_SCALE env var
lets us compress (e.g. 10 → 33min, 60 → 5.5min) for tighter
iteration; defaults to 1 for realistic comparison.
Usage:
TIME_SCALE=10 bash scripts/sweep_e4_pressured.sh
TIME_SCALE=60 bash scripts/sweep_e4_pressured.sh
Switches the default --trace from outputs/inferact_50sess.jsonl
(median 63K, p99 143K, 1285 reqs) to
third_party/traces/qwen35-swebench-50sess.jsonl (median 27K,
p99 92K, 4449 reqs across 52 sessions). Smaller per-request
inputs let us check whether the queue-induced TTFT collapse
the user flagged is workload-specific. Total trace is 3.5x
larger so the run will cover more turns per session.
E4-v1 / v2 / pressured-v1 all failed to fire admission rejections in
this workload because the default 0.6 mem-fraction-static gives
288K-token kv_pool per decoder, more than enough to absorb the
50-session trace even at concurrency=32.
This commit adds:
--decode-mem-fraction-static (overrides per-decode SGLang arg)
--prefill-mem-fraction-static (symmetric for completeness)
Plumbed via topology.{decode,prefill}_extra_server_args. The
pressured sweep now uses --decode-mem-fraction-static 0.4 which
shrinks decoder kv_pool to ~192K tokens — should force enough
admission rejections to actually exercise the D→P snapshot path.
E4-v1 produced 272 admission rejects (good) but zero /_snapshot HTTP
calls (bad, entrance gate bug fixed in e729d62). E4-v2 went the other
way: 0 rejects through 53% of trace, sync function never even called.
E4-pressured locks in the *fix-verified* code path by lowering
--kvcache-migration-reject-threshold from 3 to 1. After ONE
rejection the policy forces session migration, which lands in
_invoke_kvcache_seeded_router → _attempt_d_to_p_sync.
With the e729d62 fix in place, the d-to-p-sync.jsonl structural log
should now capture every prepare/dump/finalize decision so we can
forensic verify the D→P fast path is actually delivering KV bytes
to P's radix tree.
