Production-realistic baseline: APC 67.5%, TPOT +139% from interference
Updated methodology: - Window+thin sampling preserves cross-session sharing (48% vs 16%) - --max-single-turn-ratio 0.3 boosts multi-turn to 70% - --window-seconds 600 for 10-min contiguous window - Trace-driven replay (no session limit, no time compression) - Daily config: --requests 850 (~13 min, APC~76%) Key result: TPOT p90=0.175s (vs 0.073s in legacy 1-req/GPU setup), confirming prefill-decode interference is real at production concurrency. APC 67.5% (vs 44%) from better KV reuse preservation. Also fixed KV reuse breakdown: 62% intra-session / 38% cross-session (was incorrectly reported as 91% / 9%). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -21,7 +21,7 @@ VENV="${VENV_PATH:-$PROJECT_DIR/.venv/bin}"
|
||||
PYTHON="$VENV/python"
|
||||
VLLM="$VENV/vllm"
|
||||
MODEL="${MODEL_PATH:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
|
||||
TRACE="$PROJECT_DIR/traces/sampled_1000req_seed42.jsonl"
|
||||
TRACE="${TRACE:-$PROJECT_DIR/traces/sampled_1000req_seed42.jsonl}"
|
||||
|
||||
# Defaults
|
||||
TAG=""
|
||||
@@ -44,6 +44,7 @@ while [[ $# -gt 0 ]]; do
|
||||
--policy) POLICY="$2"; shift 2 ;;
|
||||
--instances) N_INSTANCES="$2"; shift 2 ;;
|
||||
--requests) REQUESTS="$2"; shift 2 ;;
|
||||
--trace) TRACE="$2"; shift 2 ;;
|
||||
--heavy-threshold) HEAVY_THRESHOLD="$2"; shift 2 ;;
|
||||
--no-offload) NO_OFFLOAD=true; shift ;;
|
||||
--overload-factor) OVERLOAD_FACTOR_ARG="$2"; shift 2 ;;
|
||||
|
||||
Reference in New Issue
Block a user