Gahow Wang
05592e6adc
Systematic study of prefill-decode disaggregation for agentic LLM workloads using production GLM-5.1 coder trace (2.1M requests, 71B input tokens). Key findings: - Cache-aware routing improves TPOT p90 by 15% and APC from 20.8% to 44.7% without PD separation, matching PD-Sep's decode isolation benefit - PD separation adds +72% TTFT overhead (KV transfer) with no TPOT gain when using the same cache-aware scheduler - Prefill remains compute-bound even at 95% KV cache reuse (AI >1000x vs decode AI <2), but absolute FLOPs drop 71% from cache hits - For agentic MoE workloads, cache-aware routing > PD separation Infrastructure: - Trace sampler preserving session structure + hash_ids for prefix sharing - Async trace replayer with streaming TTFT/TPOT/E2E measurement - Unified cache-aware + token-level load-balanced global scheduler proxy supporting both PD-colocated and PD-disaggregated (Mooncake/RDMA) modes - vLLM 0.18.1 scheduler patch for KV transfer abort race condition - Roofline analysis tool for prefill/decode compute characterization Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
24 lines
632 B
Bash
Executable File
24 lines
632 B
Bash
Executable File
#!/bin/bash
|
|
# Launch vLLM 0.18.1 in PD-combined mode (TP=8, all GPUs).
|
|
#
|
|
# Usage: bash scripts/launch_vllm.sh
|
|
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
|
|
VLLM="$PROJECT_DIR/.venv/bin/vllm"
|
|
|
|
MODEL_PATH="${MODEL_PATH:-$HOME/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
|
|
HOST="${HOST:-0.0.0.0}"
|
|
PORT="${PORT:-8000}"
|
|
|
|
echo "Starting vLLM 0.18.1 in PD-combined mode (TP=8) on port $PORT ..."
|
|
$VLLM serve "$MODEL_PATH" \
|
|
--trust-remote-code \
|
|
--enable-prefix-caching \
|
|
--dtype auto \
|
|
--tensor-parallel-size 8 \
|
|
--host "$HOST" \
|
|
--port "$PORT"
|