Gahow Wang
7437422618
Two new files prepare measurement of T_transfer(KV_size, network_path),
the gap §3.2's PD-disagg cost argument has had since day one.
microbench/fresh_setup/start_vllm_pair.sh
start | status | stop two vLLM 0.18.1 instances on local GPUs (A, B)
with --kv-transfer-config '{"kv_connector":"MooncakeConnector",
"kv_role":"kv_both"}' running off the fresh venv (vanilla wheel +
vanilla mooncake 0.3.11, NOT the dash0 patched build). GPU IDs and
ports are env-overridable so the same script drives the intra-node
pair (GPU_A=0 GPU_B=1 on one host) and the inter-node pair (GPU_A=0
on dash1, GPU_B=0 on dash2 — launched per host separately).
microbench/fresh_setup/mb2_kv_transfer.py
Three-step measurement borrowed from connector_tax/.../smoke_test_
migrate_cache.py:
1. do_remote_decode on A (compute & cache KV; max_tokens=1)
2. do_remote_prefill on B (pull KV from A — this is the timed step)
3. plain completion on B (sanity check: cached_tokens ≈ prompt len)
Sweeps input_tokens ∈ {512, 1k, 2k, 4k, 8k, 16k, 32k, 64k} with 5
repeats each; reports mean / p50 / p90 transfer time and a per-size
raw log. Per-token KV is 98304 B (Qwen3-Coder-30B-A3B), so the upper
end ≈ 6 GiB transfers — within the p99 11.5 GiB range from §2 but
below it (the model's max_model_len 200000 caps the absolute upper).
What we will NOT learn from this design:
- Bandwidth saturation when the system is loaded (single-request bench)
- vLLM-internal scheduling overhead vs pure transfer (the timed step
folds them together — but for the §3.2 argument that's the right
"what does PD-disagg actually pay" number)
Intentionally not committed yet: an orchestrator that loops over
intra-/inter-node configs. We start manual on dash1 intra-node to
verify the measurement is sane before scaling out.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
106 lines
3.0 KiB
Bash
Executable File
106 lines
3.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Start 2 vLLM instances with Mooncake kv_connector (kv_both) for MB2.
|
|
#
|
|
# Default config: both on local GPU 0 and 1 (intra-node A/B test).
|
|
# Override via GPU_A / GPU_B / HOST_A / HOST_B env vars.
|
|
#
|
|
# This uses the FRESH venv at /home/admin/cpfs/wjh/agentic-kv-fresh/.venv
|
|
# (vanilla vllm 0.18.1 + vanilla mooncake-transfer-engine 0.3.11), NOT
|
|
# the dash0 patched build.
|
|
#
|
|
# Usage:
|
|
# GPU_A=0 GPU_B=1 bash microbench/fresh_setup/start_vllm_pair.sh
|
|
# bash microbench/fresh_setup/start_vllm_pair.sh status
|
|
# bash microbench/fresh_setup/start_vllm_pair.sh stop
|
|
|
|
set -eo pipefail
|
|
|
|
FRESH_ROOT="/home/admin/cpfs/wjh/agentic-kv-fresh"
|
|
VENV="${FRESH_ROOT}/.venv"
|
|
MODEL="${MODEL:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
|
|
LOGS_DIR="${LOGS_DIR:-${FRESH_ROOT}/mb2_logs}"
|
|
mkdir -p "${LOGS_DIR}"
|
|
|
|
GPU_A="${GPU_A:-0}"
|
|
GPU_B="${GPU_B:-1}"
|
|
PORT_A=8000
|
|
PORT_B=8001
|
|
BP_A=8998
|
|
BP_B=8999
|
|
MASTER_A=29500
|
|
MASTER_B=29501
|
|
|
|
stop_all() {
|
|
pkill -9 -f "vllm serve" 2>/dev/null || true
|
|
pkill -9 -f "EngineCore" 2>/dev/null || true
|
|
sleep 2
|
|
}
|
|
|
|
case "${1:-start}" in
|
|
stop)
|
|
stop_all
|
|
exit 0
|
|
;;
|
|
status)
|
|
for p in "${PORT_A}" "${PORT_B}"; do
|
|
if curl -sf "http://127.0.0.1:${p}/health" >/dev/null 2>&1; then
|
|
echo "port ${p}: UP"
|
|
else
|
|
echo "port ${p}: DOWN"
|
|
fi
|
|
done
|
|
exit 0
|
|
;;
|
|
start)
|
|
;;
|
|
*)
|
|
echo "Unknown command: $1"; exit 1;;
|
|
esac
|
|
|
|
stop_all
|
|
|
|
source "${VENV}/bin/activate"
|
|
|
|
launch() {
|
|
local idx="$1" gpu="$2" port="$3" bp="$4" master="$5"
|
|
echo "[mb2] launching instance ${idx} on GPU ${gpu}, port ${port}, bp ${bp}"
|
|
PYTHONHASHSEED=42 \
|
|
VLLM_MOONCAKE_BOOTSTRAP_PORT="${bp}" \
|
|
CUDA_VISIBLE_DEVICES="${gpu}" \
|
|
MASTER_PORT="${master}" \
|
|
nohup vllm serve "${MODEL}" \
|
|
--host 0.0.0.0 --port "${port}" \
|
|
--tensor-parallel-size 1 \
|
|
--trust-remote-code --enable-prefix-caching \
|
|
--dtype auto --gpu-memory-utilization 0.9 \
|
|
--max-model-len 200000 \
|
|
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \
|
|
--enable-prompt-tokens-details \
|
|
> "${LOGS_DIR}/vllm_${idx}_gpu${gpu}.log" 2>&1 &
|
|
disown
|
|
}
|
|
|
|
launch A "${GPU_A}" "${PORT_A}" "${BP_A}" "${MASTER_A}"
|
|
sleep 3
|
|
launch B "${GPU_B}" "${PORT_B}" "${BP_B}" "${MASTER_B}"
|
|
|
|
echo "[mb2] waiting for both /health endpoints..."
|
|
for port in "${PORT_A}" "${PORT_B}"; do
|
|
tries=0
|
|
while ! curl -sf "http://127.0.0.1:${port}/health" >/dev/null 2>&1; do
|
|
tries=$((tries+1))
|
|
if [ ${tries} -gt 180 ]; then
|
|
echo "[mb2] FATAL port ${port} did not come up in 6 min"
|
|
tail -40 "${LOGS_DIR}/vllm_"*"_gpu"*".log" || true
|
|
exit 1
|
|
fi
|
|
sleep 2
|
|
done
|
|
echo " port=${port} ready"
|
|
done
|
|
|
|
echo "[mb2] both instances UP"
|
|
echo " A: 127.0.0.1:${PORT_A} (GPU ${GPU_A}, bp ${BP_A})"
|
|
echo " B: 127.0.0.1:${PORT_B} (GPU ${GPU_B}, bp ${BP_B})"
|
|
echo " logs: ${LOGS_DIR}"
|