Gahow Wang
72790ae6c1
Instrumentation patches (microbench/patches/):
- pd_profile.py: shared event emitter (VLLM_PD_PROFILE_LOG env var)
- apply_patches.py: idempotent patch installer for mooncake_connector.py
and scheduler.py, marks insertions with # PD_PROFILE_PATCH
- analyze_events.py: joins per-process JSONL event logs by transfer_id
into per-request phase durations
Seven events captured per request:
D_get_num_matched → P_zmq_received → P_prefill_done →
P_rdma_start → P_rdma_end → D_recv_complete → D_request_promoted
Driver fix (microbench/lifecycle/driver.py):
seed_prefix_cache now sends via the proxy URL so P and D both cache
the seeded prefix with matching block hashes. Previously seeding D
directly produced different block hashes than the proxy-routed
measurement requests, making incremental transfer impossible.
Real breakdown (fig_breakdown_real.png, server_breakdown.csv, n=93):
prefill_compute 620 ms median (95% of overhead)
rdma_transfer 42 ms median (~71 Gbps effective)
other overhead 10 ms median (dispatch + params + signal + promote)
Mooncake transfer is NOT the bottleneck. Even with bulk RDMA the
transfer cost is <10% of prefill cost for Qwen3-30B-A3B on H20.
2 lines
5.1 KiB
JSON
2 lines
5.1 KiB
JSON
[{"D": 1, "P": 16384, "rep": 0, "prefill_ttft_ms": 1551.6944919945672, "tpot_baseline_p50": 6.0938840033486485}, {"D": 1, "P": 16384, "rep": 1, "prefill_ttft_ms": 1538.5930379852653, "tpot_baseline_p50": 5.923410004470497}, {"D": 1, "P": 16384, "rep": 2, "prefill_ttft_ms": 1537.489719048608, "tpot_baseline_p50": 5.858020973391831}, {"D": 1, "P": 2048, "rep": 0, "prefill_ttft_ms": 138.6167829623446, "tpot_baseline_p50": 5.895276990486309}, {"D": 1, "P": 2048, "rep": 1, "prefill_ttft_ms": 135.26123203337193, "tpot_baseline_p50": 5.84050000179559}, {"D": 1, "P": 2048, "rep": 2, "prefill_ttft_ms": 139.52087302459404, "tpot_baseline_p50": 5.846929037943482}, {"D": 1, "P": 32768, "rep": 0, "prefill_ttft_ms": 4565.813232969958, "tpot_baseline_p50": 5.815482989419252}, {"D": 1, "P": 32768, "rep": 1, "prefill_ttft_ms": 4564.696346002165, "tpot_baseline_p50": 5.884206970222294}, {"D": 1, "P": 32768, "rep": 2, "prefill_ttft_ms": 4559.26972796442, "tpot_baseline_p50": 5.844508996233344}, {"D": 1, "P": 8192, "rep": 0, "prefill_ttft_ms": 592.7710899850354, "tpot_baseline_p50": 5.8953959960490465}, {"D": 1, "P": 8192, "rep": 1, "prefill_ttft_ms": 587.9120359895751, "tpot_baseline_p50": 5.859997007064521}, {"D": 1, "P": 8192, "rep": 2, "prefill_ttft_ms": 585.4344139806926, "tpot_baseline_p50": 5.852615984622389}, {"D": 2, "P": 16384, "rep": 0, "prefill_ttft_ms": 1542.8870850009844, "tpot_baseline_p50": 6.665366003289819}, {"D": 2, "P": 16384, "rep": 1, "prefill_ttft_ms": 1544.4676239858381, "tpot_baseline_p50": 6.215651519596577}, {"D": 2, "P": 16384, "rep": 2, "prefill_ttft_ms": 1547.6729370420799, "tpot_baseline_p50": 6.198290007887408}, {"D": 2, "P": 2048, "rep": 0, "prefill_ttft_ms": 140.2357509941794, "tpot_baseline_p50": 6.308203533990309}, {"D": 2, "P": 2048, "rep": 1, "prefill_ttft_ms": 134.00237500900403, "tpot_baseline_p50": 6.295669532846659}, {"D": 2, "P": 2048, "rep": 2, "prefill_ttft_ms": 133.65825096843764, "tpot_baseline_p50": 6.221923045814037}, {"D": 2, "P": 32768, "rep": 0, "prefill_ttft_ms": 4562.765512033366, "tpot_baseline_p50": 6.217204965651035}, {"D": 2, "P": 32768, "rep": 1, "prefill_ttft_ms": 4588.010042964015, "tpot_baseline_p50": 6.249245023354888}, {"D": 2, "P": 32768, "rep": 2, "prefill_ttft_ms": 4575.238945020828, "tpot_baseline_p50": 6.3047730072867125}, {"D": 2, "P": 8192, "rep": 0, "prefill_ttft_ms": 591.6664049727842, "tpot_baseline_p50": 6.216273992322385}, {"D": 2, "P": 8192, "rep": 1, "prefill_ttft_ms": 589.0314219868742, "tpot_baseline_p50": 6.219989008968696}, {"D": 2, "P": 8192, "rep": 2, "prefill_ttft_ms": 589.9350599502213, "tpot_baseline_p50": 6.202562013641}, {"D": 4, "P": 16384, "rep": 0, "prefill_ttft_ms": 1548.318826011382, "tpot_baseline_p50": 6.581096502486616}, {"D": 4, "P": 16384, "rep": 1, "prefill_ttft_ms": 1556.7272949847393, "tpot_baseline_p50": 6.632126489421353}, {"D": 4, "P": 16384, "rep": 2, "prefill_ttft_ms": 1548.7530149985105, "tpot_baseline_p50": 7.3001839919015765}, {"D": 4, "P": 2048, "rep": 0, "prefill_ttft_ms": 141.15837798453867, "tpot_baseline_p50": 6.574946513865143}, {"D": 4, "P": 2048, "rep": 1, "prefill_ttft_ms": 144.0463720355183, "tpot_baseline_p50": 6.539304507896304}, {"D": 4, "P": 2048, "rep": 2, "prefill_ttft_ms": 139.0016739605926, "tpot_baseline_p50": 6.603387999348342}, {"D": 4, "P": 32768, "rep": 0, "prefill_ttft_ms": 4582.872495986521, "tpot_baseline_p50": 6.574522005394101}, {"D": 4, "P": 32768, "rep": 1, "prefill_ttft_ms": 4584.32119601639, "tpot_baseline_p50": 6.611509481444955}, {"D": 4, "P": 32768, "rep": 2, "prefill_ttft_ms": 4593.307823990472, "tpot_baseline_p50": 7.247850007843226}, {"D": 4, "P": 8192, "rep": 0, "prefill_ttft_ms": 588.1257400033064, "tpot_baseline_p50": 6.58076599938795}, {"D": 4, "P": 8192, "rep": 1, "prefill_ttft_ms": 593.7537999707274, "tpot_baseline_p50": 7.294313982129097}, {"D": 4, "P": 8192, "rep": 2, "prefill_ttft_ms": 589.4501690054312, "tpot_baseline_p50": 7.346310012508184}, {"D": 8, "P": 16384, "rep": 0, "prefill_ttft_ms": 1564.8929480230436, "tpot_baseline_p50": 8.545519027393311}, {"D": 8, "P": 16384, "rep": 1, "prefill_ttft_ms": 1548.527927021496, "tpot_baseline_p50": 8.990490983705968}, {"D": 8, "P": 16384, "rep": 2, "prefill_ttft_ms": 1554.4496020302176, "tpot_baseline_p50": 8.913785975892097}, {"D": 8, "P": 2048, "rep": 0, "prefill_ttft_ms": 138.16959800897166, "tpot_baseline_p50": 8.98516149027273}, {"D": 8, "P": 2048, "rep": 1, "prefill_ttft_ms": 139.2026540124789, "tpot_baseline_p50": 8.52142449002713}, {"D": 8, "P": 2048, "rep": 2, "prefill_ttft_ms": 138.62996903480962, "tpot_baseline_p50": 8.513206499628723}, {"D": 8, "P": 32768, "rep": 0, "prefill_ttft_ms": 4627.19459598884, "tpot_baseline_p50": 8.519548020558432}, {"D": 8, "P": 32768, "rep": 1, "prefill_ttft_ms": 4593.522923998535, "tpot_baseline_p50": 8.951486961450428}, {"D": 8, "P": 32768, "rep": 2, "prefill_ttft_ms": 4582.791887049098, "tpot_baseline_p50": 8.94658948527649}, {"D": 8, "P": 8192, "rep": 0, "prefill_ttft_ms": 595.0289380270988, "tpot_baseline_p50": 9.004380000988021}, {"D": 8, "P": 8192, "rep": 1, "prefill_ttft_ms": 599.6217179927044, "tpot_baseline_p50": 8.510532497894019}, {"D": 8, "P": 8192, "rep": 2, "prefill_ttft_ms": 594.1418160218745, "tpot_baseline_p50": 8.511811000062153}]
|