PD-sep server-side profiling: vLLM patches + per-request breakdown
Instrumentation patches (microbench/patches/):
- pd_profile.py: shared event emitter (VLLM_PD_PROFILE_LOG env var)
- apply_patches.py: idempotent patch installer for mooncake_connector.py
and scheduler.py, marks insertions with # PD_PROFILE_PATCH
- analyze_events.py: joins per-process JSONL event logs by transfer_id
into per-request phase durations
Seven events captured per request:
D_get_num_matched → P_zmq_received → P_prefill_done →
P_rdma_start → P_rdma_end → D_recv_complete → D_request_promoted
Driver fix (microbench/lifecycle/driver.py):
seed_prefix_cache now sends via the proxy URL so P and D both cache
the seeded prefix with matching block hashes. Previously seeding D
directly produced different block hashes than the proxy-routed
measurement requests, making incremental transfer impossible.
Real breakdown (fig_breakdown_real.png, server_breakdown.csv, n=93):
prefill_compute 620 ms median (95% of overhead)
rdma_transfer 42 ms median (~71 Gbps effective)
other overhead 10 ms median (dispatch + params + signal + promote)
Mooncake transfer is NOT the bottleneck. Even with bulk RDMA the
transfer cost is <10% of prefill cost for Qwen3-30B-A3B on H20.
This commit is contained in:
@@ -88,7 +88,14 @@ def make_new_tokens_prompt(num_tokens: int, unique_id: str) -> str:
|
||||
async def seed_prefix_cache(
|
||||
client: httpx.AsyncClient, url: str, model: str, num_tokens: int, session_id: str
|
||||
) -> bool:
|
||||
"""Send a request to D to warm its prefix cache with num_tokens of context.
|
||||
"""Warm BOTH P and D prefix caches by sending the seed through the PD-sep proxy.
|
||||
|
||||
Sending directly to D would only warm D's cache but produce block hashes that
|
||||
don't match what P later produces (different tokenization path). Sending through
|
||||
the proxy makes P do the prefill (P caches), pushes KV to D (D caches with
|
||||
matching hashes), so subsequent requests with the same prefix get incremental
|
||||
transfer on both sides.
|
||||
|
||||
Returns True if successful.
|
||||
"""
|
||||
if num_tokens == 0:
|
||||
@@ -318,11 +325,11 @@ async def main():
|
||||
print(f"Prior context C={C} tokens")
|
||||
print(f"{'='*60}")
|
||||
|
||||
# Seed D's prefix cache
|
||||
# Seed BOTH P and D prefix caches via the proxy
|
||||
if C > 0:
|
||||
print(f" Seeding D prefix cache with {C} tokens...")
|
||||
print(f" Seeding P+D prefix caches with {C} tokens via proxy...")
|
||||
success = await seed_prefix_cache(
|
||||
client, seed_endpoint, args.model, C, args.session_id
|
||||
client, pdsep_url, args.model, C, args.session_id
|
||||
)
|
||||
if not success:
|
||||
print(f" SKIP all configs with C={C} (cache seed failed)")
|
||||
|
||||
Reference in New Issue
Block a user