Add GPU utilization A/B test and fix cache-aware proxy bugs
- GPU monitor: 5s interval nvidia-smi sampling during benchmarks - A/B test script: clean restart + monitor + benchmark for Combined vs PD-Sep - Fixed proxy: await bootstrap init (race condition), normalized LB scoring - Fixed port conflicts: proxy 9090 to avoid bootstrap 9000 clash Key finding: PD-Sep GPU utilization is 40% of Combined (12.4% vs 30.5%) - Decode GPUs: mean=7.8%, max=47% (memory-bound, compute wasted) - Prefill GPUs: active only 17% of samples (bursty, idle between requests) - Combined: 8 GPUs flexibly used, mean=30.5%, active=64% Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -132,7 +132,7 @@ async def lifespan(app: FastAPI):
|
||||
prefill_instances.append(InstanceState(url, bp))
|
||||
for url in global_args.decode:
|
||||
decode_instances.append(InstanceState(url))
|
||||
asyncio.create_task(init_prefill_bootstrap(prefill_instances, app.state.ready))
|
||||
await init_prefill_bootstrap(prefill_instances, app.state.ready)
|
||||
print(f"PD-Sep mode: {len(prefill_instances)}P + {len(decode_instances)}D")
|
||||
|
||||
yield
|
||||
|
||||
Reference in New Issue
Block a user