Gahow Wang
dc6d24d1ca
Adds unified_nixl_both to elastic_migration_v2: same picker as unified_kv_both (never triggers PD-sep), but launches vLLM with NixlConnector instead of MooncakeConnector. Compared against plain unified and unified_kv_both (Mooncake) we can now attribute the substrate overhead between "v1 connector framework irreducible cost" (proxied by the leaner NIXL) and "Mooncake implementation extra" (Mooncake - NIXL). Result (vs plain unified, both substrates never PD-sep): metric plain NIXL Mooncake TTFT p90 7.35s +37.9% +45.3% (NIXL: +7pp better) TPOT p90 17.1ms +15.5% +24.5% (NIXL: +9pp better) E2E p90 18.03s +17.4% +27.0% (NIXL: +10pp better) hotspot 3.667 +0.2% +19.0% (NIXL: keeps it flat) APC 79.4% -0.3pp -1.1pp interference - 5.58 8.57 (NIXL: ~35% lower) The cleanest signal is hotspot: NIXL preserves plain-unified's distribution (3.674 vs 3.667), while Mooncake's per-scheduler-step O(|cache|) `set(self._block_pool.cache.keys())` diff against _known_hash_keys (mooncake_connector.py:432-456) inflates routing imbalance by 19%. The hash sync runs unconditionally even when no direct_read consumer is present. Attribution: NIXL-plain ~= v1 framework irreducible cost (kv_buffer GPU memory, per-step SchedulerOutput.kv_connector_metadata round-trip, altered kv_cache_manager block-lifecycle). Mooncake-NIXL ~= Mooncake-specific overhead (the hash-sync loop and stricter delay_free semantics). Practical implication: NIXL is meaningfully better than Mooncake on this stack, but even NIXL imposes 16-38% across metrics — too expensive for selective-PD-sep on agentic workloads where the trigger rate is < 0.5%. Launch fixes required for NIXL multi-instance: - VLLM_NIXL_SIDE_CHANNEL_PORT must be unique per instance (default 5600; we use 5600+i). Without this, 7 of 8 instances silently hang in `zmq.error.ZMQError: Address already in use` and the launcher trap kills all of them at health-check timeout. - Health-check timeout raised from 180s to 360s; NIXL initialization (UCX agent + memory registration) is ~100-150s per instance under 8-way concurrent load, vs Mooncake's ~30-60s. New figure: fig_connector_substrate_attribution.png stacks plain / framework / Mooncake-extra / v2-branch overhead per metric. Existing figures (fig_kv_both_overhead, fig_three_way_hotspot) updated to include NIXL as a fourth bar. README updated with 4-way table, Result 1 reframed as "the cost is mostly framework, not Mooncake — but Mooncake adds the hotspot penalty", and the substrate-vs-PD-sep tradeoff math. Refs: nixl_connector.py:700 handshake listener bind, factory.py register_connector for the NixlConnector entry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
237 lines
7.6 KiB
JSON
237 lines
7.6 KiB
JSON
{
|
|
"rows": [
|
|
{
|
|
"policy": "capped",
|
|
"n_ok": 770,
|
|
"n_total": 770,
|
|
"ttft_p50_s": 1.1989156164927408,
|
|
"ttft_p90_s": 12.827629912580612,
|
|
"ttft_p99_s": 46.61752380923125,
|
|
"tpot_p50_s": 0.007231239004497606,
|
|
"tpot_p90_s": 0.015998617687440243,
|
|
"tpot_p99_s": 0.11515370831539476,
|
|
"e2e_p50_s": 2.598489043477457,
|
|
"e2e_p90_s": 21.245602010778384,
|
|
"e2e_p99_s": 74.60736650204846,
|
|
"apc_ratio": 0.3158312503528108,
|
|
"interference_index": 6.331064378362814,
|
|
"hotspot_index_ttft_p90": 2.0204268015410918,
|
|
"reuse_intra_frac": 0.9192657105586233,
|
|
"reuse_cross_frac": 0.0602232594931501,
|
|
"n_slow": 185,
|
|
"failure_counts": {
|
|
"cache_miss_large_append": 60,
|
|
"hot_worker_queue": 66,
|
|
"same_worker_prefill_overlap": 45,
|
|
"unknown": 14
|
|
}
|
|
},
|
|
{
|
|
"policy": "lmetric",
|
|
"n_ok": 1214,
|
|
"n_total": 1214,
|
|
"ttft_p50_s": 0.9387824369769078,
|
|
"ttft_p90_s": 15.671339168207492,
|
|
"ttft_p99_s": 53.56683189840049,
|
|
"tpot_p50_s": 0.008854518407308914,
|
|
"tpot_p90_s": 0.02122720699121469,
|
|
"tpot_p99_s": 0.18280341184277568,
|
|
"e2e_p50_s": 2.754255389008904,
|
|
"e2e_p90_s": 24.8209177934099,
|
|
"e2e_p99_s": 80.59924928059091,
|
|
"apc_ratio": 0.5694312382571595,
|
|
"interference_index": 6.530231061794441,
|
|
"hotspot_index_ttft_p90": 2.252837147833725,
|
|
"reuse_intra_frac": 0.9321238805590836,
|
|
"reuse_cross_frac": 0.05679481258506571,
|
|
"n_slow": 295,
|
|
"failure_counts": {
|
|
"cache_miss_large_append": 94,
|
|
"hot_worker_queue": 68,
|
|
"same_worker_prefill_overlap": 69,
|
|
"unknown": 64
|
|
}
|
|
},
|
|
{
|
|
"policy": "load_only",
|
|
"n_ok": 1214,
|
|
"n_total": 1214,
|
|
"ttft_p50_s": 1.2609447415161412,
|
|
"ttft_p90_s": 20.197147866390882,
|
|
"ttft_p99_s": 52.84285237012196,
|
|
"tpot_p50_s": 0.009231464695980247,
|
|
"tpot_p90_s": 0.026851662550158716,
|
|
"tpot_p99_s": 0.3211630676943426,
|
|
"e2e_p50_s": 3.58568156149704,
|
|
"e2e_p90_s": 33.459180271782685,
|
|
"e2e_p99_s": 93.95083751494239,
|
|
"apc_ratio": 0.5412093853102866,
|
|
"interference_index": 9.16424627504275,
|
|
"hotspot_index_ttft_p90": 1.2940319990630569,
|
|
"reuse_intra_frac": 0.9353191550754928,
|
|
"reuse_cross_frac": 0.053372184678592026,
|
|
"n_slow": 379,
|
|
"failure_counts": {
|
|
"cache_miss_large_append": 151,
|
|
"hot_worker_queue": 33,
|
|
"same_worker_prefill_overlap": 108,
|
|
"unknown": 87
|
|
}
|
|
},
|
|
{
|
|
"policy": "sticky",
|
|
"n_ok": 1214,
|
|
"n_total": 1214,
|
|
"ttft_p50_s": 0.5415176274836995,
|
|
"ttft_p90_s": 18.021296651283045,
|
|
"ttft_p99_s": 74.09429564891524,
|
|
"tpot_p50_s": 0.008952101894096181,
|
|
"tpot_p90_s": 0.03641285916619554,
|
|
"tpot_p99_s": 0.35152006935195085,
|
|
"e2e_p50_s": 2.081947358994512,
|
|
"e2e_p90_s": 34.62592205510591,
|
|
"e2e_p99_s": 139.68334607904353,
|
|
"apc_ratio": 0.7720092868396378,
|
|
"interference_index": 13.651718321568111,
|
|
"hotspot_index_ttft_p90": 2.727756623171119,
|
|
"reuse_intra_frac": 0.9327723488279339,
|
|
"reuse_cross_frac": 0.05495149683864246,
|
|
"n_slow": 234,
|
|
"failure_counts": {
|
|
"cache_miss_large_append": 20,
|
|
"hot_worker_queue": 51,
|
|
"same_worker_prefill_overlap": 134,
|
|
"unknown": 29
|
|
}
|
|
},
|
|
{
|
|
"policy": "unified",
|
|
"n_ok": 1213,
|
|
"n_total": 1214,
|
|
"ttft_p50_s": 0.4997710260213353,
|
|
"ttft_p90_s": 7.345769894809922,
|
|
"ttft_p99_s": 42.34170345296613,
|
|
"tpot_p50_s": 0.008079791456705824,
|
|
"tpot_p90_s": 0.017110194704198407,
|
|
"tpot_p99_s": 0.12655874612209597,
|
|
"e2e_p50_s": 1.7495028690318577,
|
|
"e2e_p90_s": 18.033410895219994,
|
|
"e2e_p99_s": 68.80023987947489,
|
|
"apc_ratio": 0.794261466256467,
|
|
"interference_index": null,
|
|
"hotspot_index_ttft_p90": 3.667136528736114,
|
|
"reuse_intra_frac": 0.9311187350942534,
|
|
"reuse_cross_frac": 0.056702150437367635,
|
|
"n_slow": 189,
|
|
"failure_counts": {
|
|
"cache_miss_large_append": 18,
|
|
"hot_worker_queue": 116,
|
|
"unknown": 55
|
|
}
|
|
},
|
|
{
|
|
"policy": "unified_kv_both",
|
|
"n_ok": 1214,
|
|
"n_total": 1214,
|
|
"ttft_p50_s": 0.4958424885116983,
|
|
"ttft_p90_s": 10.671844050800438,
|
|
"ttft_p99_s": 45.19353310586651,
|
|
"tpot_p50_s": 0.008573156389059812,
|
|
"tpot_p90_s": 0.021303916384344358,
|
|
"tpot_p99_s": 0.21501837408937963,
|
|
"e2e_p50_s": 1.9310281965008471,
|
|
"e2e_p90_s": 22.8941433175176,
|
|
"e2e_p99_s": 76.06128971517893,
|
|
"apc_ratio": 0.7828397082703908,
|
|
"interference_index": 8.571603637346875,
|
|
"hotspot_index_ttft_p90": 4.363145984888287,
|
|
"reuse_intra_frac": 0.9313000825240145,
|
|
"reuse_cross_frac": 0.056182260858791105,
|
|
"n_slow": 198,
|
|
"failure_counts": {
|
|
"cache_miss_large_append": 28,
|
|
"hot_worker_queue": 34,
|
|
"same_worker_prefill_overlap": 87,
|
|
"unknown": 49
|
|
}
|
|
},
|
|
{
|
|
"policy": "unified_nixl_both",
|
|
"n_ok": 1214,
|
|
"n_total": 1214,
|
|
"ttft_p50_s": 0.5138550130068325,
|
|
"ttft_p90_s": 10.127110345300755,
|
|
"ttft_p99_s": 44.5789094621703,
|
|
"tpot_p50_s": 0.008423213202440761,
|
|
"tpot_p90_s": 0.019759515867947428,
|
|
"tpot_p99_s": 0.1079433335279151,
|
|
"e2e_p50_s": 1.866590676479973,
|
|
"e2e_p90_s": 21.179128799570027,
|
|
"e2e_p99_s": 96.01196486203865,
|
|
"apc_ratio": 0.791441828164218,
|
|
"interference_index": 5.580715970433481,
|
|
"hotspot_index_ttft_p90": 3.673957447190547,
|
|
"reuse_intra_frac": 0.930632797070364,
|
|
"reuse_cross_frac": 0.05718149217603143,
|
|
"n_slow": 192,
|
|
"failure_counts": {
|
|
"cache_miss_large_append": 21,
|
|
"hot_worker_queue": 75,
|
|
"same_worker_prefill_overlap": 72,
|
|
"unknown": 24
|
|
}
|
|
},
|
|
{
|
|
"policy": "unified_v2",
|
|
"n_ok": 1214,
|
|
"n_total": 1214,
|
|
"ttft_p50_s": 0.4851180294645019,
|
|
"ttft_p90_s": 10.97665627548705,
|
|
"ttft_p99_s": 49.44861259821856,
|
|
"tpot_p50_s": 0.008261419251554481,
|
|
"tpot_p90_s": 0.018414033703249108,
|
|
"tpot_p99_s": 0.20999689490980364,
|
|
"e2e_p50_s": 1.8092182099935599,
|
|
"e2e_p90_s": 22.528888442111203,
|
|
"e2e_p99_s": 82.40234094743934,
|
|
"apc_ratio": 0.7758437361549086,
|
|
"interference_index": 8.45656745230457,
|
|
"hotspot_index_ttft_p90": 3.9096187869766164,
|
|
"reuse_intra_frac": 0.9324663389938368,
|
|
"reuse_cross_frac": 0.055154184817413764,
|
|
"n_slow": 198,
|
|
"failure_counts": {
|
|
"cache_miss_large_append": 36,
|
|
"hot_worker_queue": 26,
|
|
"same_worker_prefill_overlap": 82,
|
|
"unknown": 54
|
|
}
|
|
},
|
|
{
|
|
"policy": "unified_v2_strict",
|
|
"n_ok": 1214,
|
|
"n_total": 1214,
|
|
"ttft_p50_s": 0.4849805940175429,
|
|
"ttft_p90_s": 8.960840504511737,
|
|
"ttft_p99_s": 44.63598358390898,
|
|
"tpot_p50_s": 0.008222105788569446,
|
|
"tpot_p90_s": 0.018078321745916927,
|
|
"tpot_p99_s": 0.14616439095890604,
|
|
"e2e_p50_s": 1.8335122870048508,
|
|
"e2e_p90_s": 22.435233922180526,
|
|
"e2e_p99_s": 68.254801789901,
|
|
"apc_ratio": 0.789281361129855,
|
|
"interference_index": 6.231677388887276,
|
|
"hotspot_index_ttft_p90": 2.7334230011629197,
|
|
"reuse_intra_frac": 0.9309082618411778,
|
|
"reuse_cross_frac": 0.05689887985860397,
|
|
"n_slow": 186,
|
|
"failure_counts": {
|
|
"cache_miss_large_append": 26,
|
|
"hot_worker_queue": 44,
|
|
"same_worker_prefill_overlap": 73,
|
|
"unknown": 43
|
|
}
|
|
}
|
|
]
|
|
} |