Add NIXL substrate isolation control + attribution decomposition

Adds unified_nixl_both to elastic_migration_v2: same picker as
unified_kv_both (never triggers PD-sep), but launches vLLM with
NixlConnector instead of MooncakeConnector. Compared against plain
unified and unified_kv_both (Mooncake) we can now attribute the
substrate overhead between "v1 connector framework irreducible
cost" (proxied by the leaner NIXL) and "Mooncake implementation
extra" (Mooncake - NIXL).

Result (vs plain unified, both substrates never PD-sep):

   metric          plain    NIXL          Mooncake
   TTFT p90        7.35s    +37.9%        +45.3%      (NIXL: +7pp better)
   TPOT p90        17.1ms   +15.5%        +24.5%      (NIXL: +9pp better)
   E2E p90         18.03s   +17.4%        +27.0%      (NIXL: +10pp better)
   hotspot         3.667    +0.2%         +19.0%      (NIXL: keeps it flat)
   APC             79.4%    -0.3pp        -1.1pp
   interference    -        5.58          8.57         (NIXL: ~35% lower)

The cleanest signal is hotspot: NIXL preserves plain-unified's
distribution (3.674 vs 3.667), while Mooncake's per-scheduler-step
O(|cache|) `set(self._block_pool.cache.keys())` diff against
_known_hash_keys (mooncake_connector.py:432-456) inflates routing
imbalance by 19%. The hash sync runs unconditionally even when no
direct_read consumer is present.

Attribution: NIXL-plain ~= v1 framework irreducible cost (kv_buffer
GPU memory, per-step SchedulerOutput.kv_connector_metadata
round-trip, altered kv_cache_manager block-lifecycle). Mooncake-NIXL
~= Mooncake-specific overhead (the hash-sync loop and stricter
delay_free semantics).

Practical implication: NIXL is meaningfully better than Mooncake on
this stack, but even NIXL imposes 16-38% across metrics — too
expensive for selective-PD-sep on agentic workloads where the
trigger rate is < 0.5%.

Launch fixes required for NIXL multi-instance:
- VLLM_NIXL_SIDE_CHANNEL_PORT must be unique per instance (default
  5600; we use 5600+i). Without this, 7 of 8 instances silently hang
  in `zmq.error.ZMQError: Address already in use` and the launcher
  trap kills all of them at health-check timeout.
- Health-check timeout raised from 180s to 360s; NIXL initialization
  (UCX agent + memory registration) is ~100-150s per instance under
  8-way concurrent load, vs Mooncake's ~30-60s.

New figure: fig_connector_substrate_attribution.png stacks plain /
framework / Mooncake-extra / v2-branch overhead per metric.
Existing figures (fig_kv_both_overhead, fig_three_way_hotspot)
updated to include NIXL as a fourth bar.

README updated with 4-way table, Result 1 reframed as "the cost is
mostly framework, not Mooncake — but Mooncake adds the hotspot
penalty", and the substrate-vs-PD-sep tradeoff math.

Refs: nixl_connector.py:700 handshake listener bind, factory.py
register_connector for the NixlConnector entry.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-26 16:02:12 +08:00
parent 645b067dd4
commit dc6d24d1ca
8 changed files with 235 additions and 83 deletions

View File

@@ -155,6 +155,32 @@
"unknown": 49
}
},
{
"policy": "unified_nixl_both",
"n_ok": 1214,
"n_total": 1214,
"ttft_p50_s": 0.5138550130068325,
"ttft_p90_s": 10.127110345300755,
"ttft_p99_s": 44.5789094621703,
"tpot_p50_s": 0.008423213202440761,
"tpot_p90_s": 0.019759515867947428,
"tpot_p99_s": 0.1079433335279151,
"e2e_p50_s": 1.866590676479973,
"e2e_p90_s": 21.179128799570027,
"e2e_p99_s": 96.01196486203865,
"apc_ratio": 0.791441828164218,
"interference_index": 5.580715970433481,
"hotspot_index_ttft_p90": 3.673957447190547,
"reuse_intra_frac": 0.930632797070364,
"reuse_cross_frac": 0.05718149217603143,
"n_slow": 192,
"failure_counts": {
"cache_miss_large_append": 21,
"hot_worker_queue": 75,
"same_worker_prefill_overlap": 72,
"unknown": 24
}
},
{
"policy": "unified_v2",
"n_ok": 1214,

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,24 @@
{
"hotspot_index_ttft_p90": 3.673957447190547,
"per_worker_latency_p90_s": {
"http://127.0.0.1:8000": 21.5702620673168,
"http://127.0.0.1:8001": 21.44246501957532,
"http://127.0.0.1:8002": 7.497513776784784,
"http://127.0.0.1:8003": 18.975387462502113,
"http://127.0.0.1:8004": 27.733961877820548,
"http://127.0.0.1:8005": 14.178356938017535,
"http://127.0.0.1:8006": 25.44877168269595,
"http://127.0.0.1:8007": 54.500166546402035
},
"per_worker_ttft_p90_s": {
"http://127.0.0.1:8000": 7.380765471985799,
"http://127.0.0.1:8001": 14.109222683508415,
"http://127.0.0.1:8002": 3.001173847797329,
"http://127.0.0.1:8003": 14.087287129514152,
"http://127.0.0.1:8004": 14.151121024426537,
"http://127.0.0.1:8005": 6.165523712011057,
"http://127.0.0.1:8006": 6.314287615299688,
"http://127.0.0.1:8007": 39.43635586597957
},
"status": "supported"
}