gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: gahow:06dd17544183167c8250c8b25433a74df41b65b8
Branches Tags
gahow:main
...
compare: gahow:dc6d24d1ca74268e4cc2342c6a720bc01d73cd85
Branches Tags
gahow:main

4 Commits
06dd175441 ... dc6d24d1ca

Author SHA1 Message Date
Gahow Wang
dc6d24d1ca Add NIXL substrate isolation control + attribution decomposition 
Adds unified_nixl_both to elastic_migration_v2: same picker as
unified_kv_both (never triggers PD-sep), but launches vLLM with
NixlConnector instead of MooncakeConnector. Compared against plain
unified and unified_kv_both (Mooncake) we can now attribute the
substrate overhead between "v1 connector framework irreducible
cost" (proxied by the leaner NIXL) and "Mooncake implementation
extra" (Mooncake - NIXL).

Result (vs plain unified, both substrates never PD-sep):

   metric          plain    NIXL          Mooncake
   TTFT p90        7.35s    +37.9%        +45.3%      (NIXL: +7pp better)
   TPOT p90        17.1ms   +15.5%        +24.5%      (NIXL: +9pp better)
   E2E p90         18.03s   +17.4%        +27.0%      (NIXL: +10pp better)
   hotspot         3.667    +0.2%         +19.0%      (NIXL: keeps it flat)
   APC             79.4%    -0.3pp        -1.1pp
   interference    -        5.58          8.57         (NIXL: ~35% lower)

The cleanest signal is hotspot: NIXL preserves plain-unified's
distribution (3.674 vs 3.667), while Mooncake's per-scheduler-step
O(|cache|) `set(self._block_pool.cache.keys())` diff against
_known_hash_keys (mooncake_connector.py:432-456) inflates routing
imbalance by 19%. The hash sync runs unconditionally even when no
direct_read consumer is present.

Attribution: NIXL-plain ~= v1 framework irreducible cost (kv_buffer
GPU memory, per-step SchedulerOutput.kv_connector_metadata
round-trip, altered kv_cache_manager block-lifecycle). Mooncake-NIXL
~= Mooncake-specific overhead (the hash-sync loop and stricter
delay_free semantics).

Practical implication: NIXL is meaningfully better than Mooncake on
this stack, but even NIXL imposes 16-38% across metrics — too
expensive for selective-PD-sep on agentic workloads where the
trigger rate is < 0.5%.

Launch fixes required for NIXL multi-instance:
- VLLM_NIXL_SIDE_CHANNEL_PORT must be unique per instance (default
  5600; we use 5600+i). Without this, 7 of 8 instances silently hang
  in `zmq.error.ZMQError: Address already in use` and the launcher
  trap kills all of them at health-check timeout.
- Health-check timeout raised from 180s to 360s; NIXL initialization
  (UCX agent + memory registration) is ~100-150s per instance under
  8-way concurrent load, vs Mooncake's ~30-60s.

New figure: fig_connector_substrate_attribution.png stacks plain /
framework / Mooncake-extra / v2-branch overhead per metric.
Existing figures (fig_kv_both_overhead, fig_three_way_hotspot)
updated to include NIXL as a fourth bar.

README updated with 4-way table, Result 1 reframed as "the cost is
mostly framework, not Mooncake — but Mooncake adds the hotspot
penalty", and the substrate-vs-PD-sep tradeoff math.

Refs: nixl_connector.py:700 handshake listener bind, factory.py
register_connector for the NixlConnector entry.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-26 16:02:12 +08:00
Gahow Wang
645b067dd4 Fix review bugs: PD-sep counter leaks, hardcoded paths, missing deps 
Critical:
- cache_aware_proxy: _handle_pd_sep leaked p_inst.num_requests (never
  decremented) and never managed d_inst.num_requests; fix media_type
  from application/json to text/event-stream for SSE stream

High:
- b3_sweep/b3_isolated_policy/b3_analyze: replace hardcoded
  /home/admin/cpfs/wjh/ ROOT with script-relative $(dirname "$0")/..
- b3_analyze: replace hardcoded 8-port WORKER_MAP with dynamic
  generation from BASE_PORT and N_INSTANCES

Medium:
- analyze_breakdown: warn on stderr when records are skipped (was silent)
- deploy_vllm_patches: fail-fast on SSH/SCP errors instead of
  continuing with empty VENV_SITE
- pyproject.toml: declare fastapi and uvicorn as runtime dependencies
- launch_elastic_p2p: kill EngineCore and proxy in trap handler to
  prevent GPU memory leaks on exit
 2026-05-26 15:54:55 +08:00
Gahow Wang
0eb49dcc34 Fix NIXL multi-instance port conflict: per-instance SIDE_CHANNEL_PORT 
NIXL's _nixl_handshake_listener (vllm/distributed/kv_transfer/
kv_connector/v1/nixl_connector.py:700) binds a ZMQ ROUTER socket on
the side_channel_port, which defaults to 5600. When 8 NIXL vLLMs
launch concurrently on the same host all 8 race for tcp://localhost:5600;
exactly one succeeds and the others silently hang in the listener
thread with:

    zmq.error.ZMQError: Address already in use (addr='tcp://localhost:5600')

The engines themselves never reach "Application startup complete"
and the b3_isolated_policy.sh health-check times out. First observed
when 7 of 8 inst_X.log files contained the ZMQ error and the 8th
(by random ordering) was the one healthy instance.

Fix: set VLLM_NIXL_SIDE_CHANNEL_PORT=$((5600 + i)) per instance in
the NIXL launch branch. Each engine now gets a distinct handshake
port (5600..5607 by default). Verified: all 8 instances now reach
"Application startup complete" within the 360 s health budget.

This is NIXL-specific; Mooncake uses VLLM_MOONCAKE_BOOTSTRAP_PORT
which we were already varying per instance.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-26 15:09:16 +08:00
Gahow Wang
151bf33541 Add unified_nixl_both policy: NIXL connector isolation control 
Adds a NIXL-backed counterpart to unified_kv_both so we can attribute
the kv_both substrate overhead measured in the elastic_migration_v2
section to either Mooncake-specific code or a generic v1-connector
cost shared by all connectors.

- scripts/cache_aware_proxy.py: register --policy unified_nixl_both.
  Picker is identical to unified (and unified_kv_both); routing
  decisions never go through the PD-sep branch. Differs only at the
  vLLM launch layer.
- scripts/b3_isolated_policy.sh: new KV_CONNECTOR env var
  (Mooncake|Nixl), auto-set based on POLICY. NIXL launch path uses
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
  with no VLLM_MOONCAKE_BOOTSTRAP_PORT (NIXL uses UCX side-channels).
- Health-check timeout: 90 iterations * 2s -> 180 iterations * 2s
  (180s -> 360s). Empirically NIXL needs ~100-150s per instance to
  initialize the UCX agent and register KV cache memory; 8
  concurrent NIXL launches frequently overshoot the previous 180s
  budget. Mooncake is unaffected (still finishes well inside the new
  budget). The 8-vLLM unified_nixl_both first launch tripped the
  old timeout despite 7/8 instances reaching startup-complete.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-26 14:57:54 +08:00
16 changed files with 326 additions and 108 deletions
Expand all files Collapse all files

173
analysis/characterization/elastic_migration_v2/README.md
View File

@@ -9,82 +9,127 @@ Model: Qwen3-Coder-30B-A3B-Instruct, 8 × TP1 on H20
This section explores whether the **B2-confirmed same-worker
prefill–decode interference** can be relieved by selectively
migrating prefill to a different worker for the requests where the
interference cost would dominate the transfer cost. We implement two
flavors of the policy (strict gates, then relaxed gates) and a clean
isolation control (`unified_kv_both`: same picker as `unified`, but
the vLLMs are launched in `kv_role=kv_both` so the Mooncake
substrate is on but never triggers).
interference cost would dominate the transfer cost. We implement
two flavors of the routing policy (strict gates, then relaxed
gates) and **two isolation controls** that use the unified picker
but launch vLLMs in `kv_role=kv_both` so the connector substrate
is on but never PD-seps:
Three findings:
- `unified_kv_both`: with **MooncakeConnector**
- `unified_nixl_both`: with **NixlConnector** (NVIDIA's official
v1 connector; isolates connector implementation from policy)
1. **`kv_role=kv_both` alone imposes a heavy always-on tax**: TTFT
p90 +45%, TPOT p90 +25%, hotspot index +19% vs plain `unified`,
with no PD-sep ever firing.
2. **PD-sep almost never triggers on a real agentic workload**:
Four findings:
1. **`kv_role=kv_both` imposes a substantial always-on tax even
when no PD-sep ever fires**: with Mooncake it's TTFT p90 +45%,
TPOT p90 +25%, hotspot +19%; with NIXL it's TTFT p90 +38%,
TPOT p90 +16%, hotspot +0.2%.
2. **About half of the substrate cost is generic v1-connector
framework overhead** (proxied by NIXL since it's the leanest
implementation): KV buffer GPU memory cut from the model's
working budget, `SchedulerOutput.kv_connector_metadata`
round-trip, and altered `kv_cache_manager` block-lifecycle
semantics. **NIXL is meaningfully better than Mooncake** but
still imposes a 16-38% tax vs no connector.
3. **PD-sep almost never triggers on a real agentic workload**:
0.16% with strict gates, 0.41% with relaxed gates. Agentic
workloads have 93% intra-session reuse, so most requests land on
workers that already hold cache — the uncached tail is too small
to be worth migrating.
3. **When PD-sep does fire, the cost model is wrong by ~10–20×**:
workloads have 93% intra-session reuse, so most requests land
on workers that already hold cache — the uncached tail is too
small to be worth migrating.
4. **When PD-sep does fire, the cost model is wrong by ~10–20×**:
the calibrated `0.3s + bytes / 2.7 GB/s` predicts 1–2 s migrate
cost; observed TTFT on triggered requests is 12–45 s. The same
D-side block-reservation pressure and absence of layerwise
pipelining that the E2 audit flagged still dominate.
cost; observed TTFT on triggered requests is 12–45 s.
The net latency of `unified_v2` is **not better than plain
`unified`**. Improving agentic PD-sep requires fixing the underlying
Mooncake transfer mechanism (E2 patches 6.1 lazy block reservation
and 6.3 layerwise pipelining), not the routing decision.
`unified`** under either Mooncake or NIXL substrate. Improving
agentic PD-sep requires (a) using the leaner connector (NIXL >
Mooncake by 5-19 pp across metrics), and (b) fixing the underlying
transfer mechanism (E2 patches 6.1 lazy block reservation and 6.3
layerwise pipelining), not just the routing decision.
## Substrate
We compare three policies on identical traces:
We compare four policies on identical traces:
| policy | picker | vLLM launch mode | what's it for |
|---|---|---|---|
| `unified` | hybrid affinity + LMetric | plain (no Mooncake) | the headline baseline |
| `unified_kv_both` | same as `unified` | `kv_role=kv_both` + bootstrap | isolation control: how much does kv_both *alone* cost? |
| `unified_v2` | unified + selective PD-sep | `kv_role=kv_both` + bootstrap | the actual experiment |
| `unified` | hybrid affinity + LMetric | plain (no connector) | the headline baseline |
| `unified_kv_both` | same as `unified` | `MooncakeConnector` + `kv_both` | substrate control: Mooncake cost without PD-sep |
| `unified_nixl_both` | same as `unified` | `NixlConnector` + `kv_both` | substrate control: NIXL cost without PD-sep, attributes overhead to "framework vs Mooncake" |
| `unified_v2` | unified + selective PD-sep | `MooncakeConnector` + `kv_both` + bootstrap | the actual experiment |
All three use the same trace, the same 8-instance topology, the same
All four use the same trace, the same 8-instance topology, the same
shadow-drift–corrected proxy (`scripts/cache_aware_proxy.py` post-fix
`95c8ef8`). Plain `unified` was rerun on the patched proxy
(`b3_sweep_20260525_095043/unified`) under the same conditions.
## Result 1 — kv_both is expensive by itself
NIXL required two launch fixes beyond Mooncake:
- `VLLM_NIXL_SIDE_CHANNEL_PORT` must be unique per instance
(default 5600 → 5600..5607); otherwise instances 2..8 silently
hang in `zmq.error.ZMQError: Address already in use`.
- Health-check timeout had to be raised from 180 s to 360 s
because NIXL initialization (UCX agent + memory registration)
takes ~100-150 s per instance under 8-way concurrent launch.
## Result 1 — kv_both is expensive by itself, and only partly Mooncake's fault
![](figures/fig_kv_both_overhead.png)
Switching the vLLM launch from plain to `kv_role=kv_both` without
ever triggering PD-sep already costs:
ever triggering PD-sep imposes a substrate tax. We compare the two
connectors available in vendored vLLM:
| metric | plain `unified` | `unified_kv_both` | Δ |
|---|---:|---:|---|
| TTFT p50 | 0.50 s | 0.50 s | +0% |
| TTFT p90 | 7.35 s | 10.67 s | **+45%** |
| TTFT p99 | 42.34 s | 45.19 s | +7% |
| TPOT p90 | 17.1 ms | 21.3 ms | **+25%** |
| E2E p90 | 18.03 s | 22.89 s | **+27%** |
| APC | 79.4% | 78.3% | −1.1 pp |
| hotspot index | 3.667 | **4.363** | **+19%** |
| metric | plain `unified` | `unified_nixl_both` | `unified_kv_both` (Mooncake) |
|---|---:|---:|---:|
| TTFT p50 | 0.50 s | 0.51 s (+1%) | 0.50 s (+0%) |
| **TTFT p90** | 7.35 s | **10.13 s (+38%)** | **10.67 s (+45%)** |
| TTFT p99 | 42.34 s | 44.58 s (+5%) | 45.19 s (+7%) |
| TPOT p90 | 17.1 ms | **19.8 ms (+16%)** | **21.3 ms (+25%)** |
| E2E p90 | 18.03 s | **21.18 s (+17%)** | **22.89 s (+27%)** |
| APC | 79.4% | 79.1% (−0.3 pp) | 78.3% (−1.1 pp) |
| **hotspot index** | 3.667 | **3.674 (+0.2%)** | **4.363 (+19%)** |
| interference index | n/a | 5.58 | 8.57 |
Two contributing factors:
![](figures/fig_connector_substrate_attribution.png)
1. **The Mooncake `MooncakeConnector` runs even when no transfer is
pending.** Every scheduler step it walks `set(cache.keys())`
against `_known_hash_keys` (E2 audit §6.5) and updates the
`KVConnectorMetadata`. This is O(|cache|) per step on every
engine, even when no producer/consumer relationship is active.
2. **Block reservation semantics differ** under kv_both. The
scheduler treats blocks as candidates for export-to-others, so
the prefix cache LRU pressure is slightly different (we lose 1
pp APC).
Reading the table from left to right gives a clean attribution:
Practical implication: **you don't enable kv_both for free**. If a
deployment wants the option to do PD-sep selectively, the 45% TTFT
p90 tax applies even on requests that stay local. This needs to
recoverable cost before any selective-PD-sep policy is worth
shipping.
- **NIXL−plain** = the **v1-connector framework's irreducible cost**
(TTFT p90 +38%, TPOT p90 +16%, E2E p90 +17%). This is the cost
*any* v1 KV connector imposes:
- the 1 GB `kv_buffer_size` carved from `gpu-memory-utilization`,
reducing the KV cache budget;
- per-step `SchedulerOutput.kv_connector_metadata` serialization
and round-trip through the connector worker;
- altered block-lifecycle semantics in `kv_cache_manager`
(`delay_free_blocks=True` is the default once any connector is
loaded, slowing LRU eviction).
- **Mooncake−NIXL** = the **Mooncake-implementation-specific extra**
(TTFT p90 +7 pp, TPOT p90 +9 pp, E2E p90 +10 pp, hotspot +19 pp).
This is the cost Mooncake's design choices add on top of the
generic framework:
- per-scheduler-step `set(self._block_pool.cache.keys())` diff
against `_known_hash_keys` (`mooncake_connector.py:432-456`)
walks O(|cache|) on every step on every engine, costing ~4 M
set operations per second on a 200 k-block cache;
- the hash sync runs even when no `direct_read` consumer is
present, so the cost is paid unconditionally;
- block-lifecycle is further constrained because Mooncake
requires `delay_free` until the explicit `finished_sending`
arrives, vs NIXL which can release blocks earlier.
The **most striking gap is hotspot**: Mooncake's per-step hash
sync runs on the scheduler's GIL and disrupts the timeliness of
routing decisions, amplifying load imbalance by 19%. NIXL has no
equivalent global-state maintenance and preserves the plain-unified
hotspot to within 0.2%.
Practical implication: **you don't enable any v1 KV connector for
free**, but if you have to enable one, NIXL is meaningfully cheaper
than Mooncake. Even NIXL's 38% TTFT p90 tax is large enough that
PD-sep needs to recover it on a non-trivial fraction of requests
before being worth it.
## Result 2 — PD-sep rarely fires on a real agentic trace
@@ -153,24 +198,24 @@ The first-token clock for the 49 k request is **21× the model's
prediction**. This is not a small mis-tuning — it's a structurally
different model.
## Result 4 — three-way comparison
## Result 4 — four-way comparison
![](figures/fig_three_way_hotspot.png)
The full table:
| metric | unified (plain) | unified_kv_both | unified_v2 (relaxed) |
|---|---:|---:|---:|
| n_ok | 1214 | 1214 | 1214 |
| TTFT p50 | 0.50 s | 0.50 s | 0.49 s |
| TTFT p90 | 7.35 s | 10.67 s | 10.98 s |
| TTFT p99 | 42.34 s | 45.19 s | 49.45 s |
| TPOT p90 | 17.1 ms | 21.3 ms | 18.4 ms |
| E2E p90 | 18.03 s | 22.89 s | 22.53 s |
| APC | 79.4% | 78.3% | 77.6% |
| interference index | n/a (no engine_state) | 8.57 | 8.46 |
| hotspot index | 3.667 | 4.363 | 3.910 |
| n_slow | 189 | 198 | 198 |
| metric | unified (plain) | unified_nixl_both | unified_kv_both (Mooncake) | unified_v2 (relaxed) |
|---|---:|---:|---:|---:|
| n_ok | 1214 | 1214 | 1214 | 1214 |
| TTFT p50 | 0.50 s | 0.51 s | 0.50 s | 0.49 s |
| TTFT p90 | 7.35 s | 10.13 s | 10.67 s | 10.98 s |
| TTFT p99 | 42.34 s | 44.58 s | 45.19 s | 49.45 s |
| TPOT p90 | 17.1 ms | 19.8 ms | 21.3 ms | 18.4 ms |
| E2E p90 | 18.03 s | 21.18 s | 22.89 s | 22.53 s |
| APC | 79.4% | 79.1% | 78.3% | 77.6% |
| interference index | n/a | 5.58 | 8.57 | 8.46 |
| hotspot index | 3.667 | 3.674 | 4.363 | 3.910 |
| n_slow | 189 | 192 | 198 | 198 |
### v2 vs the kv_both control (the right comparison)

26
analysis/characterization/elastic_migration_v2/data/b3_policy_comparison.json
View File

@@ -155,6 +155,32 @@
"unknown": 49
}
},
{
"policy": "unified_nixl_both",
"n_ok": 1214,
"n_total": 1214,
"ttft_p50_s": 0.5138550130068325,
"ttft_p90_s": 10.127110345300755,
"ttft_p99_s": 44.5789094621703,
"tpot_p50_s": 0.008423213202440761,
"tpot_p90_s": 0.019759515867947428,
"tpot_p99_s": 0.1079433335279151,
"e2e_p50_s": 1.866590676479973,
"e2e_p90_s": 21.179128799570027,
"e2e_p99_s": 96.01196486203865,
"apc_ratio": 0.791441828164218,
"interference_index": 5.580715970433481,
"hotspot_index_ttft_p90": 3.673957447190547,
"reuse_intra_frac": 0.930632797070364,
"reuse_cross_frac": 0.05718149217603143,
"n_slow": 192,
"failure_counts": {
"cache_miss_large_append": 21,
"hot_worker_queue": 75,
"same_worker_prefill_overlap": 72,
"unknown": 24
}
},
{
"policy": "unified_v2",
"n_ok": 1214,

1
analysis/characterization/elastic_migration_v2/data/breakdown_unified_nixl_both.json Normal file
View File

File diff suppressed because one or more lines are too long

24
analysis/characterization/elastic_migration_v2/data/per_worker_unified_nixl_both.json Normal file
View File

@@ -0,0 +1,24 @@
{
"hotspot_index_ttft_p90": 3.673957447190547,
"per_worker_latency_p90_s": {
"http://127.0.0.1:8000": 21.5702620673168,
"http://127.0.0.1:8001": 21.44246501957532,
"http://127.0.0.1:8002": 7.497513776784784,
"http://127.0.0.1:8003": 18.975387462502113,
"http://127.0.0.1:8004": 27.733961877820548,
"http://127.0.0.1:8005": 14.178356938017535,
"http://127.0.0.1:8006": 25.44877168269595,
"http://127.0.0.1:8007": 54.500166546402035
},
"per_worker_ttft_p90_s": {
"http://127.0.0.1:8000": 7.380765471985799,
"http://127.0.0.1:8001": 14.109222683508415,
"http://127.0.0.1:8002": 3.001173847797329,
"http://127.0.0.1:8003": 14.087287129514152,
"http://127.0.0.1:8004": 14.151121024426537,
"http://127.0.0.1:8005": 6.165523712011057,
"http://127.0.0.1:8006": 6.314287615299688,
"http://127.0.0.1:8007": 39.43635586597957
},
"status": "supported"
}

BIN
analysis/characterization/elastic_migration_v2/figures/fig_connector_substrate_attribution.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 83 KiB

BIN
analysis/characterization/elastic_migration_v2/figures/fig_kv_both_overhead.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 70 KiB

After

Width:  |  Height:  |  Size: 86 KiB

BIN
analysis/characterization/elastic_migration_v2/figures/fig_three_way_hotspot.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 54 KiB

After

Width:  |  Height:  |  Size: 57 KiB

94
analysis/characterization/elastic_migration_v2/render_figures.py
View File

@@ -34,37 +34,39 @@ def _load(name: str):
POLICY_COLORS = {
"unified": "#2ca02c",
"unified_kv_both": "#9467bd",
"unified_v2": "#d62728",
"unified_v2_strict": "#ff7f0e",
"unified": "#2ca02c",
"unified_kv_both": "#9467bd",
"unified_nixl_both": "#1f77b4",
"unified_v2": "#d62728",
"unified_v2_strict": "#ff7f0e",
}
def fig_kv_both_overhead():
comp = _load("b3_policy_comparison.json")
by = {r["policy"]: r for r in comp["rows"]}
pols = ["unified", "unified_kv_both", "unified_v2"]
pols = ["unified", "unified_kv_both", "unified_nixl_both", "unified_v2"]
metrics = [
("TTFT p90 (s)", lambda r: r["ttft_p90_s"]),
("TPOT p90 (ms)", lambda r: r["tpot_p90_s"] * 1000),
("E2E p90 (s)", lambda r: r["e2e_p90_s"]),
("hotspot index", lambda r: r["hotspot_index_ttft_p90"]),
]
fig, axes = plt.subplots(1, 4, figsize=(14, 4))
fig, axes = plt.subplots(1, 4, figsize=(15, 4.2))
for ax, (label, fn) in zip(axes, metrics):
vals = [fn(by[p]) for p in pols]
bars = ax.bar(pols, vals,
labels_short = [p.replace("unified_", "") for p in pols]
labels_short[0] = "plain"
bars = ax.bar(labels_short, vals,
color=[POLICY_COLORS[p] for p in pols],
edgecolor="black", linewidth=0.5)
ax.set_title(label)
ax.tick_params(axis="x", rotation=20, labelsize=9)
ax.tick_params(axis="x", rotation=15, labelsize=9)
for b, v in zip(bars, vals):
ax.text(b.get_x() + b.get_width() / 2, v,
f"{v:.2f}" if v < 100 else f"{v:.0f}",
ha="center", va="bottom", fontsize=9)
ax.grid(alpha=0.3, axis="y")
# delta annotation
baseline = vals[0]
for i, v in enumerate(vals):
if i == 0:
@@ -74,8 +76,8 @@ def fig_kv_both_overhead():
fontsize=10, fontweight="bold",
color="darkred" if pct > 0 else "darkgreen")
fig.suptitle(
"kv_both adds ~45% to TTFT p90 even without PD-sep firing.\n"
"v2's PD-sep barely recovers the gap (and overshoots TTFT p99)."
"Mooncake substrate adds 19-45% across metrics; NIXL is 5-19pp better but\n"
"still 16-38% above plain. v2's 5 PD-sep events don't recover the substrate tax."
)
fig.tight_layout()
fig.savefig(OUT / "fig_kv_both_overhead.png", dpi=120)
@@ -203,27 +205,29 @@ def fig_v2_predicted_vs_actual():
def fig_three_way_hotspot():
pols = ["unified", "unified_kv_both", "unified_v2"]
pols = ["unified", "unified_kv_both", "unified_nixl_both", "unified_v2"]
per_worker = {p: _load(f"per_worker_{p}.json") for p in pols}
workers = sorted(per_worker["unified"]["per_worker_ttft_p90_s"].keys())
x = range(len(workers))
width = 0.27
fig, ax = plt.subplots(figsize=(11, 5))
n = len(pols)
width = 0.85 / n
fig, ax = plt.subplots(figsize=(12, 5))
for i, p in enumerate(pols):
d = per_worker[p]["per_worker_ttft_p90_s"]
vals = [d[w] for w in workers]
offset = (i - 1) * width
offset = (i - (n - 1) / 2) * width
label = p.replace("unified_", "") if p != "unified" else "plain"
ax.bar([j + offset for j in x], vals, width,
label=f"{p} (hotspot={per_worker[p]['hotspot_index_ttft_p90']:.2f})",
label=f"{label} (hotspot={per_worker[p]['hotspot_index_ttft_p90']:.2f})",
color=POLICY_COLORS[p], edgecolor="black", linewidth=0.4)
short = [w.replace("http://127.0.0.1:", ":") for w in workers]
ax.set_xticks(list(x))
ax.set_xticklabels(short, rotation=0, fontsize=9)
ax.set_ylabel("worker TTFT p90 (s)")
ax.set_title(
"Per-worker TTFT p90 distribution. kv_both alone makes the hot worker hotter\n"
"(unified→kv_both: 37.7s→43.5s peak); v2's 5 PD-sep triggers nudge it back."
"Per-worker TTFT p90 distribution across substrates. Mooncake (kv_both)\n"
"amplifies the hot worker (hotspot 4.36); NIXL keeps it close to plain (3.67)."
)
ax.legend(loc="upper left", fontsize=9)
ax.grid(alpha=0.3, axis="y")
@@ -232,12 +236,64 @@ def fig_three_way_hotspot():
plt.close(fig)
def fig_connector_substrate_attribution():
"""Decomposes overhead into v1-framework cost (shared by all connectors,
proxied by NIXL since it's the leanest) and Mooncake-specific cost."""
comp = _load("b3_policy_comparison.json")
by = {r["policy"]: r for r in comp["rows"]}
metrics = [
("TTFT p90 (s)", "ttft_p90_s", False),
("TPOT p90 (ms)", "tpot_p90_s", True),
("E2E p90 (s)", "e2e_p90_s", False),
("hotspot index", "hotspot_index_ttft_p90", False),
]
fig, axes = plt.subplots(1, 4, figsize=(15, 4))
for ax, (label, key, scale_ms) in zip(axes, metrics):
plain = by["unified"][key] * (1000 if scale_ms else 1)
nixl = by["unified_nixl_both"][key] * (1000 if scale_ms else 1)
moon = by["unified_kv_both"][key] * (1000 if scale_ms else 1)
v2 = by["unified_v2"][key] * (1000 if scale_ms else 1)
framework_cost = nixl - plain # what NIXL adds = v1 framework cost
mooncake_extra = moon - nixl # extra on top from Mooncake
v2_branch_extra = v2 - moon # extra from PD-sep branch (Mooncake + 5 events)
bottom = 0
ax.bar(["overhead"], [plain], color="#cccccc",
edgecolor="black", linewidth=0.4,
label=f"plain unified ({plain:.2f})")
bottom += plain
ax.bar(["overhead"], [framework_cost], bottom=[bottom],
color="#1f77b4", edgecolor="black", linewidth=0.4,
label=f"v1 framework (+{framework_cost:.2f})")
bottom += framework_cost
ax.bar(["overhead"], [mooncake_extra], bottom=[bottom],
color="#9467bd", edgecolor="black", linewidth=0.4,
label=f"Mooncake extra (+{mooncake_extra:.2f})")
bottom += mooncake_extra
ax.bar(["overhead"], [v2_branch_extra], bottom=[bottom],
color="#d62728", edgecolor="black", linewidth=0.4,
label=f"v2 PD-sep branch ({v2_branch_extra:+.2f})")
ax.set_title(label)
ax.legend(fontsize=8, loc="upper right")
ax.grid(alpha=0.3, axis="y")
ax.tick_params(axis="x", labelbottom=False)
fig.suptitle(
"Attribution: plain unified vs NIXL substrate vs Mooncake substrate vs v2.\n"
"Blue: cost shared by any v1 connector. Purple: cost specific to Mooncake."
)
fig.tight_layout()
fig.savefig(OUT / "fig_connector_substrate_attribution.png", dpi=120)
plt.close(fig)
def main():
fig_kv_both_overhead()
fig_v2_trigger_funnel()
fig_v2_predicted_vs_actual()
fig_three_way_hotspot()
print(f"wrote 4 figures to {OUT}")
fig_connector_substrate_attribution()
print(f"wrote 5 figures to {OUT}")
if __name__ == "__main__":

2
pyproject.toml
View File

@@ -6,6 +6,8 @@ requires-python = ">=3.10"
dependencies = [
"httpx>=0.27",
"numpy>=1.24",
"fastapi>=0.110",
"uvicorn>=0.29",
]
[project.optional-dependencies]

5
scripts/analyze_breakdown.py
View File

@@ -12,9 +12,11 @@ else:
print("Total records: %d" % len(data))
results = []
skipped = 0
for d in data:
keys = ["t_proxy_recv", "t_prefill_sent", "t_prefill_done", "t_decode_sent", "t_first_token"]
if not all(k in d for k in keys):
skipped += 1
continue
results.append({
"input": d["input_length"],
@@ -26,6 +28,9 @@ for d in data:
results.sort(key=lambda x: x["input"])
print("Complete breakdown: %d" % len(results))
if skipped:
print("WARNING: %d records skipped (missing breakdown timestamps)" % skipped,
file=sys.stderr)
if not results:
print("No complete records yet")

12
scripts/b3_analyze.sh
View File

@@ -9,11 +9,19 @@
set -euo pipefail
ROOT="${ROOT:-/home/admin/cpfs/wjh/agentic-kv}"
ROOT="${ROOT:-$(cd "$(dirname "$0")/.." && pwd)}"
VENV="$ROOT/.venv/bin"
SWEEP_DIR="${1:?usage: $0 <sweep_dir>}"
WORKER_MAP="http://127.0.0.1:8000=engine_0,http://127.0.0.1:8001=engine_1,http://127.0.0.1:8002=engine_2,http://127.0.0.1:8003=engine_3,http://127.0.0.1:8004=engine_4,http://127.0.0.1:8005=engine_5,http://127.0.0.1:8006=engine_6,http://127.0.0.1:8007=engine_7"
BASE_PORT="${BASE_PORT:-8000}"
N_INSTANCES="${N_INSTANCES:-8}"
# Build WORKER_MAP dynamically from BASE_PORT and N_INSTANCES.
_worker_map_parts=()
for ((i=0; i<N_INSTANCES; i++)); do
_worker_map_parts+=("http://127.0.0.1:$((BASE_PORT + i))=engine_$i")
done
WORKER_MAP=$(IFS=,; echo "${_worker_map_parts[*]}")
_has_engine_data() {
# Return 0 (true) if $1/*.jsonl contains any non-empty file.

48
scripts/b3_isolated_policy.sh
View File

@@ -11,9 +11,9 @@
set -euo pipefail
ROOT="${ROOT:-/home/admin/cpfs/wjh/agentic-kv}"
ROOT="${ROOT:-$(cd "$(dirname "$0")/.." && pwd)}"
VENV="$ROOT/.venv/bin"
MODEL="${MODEL:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
MODEL="${MODEL:-$ROOT/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
PROXY_PORT="${PROXY_PORT:-9300}"
BASE_PORT="${BASE_PORT:-8000}"
GPU_INDICES="${GPU_INDICES:-0 1 2 3 4 5 6 7}"
@@ -31,8 +31,15 @@ TRACE="${2:?usage: $0 <policy> <trace> <rundir>}"
RUNDIR="${3:?usage: $0 <policy> <trace> <rundir>}"
# Auto-enable kv_both when the policy requires it.
# KV_CONNECTOR (Mooncake|Nixl) selects the underlying connector when KV_BOTH=1.
KV_CONNECTOR="${KV_CONNECTOR:-Mooncake}"
if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_kv_both" ]; then
ENABLE_KV_BOTH=1
KV_CONNECTOR="Mooncake"
fi
if [ "$POLICY" = "unified_nixl_both" ]; then
ENABLE_KV_BOTH=1
KV_CONNECTOR="Nixl"
fi
mkdir -p "$RUNDIR/engine_state" "$RUNDIR/logs"
@@ -49,17 +56,13 @@ trap cleanup EXIT
# Hard reset first
cleanup
echo "[isolated] launching $N_INSTANCES vLLM on GPUs $GPU_INDICES ENABLE_KV_BOTH=$ENABLE_KV_BOTH ..."
echo "[isolated] launching $N_INSTANCES vLLM on GPUs $GPU_INDICES ENABLE_KV_BOTH=$ENABLE_KV_BOTH KV_CONNECTOR=$KV_CONNECTOR ..."
i=0
kv_both_extra=""
if [ "$ENABLE_KV_BOTH" = "1" ]; then
kv_both_extra="--kv-transfer-config {\"kv_connector\":\"MooncakeConnector\",\"kv_role\":\"kv_both\"}"
fi
for gpu in $GPU_INDICES; do
port=$((BASE_PORT + i))
master=$((29500 + i))
bp=$((BOOTSTRAP_BASE_PORT + i))
if [ "$ENABLE_KV_BOTH" = "1" ]; then
if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Mooncake" ]; then
PYTHONHASHSEED=42 \
VLLM_MOONCAKE_BOOTSTRAP_PORT=$bp \
AGENTIC_STEP_LOG_PATH="$RUNDIR/engine_state/engine_${i}.jsonl" \
@@ -75,6 +78,27 @@ for gpu in $GPU_INDICES; do
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \
$EXTRA_VLLM_ARGS \
> "$RUNDIR/logs/vllm_inst_${i}_gpu${gpu}.log" 2>&1 &
elif [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Nixl" ]; then
# NixlConnector's handshake listener binds to a fixed default
# port 5600 unless VLLM_NIXL_SIDE_CHANNEL_PORT is overridden.
# Multiple instances on the same host MUST use distinct ports
# or only one will start; the rest hit
# `zmq.error.ZMQError: Address already in use`.
nixl_port=$((5600 + i))
VLLM_NIXL_SIDE_CHANNEL_PORT=$nixl_port \
AGENTIC_STEP_LOG_PATH="$RUNDIR/engine_state/engine_${i}.jsonl" \
AGENTIC_WORKER_ID="engine_${i}" \
CUDA_VISIBLE_DEVICES=$gpu \
MASTER_PORT=$master \
nohup "$VENV/vllm" serve "$MODEL" \
--host 0.0.0.0 --port "$port" \
--tensor-parallel-size 1 \
--trust-remote-code --enable-prefix-caching \
--dtype auto --gpu-memory-utilization 0.9 \
--max-model-len 200000 \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' \
$EXTRA_VLLM_ARGS \
> "$RUNDIR/logs/vllm_inst_${i}_gpu${gpu}.log" 2>&1 &
else
AGENTIC_STEP_LOG_PATH="$RUNDIR/engine_state/engine_${i}.jsonl" \
AGENTIC_WORKER_ID="engine_${i}" \
@@ -95,13 +119,17 @@ for gpu in $GPU_INDICES; do
done
echo "[isolated] waiting for vLLM health ..."
# NIXL init takes ~100-150s per instance even with concurrent launches;
# Mooncake is closer to ~30-60s. Use a generous 360s timeout to cover
# both (90s -> 360s vs the previous 180s).
HEALTH_MAX_TRIES=180
for i in $(seq 0 $((N_INSTANCES - 1))); do
port=$((BASE_PORT + i))
tries=0
while ! curl -sf "http://127.0.0.1:$port/health" >/dev/null 2>&1; do
tries=$((tries + 1))
if [ $tries -gt 90 ]; then
echo "[isolated] FATAL: inst_$i not healthy after 180s"
if [ $tries -gt $HEALTH_MAX_TRIES ]; then
echo "[isolated] FATAL: inst_$i not healthy after $((HEALTH_MAX_TRIES * 2))s"
exit 1
fi
sleep 2

4
scripts/b3_sweep.sh
View File

@@ -14,9 +14,9 @@
set -euo pipefail
ROOT="${ROOT:-/home/admin/cpfs/wjh/agentic-kv}"
ROOT="${ROOT:-$(cd "$(dirname "$0")/.." && pwd)}"
VENV="$ROOT/.venv/bin"
MODEL="${MODEL:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
MODEL="${MODEL:-$ROOT/models/Qwen/Qwen3-Coder-30B-A3B-Instruct}"
TRACE="${TRACE:-$ROOT/traces/w600_r0.0015_st30.jsonl}"
OUTDIR="${OUTDIR:-$ROOT/outputs/b3_sweep_$(date +%Y%m%d_%H%M%S)}"
PROXY_PORT="${PROXY_PORT:-9300}"

32
scripts/cache_aware_proxy.py
View File

@@ -762,6 +762,9 @@ async def lifespan(app: FastAPI):
# Bootstrap combined instances for offload (need engine_ids for KV transfer)
policy = getattr(global_args, 'policy', 'linear')
# Mooncake-based modes still need bootstrap discovery; NIXL uses
# its own UCX side-channel and doesn't go through our proxy
# bootstrap path (and unified_nixl_both never PD-seps anyway).
needs_bootstrap = (
global_args.offload
or policy in ("unified_v2", "unified_kv_both")
@@ -927,11 +930,16 @@ async def _handle_combined(api, req_data, token_ids, input_length, session_id, h
chosen, best_idx = pick_instance_sticky(
combined_instances, token_ids, session_id, input_length,
session_affinity_combined)
elif policy == "unified" or policy == "unified_kv_both":
elif policy in ("unified", "unified_kv_both", "unified_nixl_both"):
# unified_kv_both: same picker as `unified`, but the vLLMs are
# launched in kv_role=kv_both. Use this as an isolation control
# for `unified_v2` so the v2-vs-v1 gap reflects only the PD-sep
# branch, not the kv_both always-on overhead.
# launched in kv_role=kv_both with MooncakeConnector. Use this
# as an isolation control for `unified_v2` so the v2-vs-v1 gap
# reflects only the PD-sep branch, not the kv_both always-on
# overhead.
# unified_nixl_both: identical to unified_kv_both but with
# NixlConnector at the vLLM layer. Used to attribute the
# kv_both overhead to either Mooncake-specific code or a
# generic v1-connector cost.
chosen, best_idx, decision = pick_instance_unified_hybrid(
combined_instances, token_ids, session_id, input_length,
session_affinity_combined)
@@ -1168,6 +1176,7 @@ async def _handle_pd_sep(api, req_data, request_id, token_ids, input_length,
p_headers = {**headers, "X-data-parallel-rank": "0"}
p_inst.ongoing_tokens += input_length
p_inst.num_requests += 1
breakdown["t_prefill_sent"] = _time.monotonic()
breakdown["t_prefill_sent_unix"] = _time.time()
@@ -1186,9 +1195,11 @@ async def _handle_pd_sep(api, req_data, request_id, token_ids, input_length,
raise HTTPException(status_code=502, detail=f"Prefill failed: {e}")
finally:
p_inst.ongoing_tokens -= input_length
p_inst.num_requests -= 1
# Send decode
d_inst.ongoing_tokens += input_length
d_inst.num_requests += 1
parsed = urllib.parse.urlparse(str(p_inst.client.base_url))
bootstrap_addr = f"http://{parsed.hostname}:{p_inst.bootstrap_port}"
@@ -1224,9 +1235,10 @@ async def _handle_pd_sep(api, req_data, request_id, token_ids, input_length,
breakdown["t_done"] = _time.monotonic()
breakdown["t_done_unix"] = _time.time()
d_inst.ongoing_tokens -= input_length
d_inst.num_requests -= 1
_breakdown_log.append(breakdown)
return StreamingResponse(generate(), media_type="application/json")
return StreamingResponse(generate(), media_type="text/event-stream")
@app.get("/breakdown")
@@ -1291,13 +1303,17 @@ def parse_args():
help="Comma-separated bootstrap ports for combined instances (for offload mode)")
p.add_argument("--policy", type=str, default="linear",
choices=["linear", "lmetric", "load_only", "sticky",
"unified", "unified_kv_both", "unified_v2"],
"unified", "unified_kv_both",
"unified_nixl_both", "unified_v2"],
help="Routing policy: linear (cache-aware), lmetric (P_tokens × BS), "
"load_only (B3 control: pure min-num_requests), "
"sticky (B3 control: hard session affinity), "
"unified (hybrid affinity + LMetric fallback), "
"unified_kv_both (unified on kv_both vLLMs; isolation "
"control for unified_v2; PD-sep never triggers), "
"unified_kv_both (unified picker on kv_both Mooncake "
"vLLMs; isolation control for unified_v2), "
"unified_nixl_both (same as unified_kv_both but using "
"NixlConnector instead of MooncakeConnector; isolates "
"connector implementation from policy effect), "
"or unified_v2 (unified + selective per-request PD-sep "
"via Mooncake; requires --bootstrap-ports and "
"kv_role=kv_both vLLM launch)")

11
scripts/deploy_vllm_patches.sh
View File

@@ -29,11 +29,18 @@ if [ "$HOST" = "local" ]; then
done
else
# Find site-packages on remote
VENV_SITE=$(ssh "$HOST" "~/agentic-kv/.venv/bin/python -c \"import site; print(site.getsitepackages()[0])\"")
VENV_SITE=$(ssh "$HOST" "~/agentic-kv/.venv/bin/python -c \"import site; print(site.getsitepackages()[0])\"") || {
echo "ERROR: failed to resolve site-packages on $HOST" >&2; exit 1;
}
if [ -z "$VENV_SITE" ]; then
echo "ERROR: empty site-packages path from $HOST" >&2; exit 1;
fi
DST="$VENV_SITE/vllm"
echo "Deploying to $HOST:$DST"
for f in "${PATCHED_FILES[@]}"; do
scp "$VLLM_SRC/$f" "$HOST:$DST/$f"
scp "$VLLM_SRC/$f" "$HOST:$DST/$f" || {
echo "ERROR: failed to copy $f to $HOST" >&2; exit 1;
}
done
fi

2
scripts/launch_elastic_p2p.sh
View File

@@ -27,7 +27,7 @@ PROXY_PORT=9090
HEAVY_THRESHOLD="${HEAVY_THRESHOLD:-20000}"
MAX_OFFLOAD="${MAX_OFFLOAD:-4}"
trap 'echo "Cleaning up..."; kill $(jobs -p) 2>/dev/null; wait 2>/dev/null' EXIT INT TERM
trap 'echo "Cleaning up..."; kill $(jobs -p) 2>/dev/null; pkill -9 -f "vllm serve" 2>/dev/null; pkill -9 -f "EngineCore" 2>/dev/null; pkill -9 -f cache_aware_proxy 2>/dev/null; wait 2>/dev/null' EXIT INT TERM
echo "=== Elastic P2P Offload (${N_INSTANCES}× TP=1 kv_both) ==="
echo " Model: $MODEL"