agentic-kvc/analysis/characterization/elastic_migration_v2/README.md

# Elastic Migration v2: Selective PD-Separation via Mooncake

Date: 2026-05-26
Trace: `traces/w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions, 53.3 M tokens)
Model: Qwen3-Coder-30B-A3B-Instruct, 8 × TP1 on H20

## TL;DR

This section explores whether the **B2-confirmed same-worker
prefill–decode interference** can be relieved by selectively
migrating prefill to a different worker for the requests where the
interference cost would dominate the transfer cost. We implement
two flavors of the routing policy (strict gates, then relaxed
gates) and **two isolation controls** that use the unified picker
but launch vLLMs in `kv_role=kv_both` so the connector substrate
is on but never PD-seps:

- `unified_kv_both`: with **MooncakeConnector**
- `unified_nixl_both`: with **NixlConnector** (NVIDIA's official
  v1 connector; isolates connector implementation from policy)

Four findings:

1. **`kv_role=kv_both` imposes a substantial always-on tax even
   when no PD-sep ever fires**: with Mooncake it's TTFT p90 +45%,
   TPOT p90 +25%, hotspot +19%; with NIXL it's TTFT p90 +38%,
   TPOT p90 +16%, hotspot +0.2%.
2. **About half of the substrate cost is generic v1-connector
   framework overhead** (proxied by NIXL since it's the leanest
   implementation): KV buffer GPU memory cut from the model's
   working budget, `SchedulerOutput.kv_connector_metadata`
   round-trip, and altered `kv_cache_manager` block-lifecycle
   semantics. **NIXL is meaningfully better than Mooncake** but
   still imposes a 16-38% tax vs no connector.
3. **PD-sep almost never triggers on a real agentic workload**:
   0.16% with strict gates, 0.41% with relaxed gates. Agentic
   workloads have 93% intra-session reuse, so most requests land
   on workers that already hold cache — the uncached tail is too
   small to be worth migrating.
4. **When PD-sep does fire, the cost model is wrong by ~10–20×**:
   the calibrated `0.3s + bytes / 2.7 GB/s` predicts 1–2 s migrate
   cost; observed TTFT on triggered requests is 12–45 s.

The net latency of `unified_v2` is **not better than plain
`unified`** under either Mooncake or NIXL substrate. Improving
agentic PD-sep requires (a) using the leaner connector (NIXL >
Mooncake by 5-19 pp across metrics), and (b) fixing the underlying
transfer mechanism (E2 patches 6.1 lazy block reservation and 6.3
layerwise pipelining), not just the routing decision.

## Substrate

We compare four policies on identical traces:

| policy | picker | vLLM launch mode | what's it for |
|---|---|---|---|
| `unified` | hybrid affinity + LMetric | plain (no connector) | the headline baseline |
| `unified_kv_both` | same as `unified` | `MooncakeConnector` + `kv_both` | substrate control: Mooncake cost without PD-sep |
| `unified_nixl_both` | same as `unified` | `NixlConnector` + `kv_both` | substrate control: NIXL cost without PD-sep, attributes overhead to "framework vs Mooncake" |
| `unified_v2` | unified + selective PD-sep | `MooncakeConnector` + `kv_both` + bootstrap | the actual experiment |

All four use the same trace, the same 8-instance topology, the same
shadow-drift–corrected proxy (`scripts/cache_aware_proxy.py` post-fix
`95c8ef8`). Plain `unified` was rerun on the patched proxy
(`b3_sweep_20260525_095043/unified`) under the same conditions.

NIXL required two launch fixes beyond Mooncake:
- `VLLM_NIXL_SIDE_CHANNEL_PORT` must be unique per instance
  (default 5600 → 5600..5607); otherwise instances 2..8 silently
  hang in `zmq.error.ZMQError: Address already in use`.
- Health-check timeout had to be raised from 180 s to 360 s
  because NIXL initialization (UCX agent + memory registration)
  takes ~100-150 s per instance under 8-way concurrent launch.

## Result 1 — kv_both is expensive by itself, and only partly Mooncake's fault

![](figures/fig_kv_both_overhead.png)

Switching the vLLM launch from plain to `kv_role=kv_both` without
ever triggering PD-sep imposes a substrate tax. We compare the two
connectors available in vendored vLLM:

| metric | plain `unified` | `unified_nixl_both` | `unified_kv_both` (Mooncake) |
|---|---:|---:|---:|
| TTFT p50 | 0.50 s | 0.51 s (+1%) | 0.50 s (+0%) |
| **TTFT p90** | 7.35 s | **10.13 s (+38%)** | **10.67 s (+45%)** |
| TTFT p99 | 42.34 s | 44.58 s (+5%) | 45.19 s (+7%) |
| TPOT p90 | 17.1 ms | **19.8 ms (+16%)** | **21.3 ms (+25%)** |
| E2E p90 | 18.03 s | **21.18 s (+17%)** | **22.89 s (+27%)** |
| APC | 79.4% | 79.1% (−0.3 pp) | 78.3% (−1.1 pp) |
| **hotspot index** | 3.667 | **3.674 (+0.2%)** | **4.363 (+19%)** |
| interference index | n/a | 5.58 | 8.57 |

![](figures/fig_connector_substrate_attribution.png)

Reading the table from left to right gives a clean attribution:

- **NIXL−plain** = the **v1-connector framework's irreducible cost**
  (TTFT p90 +38%, TPOT p90 +16%, E2E p90 +17%). This is the cost
  *any* v1 KV connector imposes:
  - the 1 GB `kv_buffer_size` carved from `gpu-memory-utilization`,
    reducing the KV cache budget;
  - per-step `SchedulerOutput.kv_connector_metadata` serialization
    and round-trip through the connector worker;
  - altered block-lifecycle semantics in `kv_cache_manager`
    (`delay_free_blocks=True` is the default once any connector is
    loaded, slowing LRU eviction).
- **Mooncake−NIXL** = the **Mooncake-implementation-specific extra**
  (TTFT p90 +7 pp, TPOT p90 +9 pp, E2E p90 +10 pp, hotspot +19 pp).
  This is the cost Mooncake's design choices add on top of the
  generic framework:
  - per-scheduler-step `set(self._block_pool.cache.keys())` diff
    against `_known_hash_keys` (`mooncake_connector.py:432-456`)
    walks O(|cache|) on every step on every engine, costing ~4 M
    set operations per second on a 200 k-block cache;
  - the hash sync runs even when no `direct_read` consumer is
    present, so the cost is paid unconditionally;
  - block-lifecycle is further constrained because Mooncake
    requires `delay_free` until the explicit `finished_sending`
    arrives, vs NIXL which can release blocks earlier.

The **most striking gap is hotspot**: Mooncake's per-step hash
sync runs on the scheduler's GIL and disrupts the timeliness of
routing decisions, amplifying load imbalance by 19%. NIXL has no
equivalent global-state maintenance and preserves the plain-unified
hotspot to within 0.2%.

Practical implication: **you don't enable any v1 KV connector for
free**, but if you have to enable one, NIXL is meaningfully cheaper
than Mooncake. Even NIXL's 38% TTFT p90 tax is large enough that
PD-sep needs to recover it on a non-trivial fraction of requests
before being worth it.

## Result 2 — PD-sep rarely fires on a real agentic trace

![](figures/fig_v2_trigger_funnel.png)

We log every routing decision's `v2_reason` (why we did or did not
PD-sep). Two runs with different gate thresholds:

| fall-through bucket | v2.0 strict | v2.1 relaxed | what it means |
|---|---:|---:|---|
| `new_local < threshold` | 1077 (88.7%) | 924 (76.1%) | uncached tail too small to justify transfer |
| `chosen_no_active_decode` | 115 (9.5%) | 229 (18.9%) | no decode on chosen to protect |
| `src_cache_below_threshold` | 14 (1.2%) | 36 (3.0%) | no alt instance holds enough cache |
| `src_not_meaningfully_more_cache` | 6 (0.5%) | 16 (1.3%) | alt instance doesn't help vs chosen |
| `cost_benefit not enough margin` | 0 | 4 (0.3%) | model says transfer cost + interference on src ≥ local interference |
| **PD-sep TRIGGERED** | **2 (0.16%)** | **5 (0.41%)** | passed all gates and cost-benefit favored migrate |

The dominant filter is `new_local < threshold`. Even with the
threshold dropped from 16 k to 8 k tokens, three out of four requests
have less than 8 k uncached tokens at the chosen worker. This is
structural: with intra-session reuse measured at 93% on the same
trace (window_1_results.md), most turns hit prefix cache on the
session's previous worker.

The second filter, `chosen_no_active_decode`, kills another fifth.
This is a snapshot-time phenomenon: at the moment the picker runs,
the chosen worker often has its previous request still in prefill,
not yet decoding. The gate's intent ("don't migrate if no decode is
being hurt by the prefill we're routing") is correct, but it ends up
suppressing PD-sep for a real situation where decode is *about to*
start.

Even after these two filters, the cost-benefit step itself rejects
nearly half of remaining candidates (4 out of 9 in relaxed). So the
final trigger rate of 0.41% is a structural property, not a
parameter-tuning problem.

## Result 3 — when PD-sep fires, the cost model is wrong by 10–20×

![](figures/fig_v2_predicted_vs_actual.png)

The 5 PD-sep-triggered requests in v2.1 relaxed:

| input | new_local | new_src | src→dst | cost_local | cost_migrate (model) | actual TTFT | actual E2E |
|---:|---:|---:|---|---:|---:|---:|---:|
| 21963 | 21963 |  9163 | 6→5 | 4.39 s | 4.17 s |   3.69 s |   8.48 s |
|  8706 |  8706 |  2050 | 5→7 | 1.09 s | 0.73 s |  12.48 s |  14.31 s |
| 13616 | 13616 |  2352 | 4→0 | 1.70 s | 1.03 s |  18.33 s |  19.50 s |
| 49483 | 49483 |   843 | 3→4 | 11.75 s | 2.16 s | **45.13 s** | **53.55 s** |
| 19806 | 19806 |   350 | 3→6 | 3.96 s | 1.06 s |  20.06 s |  31.98 s |

The cost model predicts the migrate path will take 0.7–2.2 s; the
actual TTFT on these requests is 12–45 s. The model's `0.3 s +
bytes / 2.7 GB/s` calibration captures pure RDMA bandwidth in
isolation but misses everything else that happens on the
`decode_sent → first_token` clock: D-side scheduler step latency,
block reservation before KV arrives (so D's cache pressure
increases for the entire wait), the per-layer scatter of
`batch_transfer_sync_write`, and the next-step scheduler promotion
after `finished_recving`. The E2 audit measured this end-to-end at
p50 = 1.1 s and **p90 = 6.7 s** on production runs; the v2.1
triggered requests landed in the p99 tail of that distribution
because their dst was already loaded.

The first-token clock for the 49 k request is **21× the model's
prediction**. This is not a small mis-tuning — it's a structurally
different model.

## Result 4 — four-way comparison

![](figures/fig_three_way_hotspot.png)

The full table:

| metric | unified (plain) | unified_nixl_both | unified_kv_both (Mooncake) | unified_v2 (relaxed) |
|---|---:|---:|---:|---:|
| n_ok | 1214 | 1214 | 1214 | 1214 |
| TTFT p50 | 0.50 s | 0.51 s | 0.50 s | 0.49 s |
| TTFT p90 | 7.35 s | 10.13 s | 10.67 s | 10.98 s |
| TTFT p99 | 42.34 s | 44.58 s | 45.19 s | 49.45 s |
| TPOT p90 | 17.1 ms | 19.8 ms | 21.3 ms | 18.4 ms |
| E2E p90 | 18.03 s | 21.18 s | 22.89 s | 22.53 s |
| APC | 79.4% | 79.1% | 78.3% | 77.6% |
| interference index | n/a | 5.58 | 8.57 | 8.46 |
| hotspot index | 3.667 | 3.674 | 4.363 | 3.910 |
| n_slow | 189 | 192 | 198 | 198 |

### v2 vs the kv_both control (the right comparison)

Compared to the kv_both control — same substrate, no PD-sep — the
5 PD-sep triggers in v2:

- **slightly improve TPOT p90 (−14%) and hotspot (−10%)**
- **slightly worsen TTFT p90 (+3%) and TTFT p99 (+9%)**, because the
  triggered requests themselves take ~20× the predicted transfer
  time

The net effect against the kv_both control is in the noise. The
hotspot improvement is within the run-to-run stochastic range we saw
earlier (v2 strict run scored 2.733 hotspot under the same
substrate; v2 relaxed scored 3.910).

### v2 vs plain unified (the headline question)

`unified_v2` is **27% slower on E2E p90** and **49% slower on TTFT
p90** than plain `unified`. The 45 pp of TTFT p90 inflation is from
kv_both substrate, not the routing decision; nothing PD-sep does can
recover this in our current Mooncake implementation.

## Why v2's PD-sep is fundamentally choked

There are three independent structural problems, each by itself
enough to make v2 not win:

1. **The kv_both substrate is the wrong default**. It pays a 45%
   TTFT p90 tax on every request. To make selective PD-sep beat
   plain `unified`, the saved interference per triggered request
   times the trigger rate must exceed 45% × average TTFT, on
   average. With 0.41% trigger rate, even saving 100% of TTFT per
   triggered request would only save ~0.4%, which can't recover 45%.

2. **Agentic intra-session reuse leaves no headroom for migration**.
   Most turns hit cache on the worker that handled the previous
   turn. Migrating prefill to a *different* worker is the *exact*
   thing intra-session affinity tries to avoid: it forces the new
   worker to pay for the cached prefix transfer instead of just
   reusing what's already on the affinity worker. This is a
   structural mismatch between PD-sep semantics ("send big prefills
   to a less-busy worker") and agentic workloads ("keep sessions
   sticky to wherever the cache is").

3. **The Mooncake mechanism is 10–20× slower than the cost model
   predicts**, primarily due to D-side pre-allocation of KV blocks
   and the absence of layerwise pipelining (E2 audit §6.1 / §6.3).
   The cost model can be re-calibrated, but doing so would push the
   gate even tighter, dropping the already-tiny trigger rate to
   nearly zero.

The three are stacked: even if any two were fixed, the remaining
one would still make PD-sep a net loss on this trace.

## What this section claims for the paper

1. **Same-worker prefill–decode interference is a real mechanism**
   (B2 microbench), but **agentic workloads rarely expose it**: the
   typical request has high cache hit and small uncached tail, so
   the interference cost is bounded.
2. **Routing-only solutions (unified) already capture 79% of the
   intra-session APC ceiling and recover the latency** by avoiding
   the heavy-tail sessions through the affinity gate. The remaining
   23 pp gap to the ceiling is from APC LRU eviction under capacity
   pressure, not from prefill–decode interference.
3. **Per-request PD-sep via Mooncake on agentic workloads is not a
   net win** in our measurements, even with a carefully-gated cost
   model. The combined effect of kv_both substrate overhead, low
   trigger rate, and mechanism-vs-model gap is uniformly negative.
4. **A productive direction is mechanism-level**: fix the Mooncake
   D-side block reservation (E2 §6.1), implement layerwise transfer
   pipelining (E2 §6.3), and re-measure. Only if these patches drop
   the substrate tax to <10% and the realized transfer to ≤2 s p90
   does PD-sep become competitive with routing on agentic traces.

## What v2 still validates

- **The cost model's *qualitative* shape is correct**: when it says
  "migrate", that's a request where local interference *would have*
  been ≥ 4 s and src has ≥ 80% prefix cache. The model picks the
  right candidate requests.
- **The gate logic catches the right exclusions**: 88% by uncached
  tail size, 19% by no-decode-to-protect, the rest by missing
  source cache. Each is a structurally correct reason.
- **The proxy shadow-drift fix is necessary infrastructure** for
  any long-running routing experiment. We observed 3 phantom
  corrections per ~50-minute run.

## Files

- `data/b3_policy_comparison.json` — the four policies' headline
  metrics from the same B3 sweep root.
- `data/breakdown_<policy>.json` — per-request proxy breakdown
  including v2 gate fields and triggered-event metadata.
- `data/per_worker_<policy>.json` — per-worker TTFT/latency p90s
  used in the hotspot figure.
- `figures/*.png` — the four section figures referenced above.
- `render_figures.py` — regenerates the figures from data/.

## Cross-references

- `analysis/characterization/window_1_results.md` — B2 microbench
  (same-worker interference causal proof) and B3 baseline 5-policy
  sweep
- `analysis/characterization/agentic_dispatch_coupling.md` — why
  the saturated-replay setup matches agentic production
- `analysis/characterization/b3_policies_pseudocode.md` — pickers
  for the five baseline policies; `unified_v2` extends `unified`
- E1 / E2 subagent reports (commit `4b833d3` message and the
  conversation log) — full mechanism audit that informed v2's design