# Elastic Migration v2: Selective PD-Separation via Mooncake

Date: 2026-05-26
Trace: `traces/w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions, 53.3 M tokens)
Model: Qwen3-Coder-30B-A3B-Instruct, 8 × TP1 on H20

## TL;DR

This section explores whether the **B2-confirmed same-worker
prefill–decode interference** can be relieved by selectively
migrating prefill to a different worker for the requests where the
interference cost would dominate the transfer cost. We implement two
flavors of the policy (strict gates, then relaxed gates) and a clean
isolation control (`unified_kv_both`: same picker as `unified`, but
the vLLMs are launched in `kv_role=kv_both` so the Mooncake
substrate is on but never triggers).

Three findings:

1. **`kv_role=kv_both` alone imposes a heavy always-on tax**: TTFT
   p90 +45%, TPOT p90 +25%, hotspot index +19% vs plain `unified`,
   with no PD-sep ever firing.
2. **PD-sep almost never triggers on a real agentic workload**:
   0.16% with strict gates, 0.41% with relaxed gates. Agentic
   workloads have 93% intra-session reuse, so most requests land on
   workers that already hold cache — the uncached tail is too small
   to be worth migrating.
3. **When PD-sep does fire, the cost model is wrong by ~10–20×**:
   the calibrated `0.3s + bytes / 2.7 GB/s` predicts 1–2 s migrate
   cost; observed TTFT on triggered requests is 12–45 s. The same
   D-side block-reservation pressure and absence of layerwise
   pipelining that the E2 audit flagged still dominate.

The net latency of `unified_v2` is **not better than plain
`unified`**. Improving agentic PD-sep requires fixing the underlying
Mooncake transfer mechanism (E2 patches 6.1 lazy block reservation
and 6.3 layerwise pipelining), not the routing decision.

## Substrate

We compare three policies on identical traces:

| policy | picker | vLLM launch mode | what's it for |
|---|---|---|---|
| `unified` | hybrid affinity + LMetric | plain (no Mooncake) | the headline baseline |
| `unified_kv_both` | same as `unified` | `kv_role=kv_both` + bootstrap | isolation control: how much does kv_both *alone* cost? |
| `unified_v2` | unified + selective PD-sep | `kv_role=kv_both` + bootstrap | the actual experiment |

All three use the same trace, the same 8-instance topology, the same
shadow-drift–corrected proxy (`scripts/cache_aware_proxy.py` post-fix
`95c8ef8`). Plain `unified` was rerun on the patched proxy
(`b3_sweep_20260525_095043/unified`) under the same conditions.

## Result 1 — kv_both is expensive by itself

![](figures/fig_kv_both_overhead.png)

Switching the vLLM launch from plain to `kv_role=kv_both` without
ever triggering PD-sep already costs:

| metric | plain `unified` | `unified_kv_both` | Δ |
|---|---:|---:|---|
| TTFT p50 | 0.50 s | 0.50 s | +0% |
| TTFT p90 | 7.35 s | 10.67 s | **+45%** |
| TTFT p99 | 42.34 s | 45.19 s | +7% |
| TPOT p90 | 17.1 ms | 21.3 ms | **+25%** |
| E2E p90 | 18.03 s | 22.89 s | **+27%** |
| APC | 79.4% | 78.3% | −1.1 pp |
| hotspot index | 3.667 | **4.363** | **+19%** |

Two contributing factors:

1. **The Mooncake `MooncakeConnector` runs even when no transfer is
   pending.** Every scheduler step it walks `set(cache.keys())`
   against `_known_hash_keys` (E2 audit §6.5) and updates the
   `KVConnectorMetadata`. This is O(|cache|) per step on every
   engine, even when no producer/consumer relationship is active.
2. **Block reservation semantics differ** under kv_both. The
   scheduler treats blocks as candidates for export-to-others, so
   the prefix cache LRU pressure is slightly different (we lose 1
   pp APC).

Practical implication: **you don't enable kv_both for free**. If a
deployment wants the option to do PD-sep selectively, the 45% TTFT
p90 tax applies even on requests that stay local. This needs to
recoverable cost before any selective-PD-sep policy is worth
shipping.

## Result 2 — PD-sep rarely fires on a real agentic trace

![](figures/fig_v2_trigger_funnel.png)

We log every routing decision's `v2_reason` (why we did or did not
PD-sep). Two runs with different gate thresholds:

| fall-through bucket | v2.0 strict | v2.1 relaxed | what it means |
|---|---:|---:|---|
| `new_local < threshold` | 1077 (88.7%) | 924 (76.1%) | uncached tail too small to justify transfer |
| `chosen_no_active_decode` | 115 (9.5%) | 229 (18.9%) | no decode on chosen to protect |
| `src_cache_below_threshold` | 14 (1.2%) | 36 (3.0%) | no alt instance holds enough cache |
| `src_not_meaningfully_more_cache` | 6 (0.5%) | 16 (1.3%) | alt instance doesn't help vs chosen |
| `cost_benefit not enough margin` | 0 | 4 (0.3%) | model says transfer cost + interference on src ≥ local interference |
| **PD-sep TRIGGERED** | **2 (0.16%)** | **5 (0.41%)** | passed all gates and cost-benefit favored migrate |

The dominant filter is `new_local < threshold`. Even with the
threshold dropped from 16 k to 8 k tokens, three out of four requests
have less than 8 k uncached tokens at the chosen worker. This is
structural: with intra-session reuse measured at 93% on the same
trace (window_1_results.md), most turns hit prefix cache on the
session's previous worker.

The second filter, `chosen_no_active_decode`, kills another fifth.
This is a snapshot-time phenomenon: at the moment the picker runs,
the chosen worker often has its previous request still in prefill,
not yet decoding. The gate's intent ("don't migrate if no decode is
being hurt by the prefill we're routing") is correct, but it ends up
suppressing PD-sep for a real situation where decode is *about to*
start.

Even after these two filters, the cost-benefit step itself rejects
nearly half of remaining candidates (4 out of 9 in relaxed). So the
final trigger rate of 0.41% is a structural property, not a
parameter-tuning problem.

## Result 3 — when PD-sep fires, the cost model is wrong by 10–20×

![](figures/fig_v2_predicted_vs_actual.png)

The 5 PD-sep-triggered requests in v2.1 relaxed:

| input | new_local | new_src | src→dst | cost_local | cost_migrate (model) | actual TTFT | actual E2E |
|---:|---:|---:|---|---:|---:|---:|---:|
| 21963 | 21963 |  9163 | 6→5 | 4.39 s | 4.17 s |   3.69 s |   8.48 s |
|  8706 |  8706 |  2050 | 5→7 | 1.09 s | 0.73 s |  12.48 s |  14.31 s |
| 13616 | 13616 |  2352 | 4→0 | 1.70 s | 1.03 s |  18.33 s |  19.50 s |
| 49483 | 49483 |   843 | 3→4 | 11.75 s | 2.16 s | **45.13 s** | **53.55 s** |
| 19806 | 19806 |   350 | 3→6 | 3.96 s | 1.06 s |  20.06 s |  31.98 s |

The cost model predicts the migrate path will take 0.7–2.2 s; the
actual TTFT on these requests is 12–45 s. The model's `0.3 s +
bytes / 2.7 GB/s` calibration captures pure RDMA bandwidth in
isolation but misses everything else that happens on the
`decode_sent → first_token` clock: D-side scheduler step latency,
block reservation before KV arrives (so D's cache pressure
increases for the entire wait), the per-layer scatter of
`batch_transfer_sync_write`, and the next-step scheduler promotion
after `finished_recving`. The E2 audit measured this end-to-end at
p50 = 1.1 s and **p90 = 6.7 s** on production runs; the v2.1
triggered requests landed in the p99 tail of that distribution
because their dst was already loaded.

The first-token clock for the 49 k request is **21× the model's
prediction**. This is not a small mis-tuning — it's a structurally
different model.

## Result 4 — three-way comparison

![](figures/fig_three_way_hotspot.png)

The full table:

| metric | unified (plain) | unified_kv_both | unified_v2 (relaxed) |
|---|---:|---:|---:|
| n_ok | 1214 | 1214 | 1214 |
| TTFT p50 | 0.50 s | 0.50 s | 0.49 s |
| TTFT p90 | 7.35 s | 10.67 s | 10.98 s |
| TTFT p99 | 42.34 s | 45.19 s | 49.45 s |
| TPOT p90 | 17.1 ms | 21.3 ms | 18.4 ms |
| E2E p90 | 18.03 s | 22.89 s | 22.53 s |
| APC | 79.4% | 78.3% | 77.6% |
| interference index | n/a (no engine_state) | 8.57 | 8.46 |
| hotspot index | 3.667 | 4.363 | 3.910 |
| n_slow | 189 | 198 | 198 |

### v2 vs the kv_both control (the right comparison)

Compared to the kv_both control — same substrate, no PD-sep — the
5 PD-sep triggers in v2:

- **slightly improve TPOT p90 (−14%) and hotspot (−10%)**
- **slightly worsen TTFT p90 (+3%) and TTFT p99 (+9%)**, because the
  triggered requests themselves take ~20× the predicted transfer
  time

The net effect against the kv_both control is in the noise. The
hotspot improvement is within the run-to-run stochastic range we saw
earlier (v2 strict run scored 2.733 hotspot under the same
substrate; v2 relaxed scored 3.910).

### v2 vs plain unified (the headline question)

`unified_v2` is **27% slower on E2E p90** and **49% slower on TTFT
p90** than plain `unified`. The 45 pp of TTFT p90 inflation is from
kv_both substrate, not the routing decision; nothing PD-sep does can
recover this in our current Mooncake implementation.

## Why v2's PD-sep is fundamentally choked

There are three independent structural problems, each by itself
enough to make v2 not win:

1. **The kv_both substrate is the wrong default**. It pays a 45%
   TTFT p90 tax on every request. To make selective PD-sep beat
   plain `unified`, the saved interference per triggered request
   times the trigger rate must exceed 45% × average TTFT, on
   average. With 0.41% trigger rate, even saving 100% of TTFT per
   triggered request would only save ~0.4%, which can't recover 45%.

2. **Agentic intra-session reuse leaves no headroom for migration**.
   Most turns hit cache on the worker that handled the previous
   turn. Migrating prefill to a *different* worker is the *exact*
   thing intra-session affinity tries to avoid: it forces the new
   worker to pay for the cached prefix transfer instead of just
   reusing what's already on the affinity worker. This is a
   structural mismatch between PD-sep semantics ("send big prefills
   to a less-busy worker") and agentic workloads ("keep sessions
   sticky to wherever the cache is").

3. **The Mooncake mechanism is 10–20× slower than the cost model
   predicts**, primarily due to D-side pre-allocation of KV blocks
   and the absence of layerwise pipelining (E2 audit §6.1 / §6.3).
   The cost model can be re-calibrated, but doing so would push the
   gate even tighter, dropping the already-tiny trigger rate to
   nearly zero.

The three are stacked: even if any two were fixed, the remaining
one would still make PD-sep a net loss on this trace.

## What this section claims for the paper

1. **Same-worker prefill–decode interference is a real mechanism**
   (B2 microbench), but **agentic workloads rarely expose it**: the
   typical request has high cache hit and small uncached tail, so
   the interference cost is bounded.
2. **Routing-only solutions (unified) already capture 79% of the
   intra-session APC ceiling and recover the latency** by avoiding
   the heavy-tail sessions through the affinity gate. The remaining
   23 pp gap to the ceiling is from APC LRU eviction under capacity
   pressure, not from prefill–decode interference.
3. **Per-request PD-sep via Mooncake on agentic workloads is not a
   net win** in our measurements, even with a carefully-gated cost
   model. The combined effect of kv_both substrate overhead, low
   trigger rate, and mechanism-vs-model gap is uniformly negative.
4. **A productive direction is mechanism-level**: fix the Mooncake
   D-side block reservation (E2 §6.1), implement layerwise transfer
   pipelining (E2 §6.3), and re-measure. Only if these patches drop
   the substrate tax to <10% and the realized transfer to ≤2 s p90
   does PD-sep become competitive with routing on agentic traces.

## What v2 still validates

- **The cost model's *qualitative* shape is correct**: when it says
  "migrate", that's a request where local interference *would have*
  been ≥ 4 s and src has ≥ 80% prefix cache. The model picks the
  right candidate requests.
- **The gate logic catches the right exclusions**: 88% by uncached
  tail size, 19% by no-decode-to-protect, the rest by missing
  source cache. Each is a structurally correct reason.
- **The proxy shadow-drift fix is necessary infrastructure** for
  any long-running routing experiment. We observed 3 phantom
  corrections per ~50-minute run.

## Files

- `data/b3_policy_comparison.json` — the four policies' headline
  metrics from the same B3 sweep root.
- `data/breakdown_<policy>.json` — per-request proxy breakdown
  including v2 gate fields and triggered-event metadata.
- `data/per_worker_<policy>.json` — per-worker TTFT/latency p90s
  used in the hotspot figure.
- `figures/*.png` — the four section figures referenced above.
- `render_figures.py` — regenerates the figures from data/.

## Cross-references

- `analysis/characterization/window_1_results.md` — B2 microbench
  (same-worker interference causal proof) and B3 baseline 5-policy
  sweep
- `analysis/characterization/agentic_dispatch_coupling.md` — why
  the saturated-replay setup matches agentic production
- `analysis/characterization/b3_policies_pseudocode.md` — pickers
  for the five baseline policies; `unified_v2` extends `unified`
- E1 / E2 subagent reports (commit `4b833d3` message and the
  conversation log) — full mechanism audit that informed v2's design