agentic-kvc/analysis/characterization/elastic_migration_v2/README.md

Elastic Migration v2: Selective PD-Separation via Mooncake

Date: 2026-05-26 Trace: traces/w600_r0.0015_st30.jsonl (1214 reqs, 274 sessions, 53.3 M tokens) Model: Qwen3-Coder-30B-A3B-Instruct, 8 × TP1 on H20

TL;DR

This section explores whether the B2-confirmed same-worker prefill–decode interference can be relieved by selectively migrating prefill to a different worker for the requests where the interference cost would dominate the transfer cost. We implement two flavors of the policy (strict gates, then relaxed gates) and a clean isolation control (unified_kv_both: same picker as unified, but the vLLMs are launched in kv_role=kv_both so the Mooncake substrate is on but never triggers).

Three findings:

1. kv_role=kv_both alone imposes a heavy always-on tax: TTFT p90 +45%, TPOT p90 +25%, hotspot index +19% vs plain unified, with no PD-sep ever firing.
2. PD-sep almost never triggers on a real agentic workload: 0.16% with strict gates, 0.41% with relaxed gates. Agentic workloads have 93% intra-session reuse, so most requests land on workers that already hold cache — the uncached tail is too small to be worth migrating.
3. When PD-sep does fire, the cost model is wrong by ~10–20×: the calibrated 0.3s + bytes / 2.7 GB/s predicts 1–2 s migrate cost; observed TTFT on triggered requests is 12–45 s. The same D-side block-reservation pressure and absence of layerwise pipelining that the E2 audit flagged still dominate.

The net latency of unified_v2 is not better than plain unified. Improving agentic PD-sep requires fixing the underlying Mooncake transfer mechanism (E2 patches 6.1 lazy block reservation and 6.3 layerwise pipelining), not the routing decision.

Substrate

We compare three policies on identical traces:

policy	picker	vLLM launch mode	what's it for
unified	hybrid affinity + LMetric	plain (no Mooncake)	the headline baseline
unified_kv_both	same as unified	kv_role=kv_both + bootstrap	isolation control: how much does kv_both alone cost?
unified_v2	unified + selective PD-sep	kv_role=kv_both + bootstrap	the actual experiment

All three use the same trace, the same 8-instance topology, the same shadow-drift–corrected proxy (scripts/cache_aware_proxy.py post-fix 95c8ef8). Plain unified was rerun on the patched proxy (b3_sweep_20260525_095043/unified) under the same conditions.

Result 1 — kv_both is expensive by itself

Switching the vLLM launch from plain to kv_role=kv_both without ever triggering PD-sep already costs:

metric	plain unified	unified_kv_both	Δ
TTFT p50	0.50 s	0.50 s	+0%
TTFT p90	7.35 s	10.67 s	+45%
TTFT p99	42.34 s	45.19 s	+7%
TPOT p90	17.1 ms	21.3 ms	+25%
E2E p90	18.03 s	22.89 s	+27%
APC	79.4%	78.3%	−1.1 pp
hotspot index	3.667	4.363	+19%

Two contributing factors:

1. The Mooncake MooncakeConnector runs even when no transfer is pending. Every scheduler step it walks set(cache.keys()) against _known_hash_keys (E2 audit §6.5) and updates the KVConnectorMetadata. This is O(|cache|) per step on every engine, even when no producer/consumer relationship is active.
2. Block reservation semantics differ under kv_both. The scheduler treats blocks as candidates for export-to-others, so the prefix cache LRU pressure is slightly different (we lose 1 pp APC).

Practical implication: you don't enable kv_both for free. If a deployment wants the option to do PD-sep selectively, the 45% TTFT p90 tax applies even on requests that stay local. This needs to recoverable cost before any selective-PD-sep policy is worth shipping.

Result 2 — PD-sep rarely fires on a real agentic trace

We log every routing decision's v2_reason (why we did or did not PD-sep). Two runs with different gate thresholds:

fall-through bucket	v2.0 strict	v2.1 relaxed	what it means
new_local < threshold	1077 (88.7%)	924 (76.1%)	uncached tail too small to justify transfer
chosen_no_active_decode	115 (9.5%)	229 (18.9%)	no decode on chosen to protect
src_cache_below_threshold	14 (1.2%)	36 (3.0%)	no alt instance holds enough cache
src_not_meaningfully_more_cache	6 (0.5%)	16 (1.3%)	alt instance doesn't help vs chosen
cost_benefit not enough margin	0	4 (0.3%)	model says transfer cost + interference on src ≥ local interference
PD-sep TRIGGERED	2 (0.16%)	5 (0.41%)	passed all gates and cost-benefit favored migrate

The dominant filter is new_local < threshold. Even with the threshold dropped from 16 k to 8 k tokens, three out of four requests have less than 8 k uncached tokens at the chosen worker. This is structural: with intra-session reuse measured at 93% on the same trace (window_1_results.md), most turns hit prefix cache on the session's previous worker.

The second filter, chosen_no_active_decode, kills another fifth. This is a snapshot-time phenomenon: at the moment the picker runs, the chosen worker often has its previous request still in prefill, not yet decoding. The gate's intent ("don't migrate if no decode is being hurt by the prefill we're routing") is correct, but it ends up suppressing PD-sep for a real situation where decode is about to start.

Even after these two filters, the cost-benefit step itself rejects nearly half of remaining candidates (4 out of 9 in relaxed). So the final trigger rate of 0.41% is a structural property, not a parameter-tuning problem.

Result 3 — when PD-sep fires, the cost model is wrong by 10–20×

The 5 PD-sep-triggered requests in v2.1 relaxed:

input	new_local	new_src	src→dst	cost_local	cost_migrate (model)	actual TTFT	actual E2E
21963	21963	9163	6→5	4.39 s	4.17 s	3.69 s	8.48 s
8706	8706	2050	5→7	1.09 s	0.73 s	12.48 s	14.31 s
13616	13616	2352	4→0	1.70 s	1.03 s	18.33 s	19.50 s
49483	49483	843	3→4	11.75 s	2.16 s	45.13 s	53.55 s
19806	19806	350	3→6	3.96 s	1.06 s	20.06 s	31.98 s

The cost model predicts the migrate path will take 0.7–2.2 s; the actual TTFT on these requests is 12–45 s. The model's 0.3 s + bytes / 2.7 GB/s calibration captures pure RDMA bandwidth in isolation but misses everything else that happens on the decode_sent → first_token clock: D-side scheduler step latency, block reservation before KV arrives (so D's cache pressure increases for the entire wait), the per-layer scatter of batch_transfer_sync_write, and the next-step scheduler promotion after finished_recving. The E2 audit measured this end-to-end at p50 = 1.1 s and p90 = 6.7 s on production runs; the v2.1 triggered requests landed in the p99 tail of that distribution because their dst was already loaded.

The first-token clock for the 49 k request is 21× the model's prediction. This is not a small mis-tuning — it's a structurally different model.

Result 4 — three-way comparison

The full table:

metric	unified (plain)	unified_kv_both	unified_v2 (relaxed)
n_ok	1214	1214	1214
TTFT p50	0.50 s	0.50 s	0.49 s
TTFT p90	7.35 s	10.67 s	10.98 s
TTFT p99	42.34 s	45.19 s	49.45 s
TPOT p90	17.1 ms	21.3 ms	18.4 ms
E2E p90	18.03 s	22.89 s	22.53 s
APC	79.4%	78.3%	77.6%
interference index	n/a (no engine_state)	8.57	8.46
hotspot index	3.667	4.363	3.910
n_slow	189	198	198

v2 vs the kv_both control (the right comparison)

Compared to the kv_both control — same substrate, no PD-sep — the 5 PD-sep triggers in v2:

• slightly improve TPOT p90 (−14%) and hotspot (−10%)
• slightly worsen TTFT p90 (+3%) and TTFT p99 (+9%), because the triggered requests themselves take ~20× the predicted transfer time

The net effect against the kv_both control is in the noise. The hotspot improvement is within the run-to-run stochastic range we saw earlier (v2 strict run scored 2.733 hotspot under the same substrate; v2 relaxed scored 3.910).

v2 vs plain unified (the headline question)

unified_v2 is 27% slower on E2E p90 and 49% slower on TTFT p90 than plain unified. The 45 pp of TTFT p90 inflation is from kv_both substrate, not the routing decision; nothing PD-sep does can recover this in our current Mooncake implementation.

Why v2's PD-sep is fundamentally choked

There are three independent structural problems, each by itself enough to make v2 not win:

1. The kv_both substrate is the wrong default. It pays a 45% TTFT p90 tax on every request. To make selective PD-sep beat plain unified, the saved interference per triggered request times the trigger rate must exceed 45% × average TTFT, on average. With 0.41% trigger rate, even saving 100% of TTFT per triggered request would only save ~0.4%, which can't recover 45%.

2. Agentic intra-session reuse leaves no headroom for migration. Most turns hit cache on the worker that handled the previous turn. Migrating prefill to a different worker is the exact thing intra-session affinity tries to avoid: it forces the new worker to pay for the cached prefix transfer instead of just reusing what's already on the affinity worker. This is a structural mismatch between PD-sep semantics ("send big prefills to a less-busy worker") and agentic workloads ("keep sessions sticky to wherever the cache is").

3. The Mooncake mechanism is 10–20× slower than the cost model predicts, primarily due to D-side pre-allocation of KV blocks and the absence of layerwise pipelining (E2 audit §6.1 / §6.3). The cost model can be re-calibrated, but doing so would push the gate even tighter, dropping the already-tiny trigger rate to nearly zero.

The three are stacked: even if any two were fixed, the remaining one would still make PD-sep a net loss on this trace.

What this section claims for the paper

1. Same-worker prefill–decode interference is a real mechanism (B2 microbench), but agentic workloads rarely expose it: the typical request has high cache hit and small uncached tail, so the interference cost is bounded.
2. Routing-only solutions (unified) already capture 79% of the intra-session APC ceiling and recover the latency by avoiding the heavy-tail sessions through the affinity gate. The remaining 23 pp gap to the ceiling is from APC LRU eviction under capacity pressure, not from prefill–decode interference.
3. Per-request PD-sep via Mooncake on agentic workloads is not a net win in our measurements, even with a carefully-gated cost model. The combined effect of kv_both substrate overhead, low trigger rate, and mechanism-vs-model gap is uniformly negative.
4. A productive direction is mechanism-level: fix the Mooncake D-side block reservation (E2 §6.1), implement layerwise transfer pipelining (E2 §6.3), and re-measure. Only if these patches drop the substrate tax to <10% and the realized transfer to ≤2 s p90 does PD-sep become competitive with routing on agentic traces.

What v2 still validates

• The cost model's qualitative shape is correct: when it says "migrate", that's a request where local interference would have been ≥ 4 s and src has ≥ 80% prefix cache. The model picks the right candidate requests.
• The gate logic catches the right exclusions: 88% by uncached tail size, 19% by no-decode-to-protect, the rest by missing source cache. Each is a structurally correct reason.
• The proxy shadow-drift fix is necessary infrastructure for any long-running routing experiment. We observed 3 phantom corrections per ~50-minute run.

Files

• data/b3_policy_comparison.json — the four policies' headline metrics from the same B3 sweep root.
• data/breakdown_<policy>.json — per-request proxy breakdown including v2 gate fields and triggered-event metadata.
• data/per_worker_<policy>.json — per-worker TTFT/latency p90s used in the hotspot figure.
• figures/*.png — the four section figures referenced above.
• render_figures.py — regenerates the figures from data/.

Cross-references

• analysis/characterization/window_1_results.md — B2 microbench (same-worker interference causal proof) and B3 baseline 5-policy sweep
• analysis/characterization/agentic_dispatch_coupling.md — why the saturated-replay setup matches agentic production
• analysis/characterization/b3_policies_pseudocode.md — pickers for the five baseline policies; unified_v2 extends unified
• E1 / E2 subagent reports (commit 4b833d3 message and the conversation log) — full mechanism audit that informed v2's design