agentic-kvc/microbench/connector_tax/cache_sweep/MIGRATION_TRANSFER_COST.md

Why KV-transfer is slow during migration under real load

Question. EAR's unified+A+B routing beats migration (v3) on agentic workloads. We wanted to know whether layerwise KV transfer would shrink migration's overhead enough to make it viable. Investigating that led to a sharper question: in a real (loaded) cluster, when we migrate, the KV transfer is already slow — the effective bandwidth is far below the ~10 GB/s wire rate. Why?

This doc answers that with instrumented measurements.

TL;DR. Migration fires precisely when instances are busy (that's the trigger). But on a busy instance, KV transfer runs at ~3 GB/s instead of ~10 GB/s, because:

1. The RDMA write itself slows ~2× under compute load — GPU-direct RDMA (batch_transfer_sync_write) contends with the running attention/MLP kernels for HBM and PCIe bandwidth. (idle 7.6 GB/s → busy 4.0 GB/s)
2. The connector's Python control plane gets GIL-starved — mooncake's ZMQ handshake + transfer orchestration run on asyncio threads inside the engine process; when the engine's main thread is doing a long forward pass (e.g. a 100k-token prefill), those threads stall for seconds.

Both are inherent to upstream vLLM 0.18.1 + mooncake (reproduced on a clean fresh venv; the transfer path is byte-identical to upstream — our patches did not cause this), and both get worse, not better, with layerwise transfer. So the bandwidth gap is not a layerwise problem; it is a transfer-on-a-busy-GPU problem.

---

1. Evidence chain

Three independent measurements, all on dash0 (8×H100, Qwen3-Coder-30B-A3B, TP=1), Mooncake kv_both.

1a. Instrumented v3 trace replay — where does migration time go?

Run outputs/b3_v3_fullbreak_20260528_0338/. Instruments: instrument_dst_migration.py (dst scheduler lifecycle) + instrument_mooncake.py (connector internals: send_blocks RDMA, receive_kv window, ready_wait).

25 migrations fired over the trace. Dst-side migration overhead (T_kv_pull = scheduler marks WAITING_FOR_REMOTE_KVS → finished_recving):

component	share	what it is
RDMA-actual (batch_transfer_sync_write)	55% (55.2 s)	the real RDMA write
dst control-plane gap	45% (45.4 s)	scheduler↔receiver_loop dispatch + completion propagation
ready_wait (src KV not committed)	0%	25/25 already committed — ruled out

• Pure RDMA aggregate rate: 2.03 GB/s (vs MB2 idle 9.7 GB/s).
• RDMA rate collapses with transfer size: <3 GiB → 4–9.5 GB/s,

  5 GiB → 0.9–2.6 GB/s.

• The control-plane gap is bimodal: median 0.04 s, but a handful of requests stall ~10 s. Those are small-KV transfers (0.18 s of actual RDMA) whose T_kv_pull is 8–11 s — i.e. the dst's receiver_loop thread was GIL-starved for ~10 s while the engine did a big forward pass.

Earlier (pre-instrumentation) we wrongly attributed ~90% of migration overhead to "dst scheduler queueing" by estimating transfer at clean wire speed. With real instrumentation, dst scheduler admission is ~0 (T_admission_post_kv = 0.003 s); the time is the transfer phase (RDMA + connector control plane), both degraded by instance busy-ness.

1b. MB6 controlled microbench — does busy-ness cause it?

microbench/fresh_setup/mb6_transfer_under_load.py + run_mb6.sh: 2 instances, transfer a fixed-size KV (prefill on A → migrate to B) while holding N background decode streams on both. Sweep N.

Effective transfer bandwidth (65k-token KV ≈ 6 GiB), main venv:

background load	65k transfer	eff bandwidth
0 (idle)	747 ms	8.76 GB/s
8 (4/instance)	2423 ms	4.53 GB/s
24 (12/instance)	2015 ms	3.33 GB/s

Monotonic degradation with load. The busy level (3.3 GB/s) matches the v3 trace's 3.3 GB/s median exactly — because agentic instances run ~10+ concurrent requests, i.e. the bg=24 regime.

Decomposing the 65k transfer into RDMA-actual vs control-plane:

bg	RDMA rate	control-plane share
0 (idle)	7.56 GB/s	13%
8	4.07 GB/s	11%
24 (busy)	3.97 GB/s	15%

In the clean microbench the RDMA write itself is the dominant degrading term (7.6 → 4.0 GB/s). The ~10 s control-plane stalls seen in the trace (1a) don't reproduce here because steady decode forward passes are short; they require the long (100k-token) prefills that the real trace has.

1c. Fresh-venv comparison — is it our patch?

Same MB6 sweep on agentic-kv-fresh/.venv (clean upstream-style 0.18.1):

bg	65k eff (fresh)	65k eff (main/patched)
0	8.73 GB/s	8.76 GB/s
8	4.52 GB/s	4.53 GB/s
24	3.27 GB/s	3.33 GB/s

Identical within noise. Plus a static check: the v3 transfer path (send_kv_to_decode, _send_blocks/batch_transfer_sync_write, _build_transfer_params) is byte-identical to pristine upstream 0.18.1 (commit 445e491); receive_kv_from_single_worker differs only by a 4-line error branch. Our mooncake commits (a7df84b direct-read, ea51497 partial-prefill, e3a1d70 read→push) only touch a separate direct_read path that v3 does not use (v3 requests carry no direct_read flag → normal push path).

→ The slowdown is upstream/hardware-inherent, not introduced by us.

---

2. Root cause

Migration in agentic serving transfers KV between instances that are concurrently busy with compute — by construction, since v3 migrates away from a busy host. On a busy instance:

• HBM/PCIe contention (the dominant, irreducible part). Mooncake's transfer is GPU-direct RDMA: the NIC DMAs KV straight out of / into GPU HBM. While the GPU runs attention+MLP kernels, those kernels saturate HBM bandwidth, so the NIC's RDMA gets a smaller slice. Effective transfer bandwidth roughly halves (7.6 → 4.0 GB/s at our load), and degrades further for large multi-segment transfers.
• Control-plane GIL starvation (secondary, bursty). The connector runs its ZMQ handshake + send_kv_to_decode/receive_kv orchestration on asyncio threads (sender_loop/receiver_loop) inside the engine process. A long forward pass (100k-token prefill) holds the GIL for seconds, stalling those threads → multi-second dispatch gaps even when the actual transfer is 0.2 s.

MB2 measured 9.7 GB/s precisely because both endpoints were idle. The real-workload gap is the difference between "idle benchmark" and "transfer while the GPU is doing the day job."

---

3. Implication: layerwise is the wrong lever; migration's tax is largely irreducible

lever	effect on the gap
Model-level layerwise transfer (push each layer's KV during prefill)	Worse. Prefill is the most HBM-intensive phase, so per-layer transfers contend harder for HBM (Cause 1); and they multiply the control-plane round-trips (Cause 2).
Control-plane fix (move mooncake orchestration off the GIL-contended threads / separate process)	Addresses only the bursty ~10 s stalls (~15% in the clean case, up to ~45% of the trace tail). Does not touch the HBM-contention half.
Reduce bytes (cache-aware target so less KV moves)	Helps linearly; v3 Mechanism B already does some. Orthogonal.
Migrate to/from idle instances	Would restore ~10 GB/s — but defeats the purpose (we migrate because the host is busy).

The dominant cost (RDMA contending with compute for HBM on busy instances) is a hardware reality, not a software bug we can patch away, and not something layerwise improves. This reinforces UNIFIED_ABLATION.md: the unified no-migration path (A+B'+RaceFix) remains the right default; migration's transfer tax is structural on a loaded agentic cluster.

---

4. Repro / artifacts

• Instrumented v3 breakdown: outputs/b3_v3_fullbreak_20260528_0338/unified_v3/ (transfer_decomp.txt, dst_migration_breakdown.{csv,png}, transfer_rootcause.png)
• MB6 main: outputs/mb6_agentic-kv_20260528_0552/mb6_result.json
• MB6 fresh: outputs/mb6_fresh_20260528_0559/mb6_result.json
• Instruments: microbench/fresh_setup/instrument_dst_migration.py, microbench/fresh_setup/instrument_mooncake.py
• Microbench: microbench/fresh_setup/mb6_transfer_under_load.py + run_mb6.sh (VENV=… bash run_mb6.sh)
• Analyzers: analyze_dst_migration.py, analyze_transfer_decomp.py

All instruments apply/revert cleanly via --apply/--revert; both venvs were restored after the runs.