gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Migration transfer-cost study: KV transfer is slow on busy GPUs

Browse Source 
MIGRATION_TRANSFER_COST.md: under real load, migration KV transfer runs at
~3 GB/s vs ~10 GB/s idle. Decomposed (instruments + MB6 microbench) into
~55% RDMA-actual (HBM/PCIe contention with running kernels: 7.6->4.0 GB/s)
+ ~45% control-plane GIL starvation during long prefills. Reproduced on a
fresh upstream venv (byte-identical transfer path) -> upstream/hardware
inherent, not our patch. Layerwise is the wrong lever; the tax is structural
on a loaded agentic cluster. Includes mb6_transfer_under_load + run_mb6,
instrument_dst_migration/mooncake, and the dst/transfer decomposition analyzers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-29 11:53:01 +08:00
parent 67fcec7933
commit 1262c9c22e
11 changed files with 2055 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
microbench/fresh_setup/instrument_mooncake.py
View File

@@ -151,11 +151,65 @@ RECV_FINISH_REPLACE = f""" if response.status == MooncakeXfer
{END_MARK}
break"""
# ---- Patch 5: send_kv_to_decode entry (P-side, producer receives pull req) ----
SEND_ENTRY_TARGET = """ async def send_kv_to_decode(
self, identity: bytes, sock: zmq.asyncio.Socket, meta: MooncakeXferMetadata
):
pending_reqs: dict[ReqId, SendBlockMeta] = {}"""
SEND_ENTRY_REPLACE = f""" async def send_kv_to_decode(
self, identity: bytes, sock: zmq.asyncio.Socket, meta: MooncakeXferMetadata
):
pending_reqs: dict[ReqId, SendBlockMeta] = {{}}
{START_MARK}
try:
_mb2_log_event({{"event": "send_kv_to_decode_enter",
"d_req_ids": [str(r) for r in meta.req_blocks],
"t_start_unix": _mb2_time.time(),
"tp_rank": getattr(self, "tp_rank", -1)}})
except Exception:
pass
{END_MARK}"""
# ---- Patch 6: wait_and_ret ready-wait timing (P-side, src KV commit wait) ----
READY_WAIT_TARGET = """ async def wait_and_ret(
d_req_id: ReqId, send_meta: SendBlockMeta
) -> tuple[ReqId, SendBlockMeta]:
await send_meta.ready.wait()
return d_req_id, send_meta"""
READY_WAIT_REPLACE = f""" async def wait_and_ret(
d_req_id: ReqId, send_meta: SendBlockMeta
) -> tuple[ReqId, SendBlockMeta]:
{START_MARK}
_mb2_rw_start = _mb2_time.perf_counter()
_mb2_rw_start_unix = _mb2_time.time()
_mb2_rw_already = send_meta.ready.is_set()
{END_MARK}
await send_meta.ready.wait()
{START_MARK}
try:
_mb2_log_event({{"event": "ready_wait",
"d_req_id": str(d_req_id),
"transfer_id": str(getattr(send_meta, "transfer_id", "")),
"ready_already_set": bool(_mb2_rw_already),
"ready_wait_s": _mb2_time.perf_counter() - _mb2_rw_start,
"t_start_unix": _mb2_rw_start_unix,
"tp_rank": getattr(self, "tp_rank", -1)}})
except Exception:
pass
{END_MARK}
return d_req_id, send_meta"""
PATCHES = [
("header", HEADER_ANCHOR, HEADER_ANCHOR + HEADER_INSERT),
("_send_blocks", SEND_TARGET, SEND_REPLACE),
("receive_kv (entry)", RECV_ENTRY_TARGET, RECV_ENTRY_REPLACE),
("receive_kv (FINISH)", RECV_FINISH_TARGET, RECV_FINISH_REPLACE),
("send_kv (entry)", SEND_ENTRY_TARGET, SEND_ENTRY_REPLACE),
("ready_wait", READY_WAIT_TARGET, READY_WAIT_REPLACE),
]