agentic-kvc/analysis/mb2/README.md

# MB2 — Mooncake KV Transfer Cost (vanilla vLLM 0.18.1)

Persistent record of the per-stage KV transfer microbench used in §3.2 of
the EAR paper. Re-runs append a dated section at the bottom; the
**Summary** block at the top is what gets cited in the paper.

---

## Summary (latest)

| Path | Steady-state BW | Agentic-tail p99 transfer (11.5 GiB KV) |
|---|---|---|
| **intra-node** (dash1 GPU 0↔1) | **~9.7 GB/s** (96 MiB – 3 GiB) | p50 **1.9 s** · min **1.5 s** · max **10 s** |
| **inter-node** (dash1 GPU0 → dash2 GPU0, 200 Gbps RoCE) | **~10.0 GB/s** (essentially identical) | p50 **1.7 s** · min **1.3 s** · max **9.2 s** |

**Cross-cutting finding** (2026-05-27): **Mooncake transfer cost is
topology-independent** on this hardware. Intra-node and inter-node curves
are statistically indistinguishable (see `figs/mb2_transfer_time_compare.png`,
`figs/mb2_transfer_bw_compare.png`). Mechanism: Mooncake's
`batch_transfer_sync_write` always goes through the RDMA NIC, including
the intra-node case (RDMA loopback). The 200 Gbps NIC, not NVLink, is
the bottleneck. **Implication for §3.2**: PD-disaggregation does not
get cheaper by co-locating P and D on the same node — the ~9.7 GB/s
ceiling applies regardless. Halving the transfer cost cannot be bought
back by topology.

**What MB2 actually measures**: the **per-request charge** that
PD-disagg pays for every routed request — `T_transfer ≈ KV_size / 9.7
GB/s`. For agentic this is **8 ms (192 MiB / trace lower) – 1.9 s
(11.5 GiB / p99)**.

**⚠ Correction (2026-05-27)**: an earlier version of this README
framed §3.2 as "transfer cost (1.5–10 s) >> decode duration (50–200 ms),
so PD-disagg loses on cost-vs-benefit." That accounting was wrong:
PD-disagg's phase-isolation benefit is **per-prefill-event** and equals
`D × T_prefill` (aggregate across stalled decode streams), not the
single-request decode duration. With trace-mean `T_prefill = 4.5 s` and
D = 8, the benefit is ~36 s — far larger than the ~0.32 s transfer
cost. PD-disagg's phase-isolation axis is a *win*, not a loss.

The actual reason static PD-disagg fails in agentic is **D-side KV
capacity** (`figs/f4b_pdsep_kv_wall.png`), not a cost-vs-benefit
imbalance. See `RESULTS_SUMMARY.md` section 4 for the corrected
framing. MB2 still serves as the source of the per-request transfer
cost number used in that analysis.

---

## Setup

| Component | Value |
|---|---|
| Host | `dash1` (`ds-6348bee4-1-...-rwkv2`), 8× NVIDIA H20 96 GiB, driver 570.133.20 |
| Venv  | `/home/admin/cpfs/wjh/agentic-kv-fresh/.venv` (shared via cpfs from any dash host) |
| vLLM | 0.18.1 official wheel |
| mooncake-transfer-engine | 0.3.11.post1 (`pip install mooncake-transfer-engine`) |
| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct` |
| Per-token KV | 98304 B |
| kv_role | `kv_both` on both instances (see *Known limitations* re kv_producer/kv_consumer) |
| Per-instance config | `--tensor-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 200000 --enable-prefix-caching` |

## Method

3-step black-box bench:

1. `do_remote_decode` to A (producer) with a client-generated `transfer_id`.
   `max_tokens=1`; A computes prefill and parks the KV for later pull.
2. `do_remote_prefill` to B (consumer) with the same `transfer_id` plus
   `remote_engine_id` (from A's `/query` on bootstrap port) and
   `remote_bootstrap_addr` (`http://127.0.0.1:8998`). **This step
   triggers the actual KV transfer; it is the measured step.**
3. Plain `completion` on B (`--skip-verify` off): expect `cached_tokens
   ≈ prompt_len`, confirming the KV landed on B.

Per-stage breakdown is obtained by instrumenting the vLLM-shipped
`MooncakeConnector` (NOT the mooncake-package's `mooncake_connector_v1`,
which vLLM 0.18.1 does not load) at two sites:

- **`_send_blocks`** (P-side, line 980): emits `send_blocks` event with
  `total_bytes`, `duration_s`, `t_start_unix`. The `duration_s` is the
  wall-time of a single `batch_transfer_sync_write` call — **this is
  what we call `pure_transfer`**.
- **`receive_kv_from_single_worker`** (D-side, line 1139, async):
  emits `receive_kv_enter` at function start and `receive_kv_finish`
  on FINISH-status response. The wall-time between them is
  **`rx_total`** (= ZMQ round-trip + setup + pure_transfer + ack).

Pairing across A's and B's logs is by **time window**: each B
(enter, finish) pair is matched to the A send_blocks whose
`t_start_unix` falls in `[rx_t_start, rx_t_end]`. With single-request
benchmarks this is unambiguous.

Scripts:
- `microbench/fresh_setup/start_vllm_pair.sh` — bring up pair + apply/revert patch
- `microbench/fresh_setup/instrument_mooncake.py` — apply/revert MB2 patches
- `microbench/fresh_setup/mb2_kv_transfer.py` — client (3-step bench loop)
- `microbench/fresh_setup/analyze_mb2.py` — pair A/B events into per-size table
- `microbench/fresh_setup/plot_mb2.py` — log-log time + bandwidth curves

## Results — intra-node (2026-05-27, dash1 GPU 0+1, kv_both)

Raw events: `A_intra_kvboth.jsonl`, `B_intra_kvboth.jsonl`.
Joined + aggregated: `intra_kvboth_breakdown.json`.
Figures: `figs/mb2_transfer_time_intra.png`, `figs/mb2_transfer_bw_intra.png`.

| input_tokens | KV (MiB) | n | pure_ms p50 | pure_ms max | rx_total_ms | overhead_ms | BW p50 (GB/s) | BW max (GB/s) |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|    512 |    48 | 5 |   5.3 |   5.6 |  12.2 |  3.3 | 9.40 |  9.53 |
|   1024 |    96 | 5 |  10.4 |  10.5 |  11.9 |  1.5 | 9.68 |  9.72 |
|   2048 |   192 | 5 |  20.6 |  21.0 |  22.5 |  1.8 | 9.75 |  9.78 |
|   4096 |   384 | 5 |  41.5 |  41.7 |  43.5 |  2.0 | 9.71 |  9.72 |
|   8192 |   768 | 5 |  83.7 |  84.4 |  86.2 |  2.2 | 9.62 |  9.69 |
|  16384 |  1536 | 5 | 167.1 | 167.7 | 170.2 |  2.7 | 9.64 |  9.67 |
|  32768 |  3072 | 5 | 320.9 | 322.1 | 425.2 | 20.5 | 10.04 | 10.09 |
|  65536 |  6144 | 5 | **1895.1** | 2375.2 | 1586.1 | 69.6 | 3.40 | 9.68 |
| 131072 | 12288 | 5 | **2835.1** | 8923.6 | 4362.5 | 91.4 | 4.54 | 9.67 |

**Three regimes** in the data:

1. **<= 3 GiB** — linear in size, bandwidth ≈ **9.7 GB/s steady**.
2. **6 GiB ± a bit** — onset of variance: max bandwidth still 9.7 GB/s,
   but p50 collapses to ~3.4 GB/s. Some runs achieve full speed; others
   take 2–3 × longer.
3. **12 GiB** — wide spread (min 1.5 s, max 10 s for the same 11.5 GiB
   transfer). This is the agentic-p99 size region.

The bandwidth ceiling of ~10 GB/s is well below H20's NVLink p2p
(claimed ~900 GB/s in IB) — likely the transfer is **PCIe-staged
through host memory** rather than NVLink direct. To confirm we would
need `nvidia-smi topo -m` and `mooncake_transfer_engine_topology_dump`
analysis; not done yet.

## Known limitations of this measurement

- **kv_both, not strict PD-disagg.** vLLM 0.18.1 with
  `kv_role=kv_consumer` raises `AttributeError: 'MooncakeConnectorWorker'
  object has no attribute 'bootstrap_server'` (the attribute is only
  assigned inside `if not self.is_kv_consumer`). The transfer mechanics
  are identical — same `batch_transfer_sync_write` — so the cost
  measurement is comparable. The role gate only affects which request
  types each instance *accepts*. §5.2 strict PD-disagg baseline will
  need either to fix that bug or front the pair with a role-aware proxy.
- **Single in-flight request.** All measurements here are serial.
  Real PD-disagg will have many concurrent transfers; bandwidth
  contention is not characterized.
- **Intra-node only.** Inter-node RDMA path will be slower; not yet
  measured.
- **Sanity preamble events.** The raw logs include 6 events from
  earlier sanity runs in addition to the 45-event sweep. `analyze_mb2.py`
  treats them as additional samples (same sizes); the per-size
  aggregates use all of them.

## Implications for §3.2 PD-disagg argument

For each PD-disagg-routed request, transfer wall-time is:

```
T_transfer(KV_size) ≈ KV_size / 9.7 GB/s   for KV_size ≤ 3 GiB
                    ≈ 0.3 – 10 s            for KV_size in [3, 12] GiB
```

This is the **per-request transfer charge** of PD-disagg. It's a
real cost, but in the context of phase-isolation accounting it is
*small* compared to the benefit:

| Prefill | T_prefill (MB1) | T_transfer (MB2) | Phase-isolation benefit at D=8 = D × T_prefill |
|---:|---:|---:|---:|
| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s |
| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s |
| 125k tok (~p99) | 57 s | 1.9 s | 456 s |

On the phase-isolation axis alone, PD-disagg recovers two orders of
magnitude more decode time than it pays in transfer. **It is NOT this
axis that defeats static PD-disagg in agentic** — see colleague's
4P+4D experiment (TTFT p50 62×, success rate 99.5% → 52%) which is
driven by **D-side KV-pool overflow** on long-context requests
(`figs/f4b_pdsep_kv_wall.png`), not by transfer latency.

What MB2 contributes to the paper is therefore:
- The **per-request transfer cost number** (used as the cost input
  to the cost-benefit accounting above).
- The empirical observation that **Mooncake's transfer cost is
  topology-independent** — intra-node and inter-node both go through
  the RDMA NIC and hit the same 9.7 GB/s ceiling. PD-disagg's
  transfer cost does not get cheaper by co-locating P and D.

The dominant §3.2 failure mode of static PD-disagg in agentic is
**capacity**, not transfer cost. MB3 / MB4 / MB5 will quantify the
remaining axes (D-pool occupancy, cache reuse degradation under PD
routing, static-partition mismatch).

## Open questions / next runs

- **Inter-node RDMA**: dash1 ↔ dash2. Expected lower bandwidth (~5–15
  GB/s); want to see if the 6 GiB-onset variance moves.
- **Bandwidth ceiling investigation**: is the 9.7 GB/s ceiling PCIe (so
  the connector is not using NVLink direct) or some internal limit? If
  PCIe, can it be lifted with NVLink-direct mooncake config?
- **Variance at 6+ GiB**: investigate. Maybe related to chunking
  inside `batch_transfer_sync_write`, or GPU memory pressure when KV
  approaches HBM ceiling.
- **Concurrent transfers**: measure aggregate bandwidth when N
  simultaneous transfers happen. PD-disagg in practice does this.
- **Strict kv_producer/kv_consumer**: patch the bootstrap_server bug
  or use a proxy; verify transfer time is unchanged.

## Reproduction

```bash
# On dash machine with cpfs mount + ssh access:
bash microbench/fresh_setup/install.sh        # once (idempotent)
bash microbench/fresh_setup/deploy.sh dash1    # push scripts to cpfs

# bring up pair (intra-node)
ssh dash1 'GPU_A=0 GPU_B=1 bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/start_vllm_pair.sh start'

# sweep
ssh dash1 'source /home/admin/cpfs/wjh/agentic-kv-fresh/.venv/bin/activate && \
  python /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb2_kv_transfer.py \
    --sizes 512,1024,2048,4096,8192,16384,32768,65536,131072 \
    --repeats 5 --label intra-kvboth \
    --out /home/admin/cpfs/wjh/agentic-kv-fresh/mb2_results/intra_kvboth.json'

# pull logs
scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb2_transfer_logs/A/.efc_*_mb2_transfer_pid*.jsonl \
    analysis/mb2/A_intra_kvboth.jsonl
scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb2_transfer_logs/B/.efc_*_mb2_transfer_pid*.jsonl \
    analysis/mb2/B_intra_kvboth.jsonl

# analyze
.venv/bin/python microbench/fresh_setup/analyze_mb2.py \
  --a-log analysis/mb2/A_intra_kvboth.jsonl \
  --b-log analysis/mb2/B_intra_kvboth.jsonl \
  --out analysis/mb2/intra_kvboth_breakdown.json

.venv/bin/python microbench/fresh_setup/plot_mb2.py \
  --breakdown analysis/mb2/intra_kvboth_breakdown.json \
  --out-time figs/mb2_transfer_time_intra.png \
  --out-bw figs/mb2_transfer_bw_intra.png

# tear down
ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/start_vllm_pair.sh stop'
```

## Run log

### 2026-05-27 — intra-node, kv_both, dash1 GPU 0+1

Sweep: `512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072` tokens
× 5 repeats. Sanity preamble of `512, 2048, 8192` × 2 included in the
raw logs (counted as additional samples for those sizes).

Result table above. **9.7 GB/s steady-state up to 3 GiB**, variance
opens at 6 GiB, p99 agentic-tail transfer 1.5 – 10 s.

Committed as `de164e5`.

### 2026-05-27 — inter-node, kv_both, dash1 GPU 0 → dash2 GPU 0

Same sweep config. 200 Gbps RoCE between hosts (RTT ~0.2 ms ping).
Producer A on dash1 GPU 0, consumer B on dash2 GPU 0.
remote_bootstrap_addr=`http://172.27.123.142:8998` (dash1's internal IP).

Raw events: `A_inter_kvboth.jsonl` (45 send_blocks + 6 sanity).
B's receive_kv events are **missing** for this run — the
`MB2_LOG_DIR` env var did not propagate from the start-script through
vLLM's EngineCore subprocess on dash2 (visible via
`cat /proc/$ENGINE_PID/environ` shows empty for dash2 but contains
MB2_LOG_DIR for dash1 — bookmark for future investigation, likely
spawn-vs-fork difference in vLLM's multiproc executor across hosts).
Pure-transfer numbers below come from A's send_blocks alone; full
rx_total breakdown not available for this run.

Per-size pure-transfer (analyzed by `analyze_mb2_send_only.py`):

| input_tokens | KV (MiB) | n | pure_ms p50 | min | max | BW p50 (GB/s) | BW max |
|---:|---:|---:|---:|---:|---:|---:|---:|
|    512 |    48 | 5 |   5.2 |   5.1 |  65.8 |  9.76 |  9.81 |
|   1024 |    96 | 5 |  10.2 |  10.1 |  10.4 |  9.91 | 10.00 |
|   2048 |   192 | 5 |  20.0 |  20.0 |  20.5 | 10.06 | 10.07 |
|   4096 |   384 | 5 |  40.1 |  40.1 |  40.5 | 10.04 | 10.05 |
|   8192 |   768 | 5 |  80.9 |  80.7 |  82.5 |  9.96 |  9.98 |
|  16384 |  1536 | 5 | 161.8 | 161.7 | 164.8 |  9.96 |  9.96 |
|  32768 |  3072 | 5 | 309.6 | 307.7 | 526.9 | 10.40 | 10.47 |
|  65536 |  6144 | 5 | 1733.6 | 653.5 | 1921.2 | 3.72 | 9.86 |
| 131072 | 12288 | 5 | 2818.4 | 1283.0 | 9158.6 | 4.57 | 10.04 |

Side-by-side comparison with the 2026-05-27 intra-node run:

| Size | intra p50 ms | inter p50 ms | gap | intra GB/s | inter GB/s |
|---|---:|---:|---:|---:|---:|
|   512 |    5.3 |    5.2 | −2% | 9.40 |  9.76 |
|  1024 |   10.4 |   10.2 | −2% | 9.68 |  9.91 |
|  2048 |   20.6 |   20.0 | −3% | 9.75 | 10.06 |
|  4096 |   41.5 |   40.1 | −3% | 9.71 | 10.04 |
|  8192 |   83.7 |   80.9 | −3% | 9.62 |  9.96 |
| 16384 |  167.1 |  161.8 | −3% | 9.64 |  9.96 |
| 32768 |  320.9 |  309.6 | −3% | 10.04 | 10.40 |
| 65536 | 1895.1 | 1733.6 | −9% | 3.40 |  3.72 |
|131072 | 2835.1 | 2818.4 | −1% | 4.54 |  4.57 |

The two paths produce essentially the same numbers — **mooncake intra-
node is not using NVLink**, it's going through RDMA-loopback on the
local NIC and gets the same ~10 GB/s ceiling as cross-node RDMA. The
6+ GiB variance regime is also identical between paths.

Figures: `figs/mb2_transfer_time_inter.png`, `figs/mb2_transfer_bw_inter.png`,
`figs/mb2_transfer_time_compare.png` (overlay), `figs/mb2_transfer_bw_compare.png`.

This collapses the §3.2 narrative to a single number: **PD-disagg
across this cluster costs ~9.7–10 GB/s of transfer bandwidth no matter
how you place P and D** (within-node or across-node). For p99 agentic
KV (11.5 GiB), that's 1.3–10 s of transfer; for 6 GiB it's 0.7–2 s.
Decode is 50–200 ms. So PD-disagg's cost dominates regardless of layout.