# v2 — Evidence for the GPU-hit-first principle (§2.2)

Two experiments that turn "**Hits on GPU > hits on CPU**" + "**GPU is enough to
hold most of the *valuable* KV reuse**" from assertion into measurement.

Hardware: dash0, 1× NVIDIA H20 (97 GB) per experiment, Qwen3-Coder-30B-A3B-Instruct,
vLLM 0.18.1 (V1, prefix caching, enforce-eager). KV = 96 KiB/token (1 GiB = 10,923 tok).

## Exp (a) — three-tier hit latency  (`exp_a_tier_latency/`)
TTFT of serving a reused prefix of length L from each tier:
- **miss**  — fresh unique prompt → full prefill (recompute)
- **GPU hit** — re-request → HBM prefix cache
- **CPU hit** — warm → evict to CPU offload tier (`--kv-offloading-size`) → re-request → DRAM fetch
- **PCIe floor** — direct pinned-memory H2D transfer cost for the same KV size (backstop)

Tier of each measured request is *verified* via `vllm:prefix_cache_hits` vs
`vllm:external_prefix_cache_hits` deltas, not assumed.

Run: `GPU=0 bash v2/exp_a_tier_latency/run.sh` then `.venv/bin/python v2/exp_a_tier_latency/plot.py`.

## Exp (b) — capacity → APC → latency knee  (`exp_b_capacity_knee/`)
Replay a fixed agentic trace at several GPU KV pool sizes
(`--num-gpu-blocks-override`); measure realized APC + TTFT p90 per capacity.
The knee = the GPU capacity beyond which more HBM buys ~no extra reuse.

Run: `GPU=1 bash v2/exp_b_capacity_knee/run_sweep.sh` then
`.venv/bin/python v2/exp_b_capacity_knee/analyze_and_plot.py`.

## Results (dash0, 2026-05-30)

### Exp (a) — GPU hit > CPU hit > remote-store(RDMA) hit ≫ miss  (`figs/exp_a_tier_latency.png`)

TTFT (s, p50 over reps) to serve a reused prefix of length L from each KV tier.
Local CPU-tier hits were 100% verified via `vllm:external_prefix_cache_hits`;
the **remote KV-store** tier is a real cross-instance Mooncake hit — instance B
serves the request by **pulling the cached prefix from instance A over RDMA**
(`do_remote_prefill`) instead of recomputing (the Mooncake-Store-blog mechanism),
measured with `microbench/fresh_setup/mb2_kv_transfer.py`.

| prefix L | miss (recompute) | **remote RDMA store** | CPU-tier (local) | GPU-tier (HBM) | miss/RDMA | RDMA/CPU | CPU/GPU |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 1k  | 0.078 | 0.061 | 0.057 | 0.042 | 1.3× | 1.1× | 1.4× |
| 8k  | 0.588 | 0.151 | 0.076 | 0.053 | 3.9× | 2.0× | 1.5× |
| 16k | 1.547 | 0.262 | 0.105 | 0.063 | 5.9× | 2.5× | 1.7× |
| 32k | 4.604 | 0.680 | 0.158 | 0.080 | 6.8× | 4.3× | 2.0× |
| **64k** | **15.230** | **0.966** | **0.272** | **0.111** | **15.8×** | **3.6×** | **2.4×** |

- **GPU hit is ~flat** (42→111 ms over 1k→64k): a hit returns the whole prefix from
  HBM, only the last token is recomputed.
- **miss grows superlinearly** (→15.2 s at 64k): a miss pays the full prefill.
- **local CPU hit grows transfer-bound** (PCIe H2D measured **~54 GB/s**); CPU-hit
  TTFT ≈ GPU-hit + KV/PCIe + ~0.15 s overhead (dashed PCIe floor sits just under it).
- **remote RDMA-store hit** is the L3 tier the Mooncake-Store blog advocates: it is
  a big win over recompute (**up to 16× lower TTFT**, consistent with the blog's
  46× at higher hit rates) — but it pays the **NIC tax** (~5–7 GB/s effective here,
  cf. ~9.7 GB/s raw Mooncake RDMA in MB2; multi-NIC pooling would raise it). So it
  is **3.6× slower than a local CPU hit and ~9× slower than a GPU hit** at 64k, and
  the gap **grows with context length**.
- **Takeaway — the tier ordering is strict and widens with context:**
  **GPU < CPU-local < remote-RDMA-store ≪ miss.** A global KV store helps (vs
  recompute), which is why that approach exists; but every step *toward* the GPU is
  another 1.4–4× of TTFT. The reuse that matters most is the GPU-resident kind.

### Exp (b) — APC and latency knee at small GPU capacity  (`figs/exp_b_capacity_knee.png`)

Closed-loop replay (concurrency 4) of a controlled multi-turn workload (24 sessions
× 6 turns, cumulative intra-session prefix, per-session working set **0.91 GB**,
intra-session APC ceiling 71%), sweeping GPU KV capacity.

| GPU KV (GB) | realized APC | TTFT p50 | TTFT p90 | E2E p90 | completion |
|---:|---:|---:|---:|---:|---:|
| 1.2 | 7.4%  | 8.32 | 13.00 | 16.54 | 100% |
| 1.6 | 12.2% | 4.02 | 8.90  | 12.41 | 100% |
| 2.4 | 36.3% | 0.47 | 4.62  | 8.66  | 100% |
| **3.6** | **80.3%** | **0.41** | **0.53** | **4.33** | 100% |
| 4.8 | 72.9% | 0.49 | 0.65 | 4.27 | 100% |
| 7.2 | 72.9% | 0.49 | 0.64 | 4.25 | 100% |
| 9.7 | 72.9% | 0.49 | 0.65 | 4.19 | 100% |
| 14.5| 72.9% | 0.49 | 0.65 | 4.25 | 100% |

- **Sharp knee at 3.6 GB** = exactly the active working set (4 sessions × 0.91 GB).
  APC saturates at the ~71% ceiling; **TTFT p90 collapses 13.0 s → 0.53 s** at the
  same point. Beyond the knee, **more HBM buys nothing** (dead flat to 14.5 GB).
- Below the knee, sessions evict each other between turns → cache misses →
  recompute → 13 s TTFT. The knee is where the working set becomes GPU-resident.

## Conclusion (for §2.2)

1. **The KV-tier hierarchy is now measured, not asserted** (Exp a):
   `GPU(HBM) < CPU(local DRAM) < remote KV-store(RDMA) ≪ miss`. At 64k tokens a GPU
   hit (0.11 s) is 2.4× faster than a local CPU hit, ~9× faster than a remote RDMA
   store hit, and 137× faster than recompute; the gaps **grow with context length**.
   A global RDMA store (Mooncake-Store blog) is a real win over recompute (up to 16×
   here / 46× in the blog) — but it pays the NIC tax, so it sits a tier *below* local
   CPU and two below GPU. Each step toward the GPU is another 1.4–4× of TTFT.
2. **You only need to hold the *active working set* on GPU.** Realized APC and
   latency saturate once HBM covers the concurrent sessions' working set (3.6 GB
   here); past that, extra capacity — and the entire CPU/storage tier built to chase
   the long reuse tail — adds ~0 (Exp b). The knee scales linearly with concurrency,
   i.e. with **cluster GPU count**, which the production cluster already provides.
3. Together: maximize GPU residency of the active working set (colocation + affinity
   routing + dedup-migration); the CPU tier is a fallback, not the primary path.

## Caveats
- Exp (b) uses a controlled multi-turn workload (the production trace is 90%
  single-turn with huge per-request contexts that thrash a single instance — see
  C1/f2c); it isolates the capacity→APC→latency mechanism. Knee *position* scales
  with concurrency × per-session working set.
- Single H20; PCIe H2D ~54 GB/s is intra-node (cf. 9.7 GB/s Mooncake inter-node RDMA).
- Remote-RDMA tier is a single-node 2-instance Mooncake measurement (RDMA loopback
  through the NIC; MB2 showed intra ≈ inter, NIC-bound). `t_transfer` includes the
  request + 1-token decode + dst scheduling, so effective BW (~5–7 GB/s) is below the
  raw ~9.7 GB/s; this is the realistic end-to-end remote-hit latency, not just the
  wire transfer. The connector's retention-verify (`cached_followup`) is 0 because
  kv_both `do_remote_prefill` does not reinsert the pulled prefix into dst's
  persistent prefix cache — it does not affect the measured pull latency.
- The 80.3% point at the knee slightly exceeds the 71% intra-session ceiling
  (transient full residency / generated-token reuse); steady state is 72.9%.