# Cache-size Sweep — Results

Run: `results/20260526_1507/`
Hardware: H20 96 GB × 1, TP=1, Qwen3-Coder-30B-A3B-Instruct,
`gpu-memory-utilization=0.9`, `enable_prefix_caching=true`.
Cache ceiling reached on this GPU: **17 528 blocks**.

## TL;DR

H5 (build_connector_meta walks `set(cache.keys())` per step, so cost
grows linearly with |cache|) **passes**.

- mooncake_both: step_duration_us p50 grows from **276 μs (cache=2.6k blocks)**
  to **1528 μs (cache=16.6k blocks)** — linear fit slope **+85.6 μs / 1 000 blocks**.
- plain: **+1.0 μs / 1 000 blocks** (≈ zero, control).
- noop_connector: **−0.8 μs / 1 000 blocks** (≈ zero, control).

`build_connector_meta` accounts for **94 % of the scheduler-side
cost at full cache** (1442 / 1528 μs at the top bin). The vLLM v1
framework dispatch alone (noop_connector vs plain) is **~20 μs**.

The original microbench's **"100 % from build_meta"** claim was an
artefact of *not measuring* the worker-side path. With both sides
measured here, the picture is:

| cost component | mooncake_both (μs/step) | scaling |
|---|---:|---|
| **scheduler `build_connector_meta`** | 207 (cache=2.6k) → **1442 (cache=16.6k)** | **O(\|cache\|)** |
| **worker `get_finished()`** | **p50 = 180 μs, p99 = 333 μs** (independent of \|cache\|) | constant |
| **worker `start_load_kv()`** | p50 = 2-5 μs | constant |
| **framework dispatch** (noop−plain) | ≈ 20 μs | constant |

So the previously-uninstrumented `get_finished()` adds another **180 μs
per step on top** of the cache-dependent build_meta. At low cache size
that's the dominant connector cost; at high cache size it's
overshadowed by build_meta but still adds ~10 %.

## The figure

![per-step time vs cache_size](figure.png)

Left: full step time. Right: just the `build_connector_meta`
component. plain and noop stay flat at ~80 μs across the whole range;
mooncake_both rises near-linearly.

## How this changes the trace-replay reconciliation

The 8-instance trace replay (`analysis/characterization/elastic_migration_v2`)
ran with APC ≈ 79 %, i.e. each instance's block pool held **~13 000
blocks**. Plugging that into the fit:

```
mooncake build_meta @ |cache|=13 000 ≈ 1060 μs / step
mooncake get_finished                ≈  180 μs / step
total per-step connector cost        ≈ 1240 μs ≈ 1.24 ms / step
```

Decode-step model forward on Qwen3-Coder-30B-A3B / H20 is ~6-9 ms
TPOT, so 1.24 ms of extra scheduler-and-worker time per step is a
**+15-20 % TPOT inflation** purely from the per-step connector cost —
before any inter-instance coupling.

This matches the trace-replay TPOT p90 +25 % gap quite well. The
**residual ~7 pp** can be attributed to:

1. **Block-pool LRU churn under capacity pressure** (random-content
   bench reaches ceiling quickly; trace-replay holds at ceiling
   for the full session-coupled workload).
2. **Block-lifecycle changes** (`delay_free_blocks=True` once any
   connector is loaded; the freed-block backlog is larger under
   high APC).
3. **Multi-instance scheduler coupling**: the slowest scheduler in
   8-way load_only sets the proxy's batch latency.

For the **+45 % TTFT p90 gap**, the same scheduler tax compounds
across many chunked-prefill steps. A 50-step prefill at 1.24 ms extra
each step is +62 ms, which is on the order of the typical TTFT delta
we see at moderate load.

## How this changes the "decomposition" attribution

The original RESULTS.md said:

> +7-9 % from build_connector_meta per-step cost (this microbench)
> +20-30 % from multi-instance coupling amplification (not measurable)
> remainder from large-cache O(\|cache\|) scaling (Phase B follow-up)

The cache-size sweep replaces the third row with a measurement and
forces the first row to be re-read:

| factor | original claim | revised |
|---|---|---|
| single-instance high-conc tax | +7-9 % | unchanged — that was measured at low \|cache\| |
| multi-instance coupling | +20-30 % | still un-measured, but a *smaller* slice than thought |
| large-cache O(\|cache\|) scaling | "likely 2-3×" | **measured: +85.6 μs/1k blocks; ≈ 1 ms/step at \|cache\|=13k** |
| worker-side get_finished | not in the model | **measured: +180 μs/step (constant)** |

The "trace-replay 45 % TTFT p90" is now explainable mostly from
cache-size + worker get_finished + framework dispatch, without
having to invoke a large multi-instance coupling term. The data is
also consistent with NIXL's smaller tax (NIXL doesn't walk the
block-pool dict in scheduler.build_connector_meta; the trace-replay
NIXL vs plain gap of +38 % is consistent with "only the get_finished
+ framework constant" parts, lacking the O(\|cache\|) component).

## What this still doesn't settle

1. **Multi-instance coupling**: the 8-instance run would need its
   own cache-size sweep + per-instance step instrumentation. We
   know the per-instance per-step cost; what we don't know is how
   that propagates through the cache-aware proxy's routing
   decisions.
2. **Larger \|cache\| extrapolation**: H20 96 GB caps at ~17.5 k
   blocks at the configured memory. Settings with smaller models
   (or `gpu-memory-utilization` ≥ 0.95 on bigger GPUs) reach
   higher \|cache\|. The slope is linear in this range, but we
   have not extrapolated past ~17 k.
3. **NIXL slope**: NIXL was in the prior microbench's plan but
   not in this run. Same instrumentation on NIXL would confirm
   whether NIXL has a different (smaller) slope.

## Practical recommendation

The root cause is clearly identifiable: the per-scheduler-step
`set(self._block_pool.cached_block_hash_to_block._cache.keys())` walk
in `mooncake_connector.py:434-450`. Replacing it with an incremental
delta listener (using the block-pool's existing
`add`/`remove`/`evict` callbacks) would zero out the cache-size
slope and bring mooncake_both into the same ballpark as noop_connector
on the scheduler side.

The worker-side `get_finished` cost (180 μs constant) is also
fixable: in `kv_both` mode it submits two empty `coroutine_threadsafe`
futures every step. Caching/coalescing or short-circuiting when both
queues are empty would eliminate this constant.

## Reproducibility

```
cd microbench/connector_tax/cache_sweep
bash run_all.sh           # ~22 min on H20 single-GPU
```

The orchestrator applies v1 + v2 patches, runs the three configs
sequentially, reverts both patches on exit, and produces
`results/<date>/SUMMARY.md` + `figure.png`.

Artifacts in `results/20260526_1507/`:
- `figure.png` — the headline plot
- `SUMMARY.md` — per-config tables (this report's source)
- `per_config.json` — machine-readable
- per-config: `engine_step.jsonl`, `worker_step.r0.jsonl`,
  `requests.jsonl`, `metrics_final.txt`, vLLM stdout/stderr