agentic-kvc/microbench/connector_tax/cache_sweep/REPORT.md

Cache-size Sweep — Results

Run: results/20260526_1507/ Hardware: H20 96 GB × 1, TP=1, Qwen3-Coder-30B-A3B-Instruct, gpu-memory-utilization=0.9, enable_prefix_caching=true. Cache ceiling reached on this GPU: 17 528 blocks.

TL;DR

H5 (build_connector_meta walks set(cache.keys()) per step, so cost grows linearly with |cache|) passes.

• mooncake_both: step_duration_us p50 grows from 276 μs (cache=2.6k blocks) to 1528 μs (cache=16.6k blocks) — linear fit slope +85.6 μs / 1 000 blocks.
• plain: +1.0 μs / 1 000 blocks (≈ zero, control).
• noop_connector: −0.8 μs / 1 000 blocks (≈ zero, control).

build_connector_meta accounts for 94 % of the scheduler-side cost at full cache (1442 / 1528 μs at the top bin). The vLLM v1 framework dispatch alone (noop_connector vs plain) is ~20 μs.

The original microbench's "100 % from build_meta" claim was an artefact of not measuring the worker-side path. With both sides measured here, the picture is:

cost component	mooncake_both (μs/step)	scaling
scheduler build_connector_meta	207 (cache=2.6k) → 1442 (cache=16.6k)	O(|cache|)
worker get_finished()	p50 = 180 μs, p99 = 333 μs (independent of |cache|)	constant
worker start_load_kv()	p50 = 2-5 μs	constant
framework dispatch (noop−plain)	≈ 20 μs	constant

So the previously-uninstrumented get_finished() adds another 180 μs per step on top of the cache-dependent build_meta. At low cache size that's the dominant connector cost; at high cache size it's overshadowed by build_meta but still adds ~10 %.

The figure

Left: full step time. Right: just the build_connector_meta component. plain and noop stay flat at ~80 μs across the whole range; mooncake_both rises near-linearly.

How this changes the trace-replay reconciliation

The 8-instance trace replay (analysis/characterization/elastic_migration_v2) ran with APC ≈ 79 %, i.e. each instance's block pool held ~13 000 blocks. Plugging that into the fit:

mooncake build_meta @ |cache|=13 000 ≈ 1060 μs / step
mooncake get_finished                ≈  180 μs / step
total per-step connector cost        ≈ 1240 μs ≈ 1.24 ms / step

Decode-step model forward on Qwen3-Coder-30B-A3B / H20 is ~6-9 ms TPOT, so 1.24 ms of extra scheduler-and-worker time per step is a +15-20 % TPOT inflation purely from the per-step connector cost — before any inter-instance coupling.

This matches the trace-replay TPOT p90 +25 % gap quite well. The residual ~7 pp can be attributed to:

1. Block-pool LRU churn under capacity pressure (random-content bench reaches ceiling quickly; trace-replay holds at ceiling for the full session-coupled workload).
2. Block-lifecycle changes (delay_free_blocks=True once any connector is loaded; the freed-block backlog is larger under high APC).
3. Multi-instance scheduler coupling: the slowest scheduler in 8-way load_only sets the proxy's batch latency.

For the +45 % TTFT p90 gap, the same scheduler tax compounds across many chunked-prefill steps. A 50-step prefill at 1.24 ms extra each step is +62 ms, which is on the order of the typical TTFT delta we see at moderate load.

How this changes the "decomposition" attribution

The original RESULTS.md said:

+7-9 % from build_connector_meta per-step cost (this microbench) +20-30 % from multi-instance coupling amplification (not measurable) remainder from large-cache O(|cache|) scaling (Phase B follow-up)

The cache-size sweep replaces the third row with a measurement and forces the first row to be re-read:

factor	original claim	revised
single-instance high-conc tax	+7-9 %	unchanged — that was measured at low |cache|
multi-instance coupling	+20-30 %	still un-measured, but a smaller slice than thought
large-cache O(|cache|) scaling	"likely 2-3×"	measured: +85.6 μs/1k blocks; ≈ 1 ms/step at |cache|=13k
worker-side get_finished	not in the model	measured: +180 μs/step (constant)

The "trace-replay 45 % TTFT p90" is now explainable mostly from cache-size + worker get_finished + framework dispatch, without having to invoke a large multi-instance coupling term. The data is also consistent with NIXL's smaller tax (NIXL doesn't walk the block-pool dict in scheduler.build_connector_meta; the trace-replay NIXL vs plain gap of +38 % is consistent with "only the get_finished

• framework constant" parts, lacking the O(|cache|) component).

What this still doesn't settle

1. Multi-instance coupling: the 8-instance run would need its own cache-size sweep + per-instance step instrumentation. We know the per-instance per-step cost; what we don't know is how that propagates through the cache-aware proxy's routing decisions.
2. Larger |cache| extrapolation: H20 96 GB caps at ~17.5 k blocks at the configured memory. Settings with smaller models (or gpu-memory-utilization ≥ 0.95 on bigger GPUs) reach higher |cache|. The slope is linear in this range, but we have not extrapolated past ~17 k.
3. NIXL slope: NIXL was in the prior microbench's plan but not in this run. Same instrumentation on NIXL would confirm whether NIXL has a different (smaller) slope.

Practical recommendation

The root cause is clearly identifiable: the per-scheduler-step set(self._block_pool.cached_block_hash_to_block._cache.keys()) walk in mooncake_connector.py:434-450. Replacing it with an incremental delta listener (using the block-pool's existing add/remove/evict callbacks) would zero out the cache-size slope and bring mooncake_both into the same ballpark as noop_connector on the scheduler side.

The worker-side get_finished cost (180 μs constant) is also fixable: in kv_both mode it submits two empty coroutine_threadsafe futures every step. Caching/coalescing or short-circuiting when both queues are empty would eliminate this constant.

Reproducibility

cd microbench/connector_tax/cache_sweep
bash run_all.sh           # ~22 min on H20 single-GPU

The orchestrator applies v1 + v2 patches, runs the three configs sequentially, reverts both patches on exit, and produces results/<date>/SUMMARY.md + figure.png.

Artifacts in results/20260526_1507/:

• figure.png — the headline plot
• SUMMARY.md — per-config tables (this report's source)
• per_config.json — machine-readable
• per-config: engine_step.jsonl, worker_step.r0.jsonl, requests.jsonl, metrics_final.txt, vLLM stdout/stderr