Cache-size sweep: build_meta is O(|cache|), +85.6 μs / 1k blocks

Follow-up to Microbench 3 that finally tests H5 (cache-size
dependence) and instruments worker-side connector callbacks the
original patch missed.

Patch v2 (apply_step_timing_v2.py) adds:
  scheduler: `cache_size` field in engine_step.jsonl
  worker:    `get_finished_us` + `start_load_kv_us` in worker_step.r0.jsonl
  uses BLOCK_BEGIN/END sentinels for safe multi-line revert
  (the original v1 patch survives this v2's apply/revert cycle)

Driver: continuous open-loop (1.5 req/s, 4096x256 random per req)
that lets APC fill from 0 → ceiling within one vLLM lifetime so a
single run produces the full cache_size sweep. Decode-only steps
are filtered post-hoc to remove prefill-mix variance.

Findings (H20 96GB, ceiling reached ~17.5k blocks; n=15-18k decode
steps per config):

  config         | slope (μs / 1k blocks) | step_dur p50 @ |cache|=16.6k
  ---------------|------------------------|-----------------------------
  mooncake_both  | +85.6                  | 1528 μs (build_meta=1442, 94%)
  noop_connector | -0.8 (≈0)              |  79 μs
  plain          | +1.0 (≈0)              |  84 μs

  Worker-side get_finished p50/p90/p99 (μs/step):
    mooncake_both:  180 / 257 / 333
    noop_connector:   0 /   0 /   2

H5 PASSES. mooncake_both step_duration scales linearly with |cache|
because build_connector_meta walks set(cache.keys()) every step
(`mooncake_connector.py:434-450`). plain and noop are flat.

The previously-uninstrumented get_finished() adds a constant
180 μs/step on top — two `run_coroutine_threadsafe(...).result()`
blocking waits in kv_both mode (`mooncake_connector.py:1107-1137`)
fire every step even when no transfer is pending.

Trace-replay reconciliation (APC ≈ 79% → |cache| ≈ 13k blocks):
  build_meta @ 13k ≈ 1060 μs + get_finished ≈ 180 μs = 1.24 ms/step
  On ~7 ms decode forward → +15-20% TPOT per step.
  This explains most of the trace-replay +25% TPOT p90 gap from
  single-instance per-step cost alone, leaving a smaller residual
  for multi-instance coupling than originally assumed.

Two clear fixes pointed out in REPORT.md:
  1. replace O(|cache|) per-step walk with incremental delta
     listener using block_pool's add/remove callbacks
  2. short-circuit get_finished() when both producer/consumer
     queues are empty in kv_both

Heavy raw artifacts (engine_step.jsonl, vllm_stdout/stderr,
.vllm.pid) are .gitignored — they re-derive from `bash run_all.sh`
and SUMMARY.md / per_config.json fully capture the conclusions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

microbench/connector_tax/cache_sweep/DESIGN.md

@@ -0,0 +1,98 @@
+# Cache-size Sweep — testing H5 from connector_tax DESIGN.md
+
+## Hypothesis under test
+
+**H5**: `MooncakeConnectorScheduler.build_connector_meta()` walks
+`set(self._block_pool.cached_block_hash_to_block._cache.keys())` every
+scheduler step, so `step_duration_us` and `build_meta_us` should
+grow **linearly with |cache|** (= the number of cached block-hash
+entries in the block pool). The +45 % trace-replay tax is hypothesised
+to come from running this O(|cache|) loop at APC ≈ 79 %, which the
+prior microbench never tested (random content → cache stays small).
+
+## What we instrument
+
+The original `apply_step_timing.py` only recorded
+`step_duration_us` and `build_meta_us`. This sweep adds:
+
+| Field | Source | Why |
+|---|---|---|
+| `cache_size` | `len(scheduler.kv_cache_manager.block_pool.cached_block_hash_to_block._cache)` | The exact dict that `set(...)` walks |
+| `get_finished_us` | wraps `kv_connector.get_finished(...)` in worker mixin | The other suspected cost (two `run_coroutine_threadsafe(...).result()` blocking waits for `kv_both`) |
+| `start_load_kv_us` | wraps `kv_connector.start_load_kv(...)` in worker mixin | Mostly fast for `kv_both` w/o transfers, but include for completeness |
+
+Scheduler-side fields go to `engine_step.jsonl` (existing channel).
+Worker-side timings go to `worker_step.jsonl` (one file per worker
+process).
+
+## Method
+
+For each config in {`plain`, `noop_connector`, `mooncake_both`}:
+
+1. Launch one fresh vLLM (TP=1, H20, max_model_len=200000,
+   gpu-memory-utilization=0.9, enable_prefix_caching).
+2. Read /metrics once to record `kv_cache_max_blocks` (the dict
+   ceiling).
+3. Drive an open-loop stream:
+   - shape = 4096 in / 256 out
+   - rate = 2 req/s (kept below saturation to keep step duration
+     dominated by scheduler-not-queueing)
+   - content random per request (UUID + hash), zero prefix-cache
+     hit ⇒ `|cache|` grows monotonically until hit by LRU eviction
+   - duration = until cache fills (≤ 12 min)
+4. Collect `engine_step.jsonl` + `worker_step.jsonl` + the per-request
+   metrics from `bench_loop.py`.
+5. Tear down vLLM, wait for GPU release.
+
+After all three configs:
+- Apply LWESS-style binning on (`cache_size`, `step_duration_us`) to
+  show the curve per config.
+- Compute linear fit per config: `step_duration_us ≈ a + b · cache_size`.
+- Connector-attributable per-step tax at a given |cache|:
+  `tax_us(cache_size) = mc_step(cache_size) − plain_step(cache_size)`.
+- Same decomposition for `build_meta_us` (only mooncake / noop have
+  non-zero values; plain is 0 by construction).
+- For worker side: `get_finished_us` distribution per config; in
+  `kv_both` mode this should be non-zero even when no transfer fires.
+
+## What "passes" or "fails" H5
+
+- **PASS**: `step_duration_us` (mooncake_both) grows roughly
+  linearly with |cache|, with slope **> 5 μs / 1 000 blocks** so that
+  at |cache| ≈ 200 k it is ≥ 1 ms of per-step overhead. `plain`
+  shows no slope. This matches the source code reading.
+- **FAIL**: no measurable slope, or slope is similar for plain and
+  mooncake_both → the O(|cache|) walk is not the actual cost driver
+  and we should look elsewhere (e.g. `get_finished` blocking waits,
+  delay_free overhead).
+
+Either outcome is informative.
+
+## What this sweep does *not* answer
+
+- Multi-instance coupling (8 schedulers running the walk concurrently
+  vs proxy load-balancing).
+- Agentic session structure (long prefix reuse + short uncached tail).
+- The 8-instance trace-replay 45 % figure can only be reconciled
+  once we know the slope and combine with concurrency / coupling
+  measurements. This sweep is a necessary input, not the full
+  reconciliation.
+
+## Files
+
+```
+cache_sweep/
+├── DESIGN.md                # this file
+├── apply_step_timing_v2.py  # extends apply_step_timing.py with cache_size + worker timings
+├── run_cache_sweep.py       # bench driver: per-config continuous open-loop
+├── analyze.py               # join engine_step + worker_step, plot, fit
+├── run_all.sh               # orchestrator (apply patch → run 3 configs → revert → analyze)
+└── results/<date>/          # one subdir per run
+    └── <config>/
+        ├── engine_step.jsonl
+        ├── worker_step.jsonl
+        ├── requests.jsonl
+        ├── summary.json
+        ├── vllm_stdout.log
+        └── vllm_stderr.log
+```