# Cache-size Sweep — testing H5 from connector_tax DESIGN.md

## Hypothesis under test

**H5**: `MooncakeConnectorScheduler.build_connector_meta()` walks
`set(self._block_pool.cached_block_hash_to_block._cache.keys())` every
scheduler step, so `step_duration_us` and `build_meta_us` should
grow **linearly with |cache|** (= the number of cached block-hash
entries in the block pool). The +45 % trace-replay tax is hypothesised
to come from running this O(|cache|) loop at APC ≈ 79 %, which the
prior microbench never tested (random content → cache stays small).

## What we instrument

The original `apply_step_timing.py` only recorded
`step_duration_us` and `build_meta_us`. This sweep adds:

| Field | Source | Why |
|---|---|---|
| `cache_size` | `len(scheduler.kv_cache_manager.block_pool.cached_block_hash_to_block._cache)` | The exact dict that `set(...)` walks |
| `get_finished_us` | wraps `kv_connector.get_finished(...)` in worker mixin | The other suspected cost (two `run_coroutine_threadsafe(...).result()` blocking waits for `kv_both`) |
| `start_load_kv_us` | wraps `kv_connector.start_load_kv(...)` in worker mixin | Mostly fast for `kv_both` w/o transfers, but include for completeness |

Scheduler-side fields go to `engine_step.jsonl` (existing channel).
Worker-side timings go to `worker_step.jsonl` (one file per worker
process).

## Method

For each config in {`plain`, `noop_connector`, `mooncake_both`}:

1. Launch one fresh vLLM (TP=1, H20, max_model_len=200000,
   gpu-memory-utilization=0.9, enable_prefix_caching).
2. Read /metrics once to record `kv_cache_max_blocks` (the dict
   ceiling).
3. Drive an open-loop stream:
   - shape = 4096 in / 256 out
   - rate = 2 req/s (kept below saturation to keep step duration
     dominated by scheduler-not-queueing)
   - content random per request (UUID + hash), zero prefix-cache
     hit ⇒ `|cache|` grows monotonically until hit by LRU eviction
   - duration = until cache fills (≤ 12 min)
4. Collect `engine_step.jsonl` + `worker_step.jsonl` + the per-request
   metrics from `bench_loop.py`.
5. Tear down vLLM, wait for GPU release.

After all three configs:
- Apply LWESS-style binning on (`cache_size`, `step_duration_us`) to
  show the curve per config.
- Compute linear fit per config: `step_duration_us ≈ a + b · cache_size`.
- Connector-attributable per-step tax at a given |cache|:
  `tax_us(cache_size) = mc_step(cache_size) − plain_step(cache_size)`.
- Same decomposition for `build_meta_us` (only mooncake / noop have
  non-zero values; plain is 0 by construction).
- For worker side: `get_finished_us` distribution per config; in
  `kv_both` mode this should be non-zero even when no transfer fires.

## What "passes" or "fails" H5

- **PASS**: `step_duration_us` (mooncake_both) grows roughly
  linearly with |cache|, with slope **> 5 μs / 1 000 blocks** so that
  at |cache| ≈ 200 k it is ≥ 1 ms of per-step overhead. `plain`
  shows no slope. This matches the source code reading.
- **FAIL**: no measurable slope, or slope is similar for plain and
  mooncake_both → the O(|cache|) walk is not the actual cost driver
  and we should look elsewhere (e.g. `get_finished` blocking waits,
  delay_free overhead).

Either outcome is informative.

## What this sweep does *not* answer

- Multi-instance coupling (8 schedulers running the walk concurrently
  vs proxy load-balancing).
- Agentic session structure (long prefix reuse + short uncached tail).
- The 8-instance trace-replay 45 % figure can only be reconciled
  once we know the slope and combine with concurrency / coupling
  measurements. This sweep is a necessary input, not the full
  reconciliation.

## Files

```
cache_sweep/
├── DESIGN.md                # this file
├── apply_step_timing_v2.py  # extends apply_step_timing.py with cache_size + worker timings
├── run_cache_sweep.py       # bench driver: per-config continuous open-loop
├── analyze.py               # join engine_step + worker_step, plot, fit
├── run_all.sh               # orchestrator (apply patch → run 3 configs → revert → analyze)
└── results/<date>/          # one subdir per run
    └── <config>/
        ├── engine_step.jsonl
        ├── worker_step.jsonl
        ├── requests.jsonl
        ├── summary.json
        ├── vllm_stdout.log
        └── vllm_stderr.log
```