agentic-kvc/microbench/connector_tax/cache_sweep/DESIGN.md

Cache-size Sweep — testing H5 from connector_tax DESIGN.md

Hypothesis under test

H5: MooncakeConnectorScheduler.build_connector_meta() walks set(self._block_pool.cached_block_hash_to_block._cache.keys()) every scheduler step, so step_duration_us and build_meta_us should grow linearly with |cache| (= the number of cached block-hash entries in the block pool). The +45 % trace-replay tax is hypothesised to come from running this O(|cache|) loop at APC ≈ 79 %, which the prior microbench never tested (random content → cache stays small).

What we instrument

The original apply_step_timing.py only recorded step_duration_us and build_meta_us. This sweep adds:

Field	Source	Why
cache_size	len(scheduler.kv_cache_manager.block_pool.cached_block_hash_to_block._cache)	The exact dict that set(...) walks
get_finished_us	wraps kv_connector.get_finished(...) in worker mixin	The other suspected cost (two run_coroutine_threadsafe(...).result() blocking waits for kv_both)
start_load_kv_us	wraps kv_connector.start_load_kv(...) in worker mixin	Mostly fast for kv_both w/o transfers, but include for completeness

Scheduler-side fields go to engine_step.jsonl (existing channel). Worker-side timings go to worker_step.jsonl (one file per worker process).

Method

For each config in {plain, noop_connector, mooncake_both}:

1. Launch one fresh vLLM (TP=1, H20, max_model_len=200000, gpu-memory-utilization=0.9, enable_prefix_caching).
2. Read /metrics once to record kv_cache_max_blocks (the dict ceiling).
3. Drive an open-loop stream:
   • shape = 4096 in / 256 out
   • rate = 2 req/s (kept below saturation to keep step duration dominated by scheduler-not-queueing)
   • content random per request (UUID + hash), zero prefix-cache hit ⇒ |cache| grows monotonically until hit by LRU eviction
   • duration = until cache fills (≤ 12 min)
4. Collect engine_step.jsonl + worker_step.jsonl + the per-request metrics from bench_loop.py.
5. Tear down vLLM, wait for GPU release.

After all three configs:

• Apply LWESS-style binning on (cache_size, step_duration_us) to show the curve per config.
• Compute linear fit per config: step_duration_us ≈ a + b · cache_size.
• Connector-attributable per-step tax at a given |cache|: tax_us(cache_size) = mc_step(cache_size) − plain_step(cache_size).
• Same decomposition for build_meta_us (only mooncake / noop have non-zero values; plain is 0 by construction).
• For worker side: get_finished_us distribution per config; in kv_both mode this should be non-zero even when no transfer fires.

What "passes" or "fails" H5

• PASS: step_duration_us (mooncake_both) grows roughly linearly with |cache|, with slope > 5 μs / 1 000 blocks so that at |cache| ≈ 200 k it is ≥ 1 ms of per-step overhead. plain shows no slope. This matches the source code reading.
• FAIL: no measurable slope, or slope is similar for plain and mooncake_both → the O(|cache|) walk is not the actual cost driver and we should look elsewhere (e.g. get_finished blocking waits, delay_free overhead).

Either outcome is informative.

What this sweep does not answer

• Multi-instance coupling (8 schedulers running the walk concurrently vs proxy load-balancing).
• Agentic session structure (long prefix reuse + short uncached tail).
• The 8-instance trace-replay 45 % figure can only be reconciled once we know the slope and combine with concurrency / coupling measurements. This sweep is a necessary input, not the full reconciliation.

Files

cache_sweep/
├── DESIGN.md                # this file
├── apply_step_timing_v2.py  # extends apply_step_timing.py with cache_size + worker timings
├── run_cache_sweep.py       # bench driver: per-config continuous open-loop
├── analyze.py               # join engine_step + worker_step, plot, fit
├── run_all.sh               # orchestrator (apply patch → run 3 configs → revert → analyze)
└── results/<date>/          # one subdir per run
    └── <config>/
        ├── engine_step.jsonl
        ├── worker_step.jsonl
        ├── requests.jsonl
        ├── summary.json
        ├── vllm_stdout.log
        └── vllm_stderr.log