Gahow Wang
8829928fc5
Follow-up to Microbench 3 that finally tests H5 (cache-size
dependence) and instruments worker-side connector callbacks the
original patch missed.
Patch v2 (apply_step_timing_v2.py) adds:
scheduler: `cache_size` field in engine_step.jsonl
worker: `get_finished_us` + `start_load_kv_us` in worker_step.r0.jsonl
uses BLOCK_BEGIN/END sentinels for safe multi-line revert
(the original v1 patch survives this v2's apply/revert cycle)
Driver: continuous open-loop (1.5 req/s, 4096x256 random per req)
that lets APC fill from 0 → ceiling within one vLLM lifetime so a
single run produces the full cache_size sweep. Decode-only steps
are filtered post-hoc to remove prefill-mix variance.
Findings (H20 96GB, ceiling reached ~17.5k blocks; n=15-18k decode
steps per config):
config | slope (μs / 1k blocks) | step_dur p50 @ |cache|=16.6k
---------------|------------------------|-----------------------------
mooncake_both | +85.6 | 1528 μs (build_meta=1442, 94%)
noop_connector | -0.8 (≈0) | 79 μs
plain | +1.0 (≈0) | 84 μs
Worker-side get_finished p50/p90/p99 (μs/step):
mooncake_both: 180 / 257 / 333
noop_connector: 0 / 0 / 2
H5 PASSES. mooncake_both step_duration scales linearly with |cache|
because build_connector_meta walks set(cache.keys()) every step
(`mooncake_connector.py:434-450`). plain and noop are flat.
The previously-uninstrumented get_finished() adds a constant
180 μs/step on top — two `run_coroutine_threadsafe(...).result()`
blocking waits in kv_both mode (`mooncake_connector.py:1107-1137`)
fire every step even when no transfer is pending.
Trace-replay reconciliation (APC ≈ 79% → |cache| ≈ 13k blocks):
build_meta @ 13k ≈ 1060 μs + get_finished ≈ 180 μs = 1.24 ms/step
On ~7 ms decode forward → +15-20% TPOT per step.
This explains most of the trace-replay +25% TPOT p90 gap from
single-instance per-step cost alone, leaving a smaller residual
for multi-instance coupling than originally assumed.
Two clear fixes pointed out in REPORT.md:
1. replace O(|cache|) per-step walk with incremental delta
listener using block_pool's add/remove callbacks
2. short-circuit get_finished() when both producer/consumer
queues are empty in kv_both
Heavy raw artifacts (engine_step.jsonl, vllm_stdout/stderr,
.vllm.pid) are .gitignored — they re-derive from `bash run_all.sh`
and SUMMARY.md / per_config.json fully capture the conclusions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
99 lines
4.3 KiB
Markdown
99 lines
4.3 KiB
Markdown
# Cache-size Sweep — testing H5 from connector_tax DESIGN.md
|
||
|
||
## Hypothesis under test
|
||
|
||
**H5**: `MooncakeConnectorScheduler.build_connector_meta()` walks
|
||
`set(self._block_pool.cached_block_hash_to_block._cache.keys())` every
|
||
scheduler step, so `step_duration_us` and `build_meta_us` should
|
||
grow **linearly with |cache|** (= the number of cached block-hash
|
||
entries in the block pool). The +45 % trace-replay tax is hypothesised
|
||
to come from running this O(|cache|) loop at APC ≈ 79 %, which the
|
||
prior microbench never tested (random content → cache stays small).
|
||
|
||
## What we instrument
|
||
|
||
The original `apply_step_timing.py` only recorded
|
||
`step_duration_us` and `build_meta_us`. This sweep adds:
|
||
|
||
| Field | Source | Why |
|
||
|---|---|---|
|
||
| `cache_size` | `len(scheduler.kv_cache_manager.block_pool.cached_block_hash_to_block._cache)` | The exact dict that `set(...)` walks |
|
||
| `get_finished_us` | wraps `kv_connector.get_finished(...)` in worker mixin | The other suspected cost (two `run_coroutine_threadsafe(...).result()` blocking waits for `kv_both`) |
|
||
| `start_load_kv_us` | wraps `kv_connector.start_load_kv(...)` in worker mixin | Mostly fast for `kv_both` w/o transfers, but include for completeness |
|
||
|
||
Scheduler-side fields go to `engine_step.jsonl` (existing channel).
|
||
Worker-side timings go to `worker_step.jsonl` (one file per worker
|
||
process).
|
||
|
||
## Method
|
||
|
||
For each config in {`plain`, `noop_connector`, `mooncake_both`}:
|
||
|
||
1. Launch one fresh vLLM (TP=1, H20, max_model_len=200000,
|
||
gpu-memory-utilization=0.9, enable_prefix_caching).
|
||
2. Read /metrics once to record `kv_cache_max_blocks` (the dict
|
||
ceiling).
|
||
3. Drive an open-loop stream:
|
||
- shape = 4096 in / 256 out
|
||
- rate = 2 req/s (kept below saturation to keep step duration
|
||
dominated by scheduler-not-queueing)
|
||
- content random per request (UUID + hash), zero prefix-cache
|
||
hit ⇒ `|cache|` grows monotonically until hit by LRU eviction
|
||
- duration = until cache fills (≤ 12 min)
|
||
4. Collect `engine_step.jsonl` + `worker_step.jsonl` + the per-request
|
||
metrics from `bench_loop.py`.
|
||
5. Tear down vLLM, wait for GPU release.
|
||
|
||
After all three configs:
|
||
- Apply LWESS-style binning on (`cache_size`, `step_duration_us`) to
|
||
show the curve per config.
|
||
- Compute linear fit per config: `step_duration_us ≈ a + b · cache_size`.
|
||
- Connector-attributable per-step tax at a given |cache|:
|
||
`tax_us(cache_size) = mc_step(cache_size) − plain_step(cache_size)`.
|
||
- Same decomposition for `build_meta_us` (only mooncake / noop have
|
||
non-zero values; plain is 0 by construction).
|
||
- For worker side: `get_finished_us` distribution per config; in
|
||
`kv_both` mode this should be non-zero even when no transfer fires.
|
||
|
||
## What "passes" or "fails" H5
|
||
|
||
- **PASS**: `step_duration_us` (mooncake_both) grows roughly
|
||
linearly with |cache|, with slope **> 5 μs / 1 000 blocks** so that
|
||
at |cache| ≈ 200 k it is ≥ 1 ms of per-step overhead. `plain`
|
||
shows no slope. This matches the source code reading.
|
||
- **FAIL**: no measurable slope, or slope is similar for plain and
|
||
mooncake_both → the O(|cache|) walk is not the actual cost driver
|
||
and we should look elsewhere (e.g. `get_finished` blocking waits,
|
||
delay_free overhead).
|
||
|
||
Either outcome is informative.
|
||
|
||
## What this sweep does *not* answer
|
||
|
||
- Multi-instance coupling (8 schedulers running the walk concurrently
|
||
vs proxy load-balancing).
|
||
- Agentic session structure (long prefix reuse + short uncached tail).
|
||
- The 8-instance trace-replay 45 % figure can only be reconciled
|
||
once we know the slope and combine with concurrency / coupling
|
||
measurements. This sweep is a necessary input, not the full
|
||
reconciliation.
|
||
|
||
## Files
|
||
|
||
```
|
||
cache_sweep/
|
||
├── DESIGN.md # this file
|
||
├── apply_step_timing_v2.py # extends apply_step_timing.py with cache_size + worker timings
|
||
├── run_cache_sweep.py # bench driver: per-config continuous open-loop
|
||
├── analyze.py # join engine_step + worker_step, plot, fit
|
||
├── run_all.sh # orchestrator (apply patch → run 3 configs → revert → analyze)
|
||
└── results/<date>/ # one subdir per run
|
||
└── <config>/
|
||
├── engine_step.jsonl
|
||
├── worker_step.jsonl
|
||
├── requests.jsonl
|
||
├── summary.json
|
||
├── vllm_stdout.log
|
||
└── vllm_stderr.log
|
||
```
|