Gahow Wang
8829928fc5
Follow-up to Microbench 3 that finally tests H5 (cache-size
dependence) and instruments worker-side connector callbacks the
original patch missed.
Patch v2 (apply_step_timing_v2.py) adds:
scheduler: `cache_size` field in engine_step.jsonl
worker: `get_finished_us` + `start_load_kv_us` in worker_step.r0.jsonl
uses BLOCK_BEGIN/END sentinels for safe multi-line revert
(the original v1 patch survives this v2's apply/revert cycle)
Driver: continuous open-loop (1.5 req/s, 4096x256 random per req)
that lets APC fill from 0 → ceiling within one vLLM lifetime so a
single run produces the full cache_size sweep. Decode-only steps
are filtered post-hoc to remove prefill-mix variance.
Findings (H20 96GB, ceiling reached ~17.5k blocks; n=15-18k decode
steps per config):
config | slope (μs / 1k blocks) | step_dur p50 @ |cache|=16.6k
---------------|------------------------|-----------------------------
mooncake_both | +85.6 | 1528 μs (build_meta=1442, 94%)
noop_connector | -0.8 (≈0) | 79 μs
plain | +1.0 (≈0) | 84 μs
Worker-side get_finished p50/p90/p99 (μs/step):
mooncake_both: 180 / 257 / 333
noop_connector: 0 / 0 / 2
H5 PASSES. mooncake_both step_duration scales linearly with |cache|
because build_connector_meta walks set(cache.keys()) every step
(`mooncake_connector.py:434-450`). plain and noop are flat.
The previously-uninstrumented get_finished() adds a constant
180 μs/step on top — two `run_coroutine_threadsafe(...).result()`
blocking waits in kv_both mode (`mooncake_connector.py:1107-1137`)
fire every step even when no transfer is pending.
Trace-replay reconciliation (APC ≈ 79% → |cache| ≈ 13k blocks):
build_meta @ 13k ≈ 1060 μs + get_finished ≈ 180 μs = 1.24 ms/step
On ~7 ms decode forward → +15-20% TPOT per step.
This explains most of the trace-replay +25% TPOT p90 gap from
single-instance per-step cost alone, leaving a smaller residual
for multi-instance coupling than originally assumed.
Two clear fixes pointed out in REPORT.md:
1. replace O(|cache|) per-step walk with incremental delta
listener using block_pool's add/remove callbacks
2. short-circuit get_finished() when both producer/consumer
queues are empty in kv_both
Heavy raw artifacts (engine_step.jsonl, vllm_stdout/stderr,
.vllm.pid) are .gitignored — they re-derive from `bash run_all.sh`
and SUMMARY.md / per_config.json fully capture the conclusions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
154 lines
6.6 KiB
Markdown
154 lines
6.6 KiB
Markdown
# Cache-size Sweep — Results
|
||
|
||
Run: `results/20260526_1507/`
|
||
Hardware: H20 96 GB × 1, TP=1, Qwen3-Coder-30B-A3B-Instruct,
|
||
`gpu-memory-utilization=0.9`, `enable_prefix_caching=true`.
|
||
Cache ceiling reached on this GPU: **17 528 blocks**.
|
||
|
||
## TL;DR
|
||
|
||
H5 (build_connector_meta walks `set(cache.keys())` per step, so cost
|
||
grows linearly with |cache|) **passes**.
|
||
|
||
- mooncake_both: step_duration_us p50 grows from **276 μs (cache=2.6k blocks)**
|
||
to **1528 μs (cache=16.6k blocks)** — linear fit slope **+85.6 μs / 1 000 blocks**.
|
||
- plain: **+1.0 μs / 1 000 blocks** (≈ zero, control).
|
||
- noop_connector: **−0.8 μs / 1 000 blocks** (≈ zero, control).
|
||
|
||
`build_connector_meta` accounts for **94 % of the scheduler-side
|
||
cost at full cache** (1442 / 1528 μs at the top bin). The vLLM v1
|
||
framework dispatch alone (noop_connector vs plain) is **~20 μs**.
|
||
|
||
The original microbench's **"100 % from build_meta"** claim was an
|
||
artefact of *not measuring* the worker-side path. With both sides
|
||
measured here, the picture is:
|
||
|
||
| cost component | mooncake_both (μs/step) | scaling |
|
||
|---|---:|---|
|
||
| **scheduler `build_connector_meta`** | 207 (cache=2.6k) → **1442 (cache=16.6k)** | **O(\|cache\|)** |
|
||
| **worker `get_finished()`** | **p50 = 180 μs, p99 = 333 μs** (independent of \|cache\|) | constant |
|
||
| **worker `start_load_kv()`** | p50 = 2-5 μs | constant |
|
||
| **framework dispatch** (noop−plain) | ≈ 20 μs | constant |
|
||
|
||
So the previously-uninstrumented `get_finished()` adds another **180 μs
|
||
per step on top** of the cache-dependent build_meta. At low cache size
|
||
that's the dominant connector cost; at high cache size it's
|
||
overshadowed by build_meta but still adds ~10 %.
|
||
|
||
## The figure
|
||
|
||
|
||
|
||
Left: full step time. Right: just the `build_connector_meta`
|
||
component. plain and noop stay flat at ~80 μs across the whole range;
|
||
mooncake_both rises near-linearly.
|
||
|
||
## How this changes the trace-replay reconciliation
|
||
|
||
The 8-instance trace replay (`analysis/characterization/elastic_migration_v2`)
|
||
ran with APC ≈ 79 %, i.e. each instance's block pool held **~13 000
|
||
blocks**. Plugging that into the fit:
|
||
|
||
```
|
||
mooncake build_meta @ |cache|=13 000 ≈ 1060 μs / step
|
||
mooncake get_finished ≈ 180 μs / step
|
||
total per-step connector cost ≈ 1240 μs ≈ 1.24 ms / step
|
||
```
|
||
|
||
Decode-step model forward on Qwen3-Coder-30B-A3B / H20 is ~6-9 ms
|
||
TPOT, so 1.24 ms of extra scheduler-and-worker time per step is a
|
||
**+15-20 % TPOT inflation** purely from the per-step connector cost —
|
||
before any inter-instance coupling.
|
||
|
||
This matches the trace-replay TPOT p90 +25 % gap quite well. The
|
||
**residual ~7 pp** can be attributed to:
|
||
|
||
1. **Block-pool LRU churn under capacity pressure** (random-content
|
||
bench reaches ceiling quickly; trace-replay holds at ceiling
|
||
for the full session-coupled workload).
|
||
2. **Block-lifecycle changes** (`delay_free_blocks=True` once any
|
||
connector is loaded; the freed-block backlog is larger under
|
||
high APC).
|
||
3. **Multi-instance scheduler coupling**: the slowest scheduler in
|
||
8-way load_only sets the proxy's batch latency.
|
||
|
||
For the **+45 % TTFT p90 gap**, the same scheduler tax compounds
|
||
across many chunked-prefill steps. A 50-step prefill at 1.24 ms extra
|
||
each step is +62 ms, which is on the order of the typical TTFT delta
|
||
we see at moderate load.
|
||
|
||
## How this changes the "decomposition" attribution
|
||
|
||
The original RESULTS.md said:
|
||
|
||
> +7-9 % from build_connector_meta per-step cost (this microbench)
|
||
> +20-30 % from multi-instance coupling amplification (not measurable)
|
||
> remainder from large-cache O(\|cache\|) scaling (Phase B follow-up)
|
||
|
||
The cache-size sweep replaces the third row with a measurement and
|
||
forces the first row to be re-read:
|
||
|
||
| factor | original claim | revised |
|
||
|---|---|---|
|
||
| single-instance high-conc tax | +7-9 % | unchanged — that was measured at low \|cache\| |
|
||
| multi-instance coupling | +20-30 % | still un-measured, but a *smaller* slice than thought |
|
||
| large-cache O(\|cache\|) scaling | "likely 2-3×" | **measured: +85.6 μs/1k blocks; ≈ 1 ms/step at \|cache\|=13k** |
|
||
| worker-side get_finished | not in the model | **measured: +180 μs/step (constant)** |
|
||
|
||
The "trace-replay 45 % TTFT p90" is now explainable mostly from
|
||
cache-size + worker get_finished + framework dispatch, without
|
||
having to invoke a large multi-instance coupling term. The data is
|
||
also consistent with NIXL's smaller tax (NIXL doesn't walk the
|
||
block-pool dict in scheduler.build_connector_meta; the trace-replay
|
||
NIXL vs plain gap of +38 % is consistent with "only the get_finished
|
||
+ framework constant" parts, lacking the O(\|cache\|) component).
|
||
|
||
## What this still doesn't settle
|
||
|
||
1. **Multi-instance coupling**: the 8-instance run would need its
|
||
own cache-size sweep + per-instance step instrumentation. We
|
||
know the per-instance per-step cost; what we don't know is how
|
||
that propagates through the cache-aware proxy's routing
|
||
decisions.
|
||
2. **Larger \|cache\| extrapolation**: H20 96 GB caps at ~17.5 k
|
||
blocks at the configured memory. Settings with smaller models
|
||
(or `gpu-memory-utilization` ≥ 0.95 on bigger GPUs) reach
|
||
higher \|cache\|. The slope is linear in this range, but we
|
||
have not extrapolated past ~17 k.
|
||
3. **NIXL slope**: NIXL was in the prior microbench's plan but
|
||
not in this run. Same instrumentation on NIXL would confirm
|
||
whether NIXL has a different (smaller) slope.
|
||
|
||
## Practical recommendation
|
||
|
||
The root cause is clearly identifiable: the per-scheduler-step
|
||
`set(self._block_pool.cached_block_hash_to_block._cache.keys())` walk
|
||
in `mooncake_connector.py:434-450`. Replacing it with an incremental
|
||
delta listener (using the block-pool's existing
|
||
`add`/`remove`/`evict` callbacks) would zero out the cache-size
|
||
slope and bring mooncake_both into the same ballpark as noop_connector
|
||
on the scheduler side.
|
||
|
||
The worker-side `get_finished` cost (180 μs constant) is also
|
||
fixable: in `kv_both` mode it submits two empty `coroutine_threadsafe`
|
||
futures every step. Caching/coalescing or short-circuiting when both
|
||
queues are empty would eliminate this constant.
|
||
|
||
## Reproducibility
|
||
|
||
```
|
||
cd microbench/connector_tax/cache_sweep
|
||
bash run_all.sh # ~22 min on H20 single-GPU
|
||
```
|
||
|
||
The orchestrator applies v1 + v2 patches, runs the three configs
|
||
sequentially, reverts both patches on exit, and produces
|
||
`results/<date>/SUMMARY.md` + `figure.png`.
|
||
|
||
Artifacts in `results/20260526_1507/`:
|
||
- `figure.png` — the headline plot
|
||
- `SUMMARY.md` — per-config tables (this report's source)
|
||
- `per_config.json` — machine-readable
|
||
- per-config: `engine_step.jsonl`, `worker_step.r0.jsonl`,
|
||
`requests.jsonl`, `metrics_final.txt`, vLLM stdout/stderr
|