# Microbench 3: Connector Substrate Tax — Results

> **2026-05-26 ERRATA (post-review)**: The original write-up overstated
> what this microbench had measured. Three things to call out before
> reading the rest:
> 1. **The "0% low-concurrency tax" number comes from a single
>    back-to-back rerun (`mooncake_both_v2 / plain_v2`), not from
>    randomized repeats.** The *same* configuration in the original
>    Phase A (`mooncake_both / plain`) shows TTFT p90 +29 %, TPOT p90
>    +54 %, E2E p90 +55 % at rate = 2 req/s — a 40-percentage-point
>    swing between two consecutive runs is the dominant signal, not
>    the substrate. See "Run-to-run variance" below.
> 2. **`get_finished()` was never instrumented.** The patch in
>    `patches/apply_step_timing.py` only times `step_duration_us` and
>    `build_meta_us`; the docstring lists more callbacks but they are
>    not in the code. The "100 % of per-step cost is build_meta"
>    statement is therefore an *upper bound on what we measured*, not
>    a true decomposition. `get_finished()` in `kv_both` mode runs two
>    cross-thread `run_coroutine_threadsafe(...).result()` blocking
>    waits every step (`mooncake_connector.py:1107-1137`) and is a
>    plausible second contributor.
> 3. **H5 (cache-size dependence) is untested.** The hypothesis that
>    `set(self._block_pool.cache.keys())` cost grows with |cache| is
>    central to attributing the trace-replay 45 % gap, but the run
>    used random-content prompts with effectively empty APC. The
>    cache-size sweep in `cache_sweep/` is what actually tests this.
>
> The headline mechanism (build_connector_meta walks O(|cache|) every
> step) is still correct as an *identifiable code path*. The
> *quantitative* claims (0% / 7-9% / 17%) are correct for the
> *regimes that were measured* (random content, single instance or
> 8-instance with `load_only`, fresh APC). Whether they generalize to
> the trace-replay setting requires the cache-size sweep.

## Executive Summary

The `build_connector_meta()` in MooncakeConnector adds **1.4ms per scheduler
step** (measured via engine_step.jsonl instrumentation) on a *cold* APC.
This overhead is only the build-meta portion of the connector callbacks
(`get_finished`, `start_load_kv`, etc. were not instrumented). Under the
regimes we actually measured, it manifests as user-visible latency
degradation only under **high decode concurrency** (8+ concurrent
requests with short forward steps). Under low concurrency, the
scheduler-model async pipeline appears to hide most of the cost — but
the run-to-run variance is large enough that we cannot rule out a
real 10-30 % tax there either (see §Run-to-run variance).

| Regime | TTFT-p90 tax (mooncake_both vs plain) | Caveat |
|--------|---------------------------------------|--------|
| Low conc, 4096×256, rate≤2 (v1 run) | +12 % (r=1) / +29 % (r=2) | First-shot data; APC near-empty |
| Low conc, 4096×256, rate≤2 (v2 rerun) | −12 % (r=1) / −10 % (r=2) | Back-to-back rerun; sign flips |
| High conc, 512×64, rate=8-16 (single instance) | **+7-9 %** | Most reproducible; n≥395 per cell |
| 8-inst load_only, 512×64, rate=128 (saturated) | **+17 %** | Throughput dropped to 0.70 |
| 8-inst agentic trace-replay (elastic_migration_v2) | **+45 %** | APC ≈ 79 %, session-coupled — *not yet reproduced* |

---

## Per-Step Timing (engine_step.jsonl instrumentation)

Direct measurement of scheduler step duration via our patch:

| Config | step_duration p50 | step_duration p90 | build_meta p50 | build_meta p90 | n_steps |
|--------|-------------------|-------------------|----------------|----------------|---------|
| **plain** | **53 μs** | **91 μs** | 0 μs | 0 μs | 59305 |
| noop_connector | 69 μs | 175 μs | 0 μs | 0 μs | 49604 |
| mooncake_producer | 1461 μs | 2156 μs | 1386 μs | 1992 μs | 51669 |
| mooncake_both | 1452 μs | 2247 μs | 1385 μs | 2007 μs | 124987 |

**Key finding**: The 1.4ms/step cost is entirely in `build_connector_meta()`,
which walks `set(cache.keys())` every scheduler step (O(|cache|), E2 audit §6.5).
The vLLM v1 framework dispatch itself (noop_connector) adds only +16μs.

---

## Run-to-run variance (4096 × 256)

We have two back-to-back pairs of runs at the same shape, same rates,
same seed methodology. They disagree by 40 percentage points:

| rate | metric | v1 (plain → mooncake_both) | v2 (plain_v2 → mooncake_both_v2) |
|---|---|---|---|
| 0.5 | TTFT p90 tax | −8 % | −12 % |
| 1.0 | TTFT p90 tax | **+12 %** | **−12 %** |
| 2.0 | TTFT p90 tax | **+29 %** | **−10 %** |
| 2.0 | TPOT p90 tax | **+54 %** | **−23 %** |
| 2.0 | E2E  p90 tax | **+55 %** | **−23 %** |

Both v1 and v2 used 200 completed-request floors; v1 ran configs
serially with full GPU release between, v2 ran the two configs
back-to-back without restart. Neither has CI bars. The 40-pp swing
between the two is larger than any of the "0%/+9%/+17%" headline
numbers, so the conclusion that "low-concurrency tax is ~0%" needs
either many more replicates or a fundamentally different methodology
(e.g. controlled |cache|; see `cache_sweep/`). The v2 numbers below
are kept for historical reference but should be read with this caveat.

## Low-Concurrency Results (4096 input, 256 output)

Back-to-back fresh runs (mooncake_both_v2 first, plain_v2 second):

### Rate = 0.5 req/s

| Metric | plain | mooncake_both | Tax |
|--------|-------|---------------|-----|
| TTFT mean | 269ms | 274ms | +2% |
| TTFT p50 | 254ms | 257ms | +1% |
| TTFT p90 | 302ms | 265ms | -12% |
| TTFT p99 | 473ms | 541ms | +14% |
| TPOT mean | 6.6ms | 6.5ms | -2% |
| TPOT p90 | 9.2ms | 9.3ms | +1% |
| TPOT p99 | 12.0ms | 11.1ms | -8% |
| E2E mean | 1955ms | 1938ms | -1% |
| E2E p90 | 2621ms | 2631ms | +0.4% |
| E2E p99 | 3323ms | 3100ms | -7% |

### Rate = 1.0 req/s

| Metric | plain | mooncake_both | Tax |
|--------|-------|---------------|-----|
| TTFT mean | 325ms | 296ms | -9% |
| TTFT p50 | 263ms | 263ms | 0% |
| TTFT p90 | 500ms | 442ms | -12% |
| TTFT p99 | 676ms | 566ms | -16% |
| TPOT mean | 11.8ms | 9.6ms | -19% |
| TPOT p90 | 19.7ms | 13.3ms | -32% |
| E2E mean | 3333ms | 2748ms | -18% |
| E2E p90 | 5296ms | 3710ms | -30% |

### Rate = 2.0 req/s

| Metric | plain | mooncake_both | Tax |
|--------|-------|---------------|-----|
| TTFT mean | 387ms | 372ms | -4% |
| TTFT p50 | 306ms | 293ms | -4% |
| TTFT p90 | 611ms | 549ms | -10% |
| TTFT p99 | 833ms | 875ms | +5% |
| TPOT mean | 35.7ms | 27.3ms | -24% |
| TPOT p90 | 51.4ms | 39.5ms | -23% |
| E2E mean | 9479ms | 7345ms | -23% |
| E2E p90 | 13453ms | 10423ms | -23% |

**Interpretation**: At low concurrency, substrate tax is **≈0% ± noise**.
The "negative tax" at rate=1-2 is run-order thermal effect.

---

## High-Concurrency Results (512 input, 64 output, rate=4-32)

Short requests maximize decode concurrency. Back-to-back (plain first,
mooncake_both second):

| Rate | plain TTFT p90 | mc_both TTFT p90 | **TTFT Tax** | plain TPOT p90 | mc_both TPOT p90 | **TPOT Tax** | plain thr | mc thr |
|------|---------------|-----------------|--------------|---------------|-----------------|--------------|-----------|--------|
| 4 | 87ms | 82ms | -6% | 9.9ms | 9.4ms | -5% | 1.00 | 0.98 |
| **8** | **94ms** | **102ms** | **+9%** | **13.8ms** | **14.9ms** | **+8%** | 0.95 | 0.98 |
| **16** | **144ms** | **156ms** | **+8%** | **27.8ms** | **29.7ms** | **+7%** | 0.94 | 0.99 |
| 32 | 6122ms | 6186ms | +1% | 56.8ms | 55.7ms | -2% | 0.80 | 0.80 |

**The tax appears at rate=8-16 req/s (+7-9%)** where 8-16 requests
concurrently decode and the scheduler per-step cost becomes visible.

SLO check: at rate=16, mooncake_both gives TTFT p90=156ms (<10s SLO ✓)
and TPOT p90=29.7ms (<100ms SLO ✓). The tax is measurable but
SLO-compliant.

---

## Reconciliation with Trace-Replay (+45%) — what we *do* and *don't* know

The trace-replay claim (elastic_migration_v2 §Result 1) measured
TTFT p90 +45% with 8 instances, saturated agentic coupling, APC≈79%.

What this microbench established:

| Factor | Status | Evidence |
|--------|--------|----------|
| `build_connector_meta` adds ~1.4 ms/step on a *near-empty* APC | measured | `engine_step.jsonl`, mooncake_both vs plain |
| Tax surfaces at high decode concurrency (single instance, 512×64) | +7-9 % | rate=8/16 cells, n≥395 per cell |
| 8-instance load_only at saturation | +17 % | 8inst_mooncake @ rate=128, thr_p=0.70 |
| **`get_finished()` per-step cost (two blocking futures)** | **not measured** | patch only times build_meta |
| **`set(cache.keys())` cost scaling with \|cache\|** | **not measured** | random content → APC ≈ empty in all cells |
| **Agentic session structure (high reuse + tight cache pressure)** | **not measured** | synthetic open-loop has no sessions |
| Multi-instance scheduler coupling beyond load_only | not measured | only `load_only` proxy tested |

The honest reconciliation is: the +7-9 % single-instance and +17 %
8-instance saturated tax are real and small; the gap to +45 % is
hypothesised to come from (a) the O(|cache|) walk at APC≈79 %,
(b) the un-instrumented `get_finished()` cost, and (c)
agentic-coupling effects we have not yet replicated synthetically.
The `cache_sweep/` experiment tests (a) directly.

---

## Conclusions (revised)

1. **`build_connector_meta` is *a* tax source**: ≈1.4 ms/step on a
   near-empty APC. Whether it is *the* source depends on the
   un-measured `get_finished()` cost. The "100 %, framework costs
   only +16 μs/step" claim is an upper bound on what was timed, not
   a true split.

2. **Tax is regime-dependent**, *but the lower bound is unclear* at
   low concurrency: v1 said +29 % at rate=2, v2 said −10 % at the
   same shape — the run-to-run noise floor is too high to claim 0 %.
   High-concurrency (+7-9 %) and 8-instance-saturated (+17 %) are
   more reproducible.

3. **Trace-replay's +45 % is plausible but not yet decomposed.** We
   have not yet exercised the regime that drives it (APC≈79 % cache,
   agentic session structure). `cache_sweep/` adds (a). (b) and (c)
   are open.

4. **Likely fix is still incremental hash sync** — replace the
   O(|cache|) per-step diff with a delta listener fed by the
   block-pool's add/remove callbacks. Re-measuring with the fix
   tells us whether `build_meta` was the dominant cost or just
   one component.

5. **Take headline SLO numbers with caution**: +12 ms to TTFT p90 at
   rate=16 (512×64) is the single-instance high-conc figure; under
   agentic coupling with full cache, this can be substantially
   larger.