agentic-kvc/microbench/connector_tax/results/RESULTS.md

Microbench 3: Connector Substrate Tax — Results

2026-05-26 ERRATA (post-review): The original write-up overstated what this microbench had measured. Three things to call out before reading the rest:

1. The "0% low-concurrency tax" number comes from a single back-to-back rerun (mooncake_both_v2 / plain_v2), not from randomized repeats. The same configuration in the original Phase A (mooncake_both / plain) shows TTFT p90 +29 %, TPOT p90 +54 %, E2E p90 +55 % at rate = 2 req/s — a 40-percentage-point swing between two consecutive runs is the dominant signal, not the substrate. See "Run-to-run variance" below.
2. get_finished() was never instrumented. The patch in patches/apply_step_timing.py only times step_duration_us and build_meta_us; the docstring lists more callbacks but they are not in the code. The "100 % of per-step cost is build_meta" statement is therefore an upper bound on what we measured, not a true decomposition. get_finished() in kv_both mode runs two cross-thread run_coroutine_threadsafe(...).result() blocking waits every step (mooncake_connector.py:1107-1137) and is a plausible second contributor.
3. H5 (cache-size dependence) is untested. The hypothesis that set(self._block_pool.cache.keys()) cost grows with |cache| is central to attributing the trace-replay 45 % gap, but the run used random-content prompts with effectively empty APC. The cache-size sweep in cache_sweep/ is what actually tests this.

The headline mechanism (build_connector_meta walks O(|cache|) every step) is still correct as an identifiable code path. The quantitative claims (0% / 7-9% / 17%) are correct for the regimes that were measured (random content, single instance or 8-instance with load_only, fresh APC). Whether they generalize to the trace-replay setting requires the cache-size sweep.

Executive Summary

The build_connector_meta() in MooncakeConnector adds 1.4ms per scheduler step (measured via engine_step.jsonl instrumentation) on a cold APC. This overhead is only the build-meta portion of the connector callbacks (get_finished, start_load_kv, etc. were not instrumented). Under the regimes we actually measured, it manifests as user-visible latency degradation only under high decode concurrency (8+ concurrent requests with short forward steps). Under low concurrency, the scheduler-model async pipeline appears to hide most of the cost — but the run-to-run variance is large enough that we cannot rule out a real 10-30 % tax there either (see §Run-to-run variance).

Regime	TTFT-p90 tax (mooncake_both vs plain)	Caveat
Low conc, 4096×256, rate≤2 (v1 run)	+12 % (r=1) / +29 % (r=2)	First-shot data; APC near-empty
Low conc, 4096×256, rate≤2 (v2 rerun)	−12 % (r=1) / −10 % (r=2)	Back-to-back rerun; sign flips
High conc, 512×64, rate=8-16 (single instance)	+7-9 %	Most reproducible; n≥395 per cell
8-inst load_only, 512×64, rate=128 (saturated)	+17 %	Throughput dropped to 0.70
8-inst agentic trace-replay (elastic_migration_v2)	+45 %	APC ≈ 79 %, session-coupled — not yet reproduced

---

Per-Step Timing (engine_step.jsonl instrumentation)

Direct measurement of scheduler step duration via our patch:

Config	step_duration p50	step_duration p90	build_meta p50	build_meta p90	n_steps
plain	53 μs	91 μs	0 μs	0 μs	59305
noop_connector	69 μs	175 μs	0 μs	0 μs	49604
mooncake_producer	1461 μs	2156 μs	1386 μs	1992 μs	51669
mooncake_both	1452 μs	2247 μs	1385 μs	2007 μs	124987

Key finding: The 1.4ms/step cost is entirely in build_connector_meta(), which walks set(cache.keys()) every scheduler step (O(|cache|), E2 audit §6.5). The vLLM v1 framework dispatch itself (noop_connector) adds only +16μs.

---

Run-to-run variance (4096 × 256)

We have two back-to-back pairs of runs at the same shape, same rates, same seed methodology. They disagree by 40 percentage points:

rate	metric	v1 (plain → mooncake_both)	v2 (plain_v2 → mooncake_both_v2)
0.5	TTFT p90 tax	−8 %	−12 %
1.0	TTFT p90 tax	+12 %	−12 %
2.0	TTFT p90 tax	+29 %	−10 %
2.0	TPOT p90 tax	+54 %	−23 %
2.0	E2E p90 tax	+55 %	−23 %

Both v1 and v2 used 200 completed-request floors; v1 ran configs serially with full GPU release between, v2 ran the two configs back-to-back without restart. Neither has CI bars. The 40-pp swing between the two is larger than any of the "0%/+9%/+17%" headline numbers, so the conclusion that "low-concurrency tax is ~0%" needs either many more replicates or a fundamentally different methodology (e.g. controlled |cache|; see cache_sweep/). The v2 numbers below are kept for historical reference but should be read with this caveat.

Low-Concurrency Results (4096 input, 256 output)

Back-to-back fresh runs (mooncake_both_v2 first, plain_v2 second):

Rate = 0.5 req/s

Metric	plain	mooncake_both	Tax
TTFT mean	269ms	274ms	+2%
TTFT p50	254ms	257ms	+1%
TTFT p90	302ms	265ms	-12%
TTFT p99	473ms	541ms	+14%
TPOT mean	6.6ms	6.5ms	-2%
TPOT p90	9.2ms	9.3ms	+1%
TPOT p99	12.0ms	11.1ms	-8%
E2E mean	1955ms	1938ms	-1%
E2E p90	2621ms	2631ms	+0.4%
E2E p99	3323ms	3100ms	-7%

Rate = 1.0 req/s

Metric	plain	mooncake_both	Tax
TTFT mean	325ms	296ms	-9%
TTFT p50	263ms	263ms	0%
TTFT p90	500ms	442ms	-12%
TTFT p99	676ms	566ms	-16%
TPOT mean	11.8ms	9.6ms	-19%
TPOT p90	19.7ms	13.3ms	-32%
E2E mean	3333ms	2748ms	-18%
E2E p90	5296ms	3710ms	-30%

Rate = 2.0 req/s

Metric	plain	mooncake_both	Tax
TTFT mean	387ms	372ms	-4%
TTFT p50	306ms	293ms	-4%
TTFT p90	611ms	549ms	-10%
TTFT p99	833ms	875ms	+5%
TPOT mean	35.7ms	27.3ms	-24%
TPOT p90	51.4ms	39.5ms	-23%
E2E mean	9479ms	7345ms	-23%
E2E p90	13453ms	10423ms	-23%

Interpretation: At low concurrency, substrate tax is ≈0% ± noise. The "negative tax" at rate=1-2 is run-order thermal effect.

---

High-Concurrency Results (512 input, 64 output, rate=4-32)

Short requests maximize decode concurrency. Back-to-back (plain first, mooncake_both second):

Rate	plain TTFT p90	mc_both TTFT p90	TTFT Tax	plain TPOT p90	mc_both TPOT p90	TPOT Tax	plain thr	mc thr
4	87ms	82ms	-6%	9.9ms	9.4ms	-5%	1.00	0.98
8	94ms	102ms	+9%	13.8ms	14.9ms	+8%	0.95	0.98
16	144ms	156ms	+8%	27.8ms	29.7ms	+7%	0.94	0.99
32	6122ms	6186ms	+1%	56.8ms	55.7ms	-2%	0.80	0.80

The tax appears at rate=8-16 req/s (+7-9%) where 8-16 requests concurrently decode and the scheduler per-step cost becomes visible.

SLO check: at rate=16, mooncake_both gives TTFT p90=156ms (<10s SLO ✓) and TPOT p90=29.7ms (<100ms SLO ✓). The tax is measurable but SLO-compliant.

---

Reconciliation with Trace-Replay (+45%) — what we do and don't know

The trace-replay claim (elastic_migration_v2 §Result 1) measured TTFT p90 +45% with 8 instances, saturated agentic coupling, APC≈79%.

What this microbench established:

Factor	Status	Evidence
build_connector_meta adds ~1.4 ms/step on a near-empty APC	measured	engine_step.jsonl, mooncake_both vs plain
Tax surfaces at high decode concurrency (single instance, 512×64)	+7-9 %	rate=8/16 cells, n≥395 per cell
8-instance load_only at saturation	+17 %	8inst_mooncake @ rate=128, thr_p=0.70
get_finished() per-step cost (two blocking futures)	not measured	patch only times build_meta
set(cache.keys()) cost scaling with |cache|	not measured	random content → APC ≈ empty in all cells
Agentic session structure (high reuse + tight cache pressure)	not measured	synthetic open-loop has no sessions
Multi-instance scheduler coupling beyond load_only	not measured	only load_only proxy tested

The honest reconciliation is: the +7-9 % single-instance and +17 % 8-instance saturated tax are real and small; the gap to +45 % is hypothesised to come from (a) the O(|cache|) walk at APC≈79 %, (b) the un-instrumented get_finished() cost, and (c) agentic-coupling effects we have not yet replicated synthetically. The cache_sweep/ experiment tests (a) directly.

---

Conclusions (revised)

1. build_connector_meta is a tax source: ≈1.4 ms/step on a near-empty APC. Whether it is the source depends on the un-measured get_finished() cost. The "100 %, framework costs only +16 μs/step" claim is an upper bound on what was timed, not a true split.

2. Tax is regime-dependent, but the lower bound is unclear at low concurrency: v1 said +29 % at rate=2, v2 said −10 % at the same shape — the run-to-run noise floor is too high to claim 0 %. High-concurrency (+7-9 %) and 8-instance-saturated (+17 %) are more reproducible.

3. Trace-replay's +45 % is plausible but not yet decomposed. We have not yet exercised the regime that drives it (APC≈79 % cache, agentic session structure). cache_sweep/ adds (a). (b) and (c) are open.

4. Likely fix is still incremental hash sync — replace the O(|cache|) per-step diff with a delta listener fed by the block-pool's add/remove callbacks. Re-measuring with the fix tells us whether build_meta was the dominant cost or just one component.

5. Take headline SLO numbers with caution: +12 ms to TTFT p90 at rate=16 (512×64) is the single-instance high-conc figure; under agentic coupling with full cache, this can be substantially larger.