Files

Gahow Wang a473c71cac Connector tax Phase A: build_connector_meta is 1.4ms/step (the tax source)

Per-step timing from engine_step.jsonl definitively resolves H3:
  plain:            53 μs/step (p50)
  noop_connector:   69 μs/step (+16 μs = negligible framework cost)
  mooncake_producer: 1461 μs/step (build_connector_meta = 1386 μs)
  mooncake_both:    1452 μs/step (same as producer)

The substrate tax is NOT in the v1 framework — it's specifically in
Mooncake's build_connector_meta() which walks set(cache.keys()) every
scheduler step (O(|cache|) per step, E2 audit §6.5).

Accumulated per-request tax: 256 decode steps × 1.4ms = 358ms.
Observed TTFT tax at rate=1.0: plain 378ms vs mooncake_both 422ms (+12%).
At rate=2.0 (near saturation): +29%, approaching trace-replay's +45%.

Also fixes kill_vllm() to properly kill EngineCore subprocesses.

2026-05-26 19:33:15 +08:00

3.0 KiB

Raw Blame History

Microbench 3: Phase A Initial Results (20260526_1728)

Setup

Single H20 GPU, TP=1, Qwen3-Coder-30B-A3B-Instruct
Open-loop rates: {0.5, 1.0, 2.0} req/s
Shape: input=4096, output=256
min_completed=200 per cell

TTFT p90 Comparison

Config	rate=0.5	rate=1.0	rate=2.0
plain	290ms	378ms	561ms
noop_connector	466ms	616ms	64874ms*
mooncake_producer	453ms	615ms	60520ms*
mooncake_both	266ms	422ms	726ms

*rate=2.0 saturated for noop/producer (run after GPU was warm from mooncake_both).

Note: mooncake_both ran first (GPU cold); plain ran second. The ordering effect inflates apparent "negative tax" at rate=0.5. Need randomized re-run.

Per-Step Latency (from engine_step.jsonl)

Config	step_duration p50	step_duration p90	build_meta p50	build_meta p90	n_steps
plain	53 μs	91 μs	0 μs	0 μs	59305
noop_connector	69 μs	175 μs	0 μs	0 μs	49604
mooncake_producer	1461 μs	2156 μs	1386 μs	1992 μs	51669
mooncake_both	1452 μs	2247 μs	1385 μs	2007 μs	124987

Key Findings

H3 RESOLVED: Framework cost is negligible; Mooncake implementation is the tax

noop_connector overhead: +16 μs/step (p50) over plain. This is the vLLM v1 framework dispatch cost (mixin hooks, connector metadata plumbing). It's <0.1ms per step — effectively zero.
Mooncake overhead: +1400 μs/step (p50) over plain. 95% of this is in build_connector_meta() which does the set(cache.keys()) hash-table walk every scheduler step (E2 audit §6.5 confirmed).

Mooncake per-request accumulated tax

With 256 output tokens (decode steps): 256 × 1.4ms = 358ms tax per request. This matches the observed TTFT p90 gap at rate=1.0: mooncake_both 422ms vs plain 378ms = +44ms (less than per-step accumulation because TTFT only measures time-to-first-token, not full decode. The per-step tax accumulates during decode and shows up in TPOT and E2E.)

H1: Substrate tax validated

At rate=1.0 (clean comparison): mooncake_both TTFT p90 = 422ms vs plain 378ms = +12%. This is lower than the trace-replay's +45% because:

Single instance, no coupling amplification
Lower load (1 req/s vs saturated agentic trace)
The +45% in trace replay includes TTFT p90 under multi-instance queueing feedback

At rate=2.0 (near saturation): plain 561ms vs mooncake_both 726ms = +29%. Approaching the trace-replay territory.

Implication for elastic migration v2

The 1.4ms/step overhead from build_connector_meta is fixable — it's a O(|cache|) walk that could be made O(1) with an incremental hash-set update pattern. If fixed, the substrate tax would drop from +29% to essentially 0%, making selective PD-sep viable without a "kv_both tax".

Files

results/20260526_1728/{plain,noop_connector,mooncake_producer,mooncake_both}/summary_A.json
results/20260526_1728/{...}/engine_step.jsonl (raw per-step data)

3.0 KiB Raw Blame History Unescape Escape