Commit Graph

3 Commits

Author SHA1 Message Date
54de78eb11 Connector tax RESULTS.md: errata + run-to-run variance disclosure
The prior write-up presented one specific reading of the data as
the headline without flagging methodology gaps. Three corrections:

1. The "0% low-concurrency tax" comes from a single back-to-back
   mooncake_both_v2/plain_v2 rerun. The original Phase A pair
   showed TTFT p90 +29%, TPOT p90 +54%, E2E p90 +55% at rate=2
   — a 40 percentage-point swing between two consecutive runs
   that the original write-up did not call out. The run-to-run
   noise floor is too high to claim "0%" at low concurrency.

2. get_finished() was never instrumented. The patch only times
   step_duration_us and build_meta_us. "100% of per-step cost is
   build_meta" is an upper bound on what was timed, not a true
   decomposition.

3. H5 (cache-size dependence) was the central hypothesis but
   was never tested in the prior run; random content kept APC
   near empty.

The +7-9% high-concurrency (single instance, 512x64, rate=8-16)
and +17% 8-instance-saturated numbers are kept; they were
measured with adequate sample sizes and are reproducible.

The follow-up sweep in cache_sweep/ tests H5 directly and
revises the decomposition.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-26 23:33:01 +08:00
c8ec73c548 Connector tax: high-concurrency confirms +7-9% tax, resolves trace-replay gap
High-concurrency test (512 input, 64 output, rates 4-32 req/s):
  Rate=8:  plain TTFT p90=94ms, mooncake_both=102ms → +9% tax
  Rate=16: plain TTFT p90=144ms, mooncake_both=156ms → +8% tax
  Rate=32: both saturated at ~6.1s → no distinguishable difference

Low-concurrency back-to-back retest (4096 input, 256 output):
  mooncake_both_v2 vs plain_v2: tax is ≈0% (within noise)
  because scheduler's 1.4ms/step is hidden behind model forward.

Decomposition of trace-replay's +45%:
  +7-9% from build_connector_meta per-step cost (this microbench)
  +20-30% from multi-instance coupling amplification (not measurable here)
  remainder from large-cache O(|cache|) scaling (Phase B follow-up)

Also: bench_loop.py now emits mean/p50/p90/p99 for all three metrics.
2026-05-26 21:00:25 +08:00
a473c71cac Connector tax Phase A: build_connector_meta is 1.4ms/step (the tax source)
Per-step timing from engine_step.jsonl definitively resolves H3:
  plain:            53 μs/step (p50)
  noop_connector:   69 μs/step (+16 μs = negligible framework cost)
  mooncake_producer: 1461 μs/step (build_connector_meta = 1386 μs)
  mooncake_both:    1452 μs/step (same as producer)

The substrate tax is NOT in the v1 framework — it's specifically in
Mooncake's build_connector_meta() which walks set(cache.keys()) every
scheduler step (O(|cache|) per step, E2 audit §6.5).

Accumulated per-request tax: 256 decode steps × 1.4ms = 358ms.
Observed TTFT tax at rate=1.0: plain 378ms vs mooncake_both 422ms (+12%).
At rate=2.0 (near saturation): +29%, approaching trace-replay's +45%.

Also fixes kill_vllm() to properly kill EngineCore subprocesses.
2026-05-26 19:33:15 +08:00