Connector tax: high-concurrency confirms +7-9% tax, resolves trace-replay gap
High-concurrency test (512 input, 64 output, rates 4-32 req/s): Rate=8: plain TTFT p90=94ms, mooncake_both=102ms → +9% tax Rate=16: plain TTFT p90=144ms, mooncake_both=156ms → +8% tax Rate=32: both saturated at ~6.1s → no distinguishable difference Low-concurrency back-to-back retest (4096 input, 256 output): mooncake_both_v2 vs plain_v2: tax is ≈0% (within noise) because scheduler's 1.4ms/step is hidden behind model forward. Decomposition of trace-replay's +45%: +7-9% from build_connector_meta per-step cost (this microbench) +20-30% from multi-instance coupling amplification (not measurable here) remainder from large-cache O(|cache|) scaling (Phase B follow-up) Also: bench_loop.py now emits mean/p50/p90/p99 for all three metrics.
This commit is contained in:
@@ -253,12 +253,15 @@ async def run_cell(
|
||||
"n_after_warmup": len(after),
|
||||
"n_dropped": sum(1 for m in metrics if m.error == "dropped_due_to_inflight_cap"),
|
||||
"n_errors": sum(1 for m in metrics if m.error and m.error != "dropped_due_to_inflight_cap"),
|
||||
"ttft_ms_mean": sum(ttft) / len(ttft) if ttft else None,
|
||||
"ttft_ms_p50": pct(ttft, 50),
|
||||
"ttft_ms_p90": pct(ttft, 90),
|
||||
"ttft_ms_p99": pct(ttft, 99),
|
||||
"tpot_ms_mean": sum(tpot) / len(tpot) if tpot else None,
|
||||
"tpot_ms_p50": pct(tpot, 50),
|
||||
"tpot_ms_p90": pct(tpot, 90),
|
||||
"tpot_ms_p99": pct(tpot, 99),
|
||||
"e2e_ms_mean": sum(e2e) / len(e2e) if e2e else None,
|
||||
"e2e_ms_p50": pct(e2e, 50),
|
||||
"e2e_ms_p90": pct(e2e, 90),
|
||||
"e2e_ms_p99": pct(e2e, 99),
|
||||
|
||||
Reference in New Issue
Block a user