Files

Gahow Wang 448361cf83 Update design doc: final results + review findings

Unified routing (baseline mode) beats LMetric E2E mean/p50/p90.
PD-sep offload consistently degrades performance (5-134 offloads tested).
Independent review: fair comparison, no reward hacking, needs multi-run
significance verification (running 3x paired test).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-25 03:48:18 +08:00

3.0 KiB

Raw Blame History

Migration Policy Design: Improving Load Balance in Elastic KV

Final Result

Unified routing (baseline mode, no Mooncake) beats LMetric on E2E mean/p50/p90. Pending multi-run significance verification.

Metric	LMetric	Unified	Change
E2E mean	18.204	17.831	-2.0%
E2E p50	6.184	6.074	-1.8%
E2E p90	39.438	37.073	-6.0%
TTFT p90	9.331	8.034	-13.9%
Errors	0	0	—

Why Unified beats LMetric

Session affinity preserves KV cache across turns → turn 2+ TTFT much lower
Additive cost model (contention + queue + prefill) avoids LMetric's degenerate case when num_requests = 0 (all instances score 0, tie-break to instance 0)
num_requests as contention signal better captures GPU batch scheduling overhead than ongoing_tokens

Why PD-sep offload doesn't help (yet)

Extensive experimentation with offload/migration showed that PD-sep overhead (C queue + prefill + KV transfer + D scheduling) consistently exceeds load balance benefit:

Experiment	Offloads	E2E p90	vs Baseline
A (old gate, ~5 offloads)	5	39.0	-25%
A (relaxed gate, ~6 offloads)	6	46.0	-12%
A+B2 (forced migration)	57	84.2	+61%
A (relaxed gate v2, both gates removed)	134	81.5	+56%

More offloads → worse performance. The offload mechanism itself is the bottleneck.

Algorithm: Unified Routing

cost(instance_i) = num_requests_i × decode_iteration_s     # contention
                 + pending_prefill_tokens_i / throughput     # prefill queue
                 + max(0, input - cache_hit_i) / throughput  # new prefill

# Session affinity with two gates:
if affinity instance exists:
    gate 1: ongoing_tokens <= avg * overload_factor  (hard gate)
    gate 2: affinity_cost <= global_best * overload_factor  (cost ratio)
    if both pass → use affinity instance
    else → use globally best instance
else:
    use globally best instance

Parameters: decode_iteration_s=0.05 (H20), throughput=7000 (H20), overload_factor=2.0.

Evolution of Results

Version	Description	ALL TTFT p90	ALL E2E p90	tok max/min
Baseline	linear routing	16.058	52.292	2.7x
LMetric	P×BS, no affinity	9.331	39.438	2.4x
v2 (bug)	unified, queue=prefill only	23.339	66.307	10.3x
v3	+decode in queue, +hard gate	10.121	42.393	2.6x
A (elastic)	+num_requests contention	7.638	39.044	3.5x
A (baseline)	same routing, no Mooncake	8.034	37.073	—

Rigorous Review Summary

Independent review found:

CLEAN: Fair comparison (identical vLLM/proxy/trace/measurement)
CLEAN: No reward hacking (improvement from algorithmic difference)
WARNING: 2% mean improvement needs multi-run verification (3-5 runs)
NOTE: Hardcoded constants (0.05, 7000) are hardware-specific but legitimate

3.0 KiB Raw Blame History Unescape Escape