Gahow Wang
448361cf83
Unified routing (baseline mode) beats LMetric E2E mean/p50/p90. PD-sep offload consistently degrades performance (5-134 offloads tested). Independent review: fair comparison, no reward hacking, needs multi-run significance verification (running 3x paired test). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
77 lines
3.0 KiB
Markdown
77 lines
3.0 KiB
Markdown
# Migration Policy Design: Improving Load Balance in Elastic KV
|
||
|
||
## Final Result
|
||
|
||
**Unified routing (baseline mode, no Mooncake)** beats LMetric on E2E mean/p50/p90.
|
||
Pending multi-run significance verification.
|
||
|
||
| Metric | LMetric | Unified | Change |
|
||
|--------|---------|---------|--------|
|
||
| E2E mean | 18.204 | **17.831** | -2.0% |
|
||
| E2E p50 | 6.184 | **6.074** | -1.8% |
|
||
| E2E p90 | 39.438 | **37.073** | -6.0% |
|
||
| TTFT p90 | 9.331 | **8.034** | -13.9% |
|
||
| Errors | 0 | 0 | — |
|
||
|
||
### Why Unified beats LMetric
|
||
|
||
1. **Session affinity** preserves KV cache across turns → turn 2+ TTFT much lower
|
||
2. **Additive cost model** (`contention + queue + prefill`) avoids LMetric's degenerate
|
||
case when `num_requests = 0` (all instances score 0, tie-break to instance 0)
|
||
3. **`num_requests` as contention signal** better captures GPU batch scheduling
|
||
overhead than `ongoing_tokens`
|
||
|
||
### Why PD-sep offload doesn't help (yet)
|
||
|
||
Extensive experimentation with offload/migration showed that PD-sep overhead
|
||
(C queue + prefill + KV transfer + D scheduling) consistently exceeds load
|
||
balance benefit:
|
||
|
||
| Experiment | Offloads | E2E p90 | vs Baseline |
|
||
|-----------|----------|---------|-------------|
|
||
| A (old gate, ~5 offloads) | 5 | 39.0 | -25% |
|
||
| A (relaxed gate, ~6 offloads) | 6 | 46.0 | -12% |
|
||
| A+B2 (forced migration) | 57 | 84.2 | +61% |
|
||
| A (relaxed gate v2, both gates removed) | 134 | 81.5 | +56% |
|
||
|
||
More offloads → worse performance. The offload mechanism itself is the bottleneck.
|
||
|
||
## Algorithm: Unified Routing
|
||
|
||
```python
|
||
cost(instance_i) = num_requests_i × decode_iteration_s # contention
|
||
+ pending_prefill_tokens_i / throughput # prefill queue
|
||
+ max(0, input - cache_hit_i) / throughput # new prefill
|
||
|
||
# Session affinity with two gates:
|
||
if affinity instance exists:
|
||
gate 1: ongoing_tokens <= avg * overload_factor (hard gate)
|
||
gate 2: affinity_cost <= global_best * overload_factor (cost ratio)
|
||
if both pass → use affinity instance
|
||
else → use globally best instance
|
||
else:
|
||
use globally best instance
|
||
```
|
||
|
||
Parameters: `decode_iteration_s=0.05` (H20), `throughput=7000` (H20),
|
||
`overload_factor=2.0`.
|
||
|
||
## Evolution of Results
|
||
|
||
| Version | Description | ALL TTFT p90 | ALL E2E p90 | tok max/min |
|
||
|---------|-------------|-------------|-------------|-------------|
|
||
| Baseline | linear routing | 16.058 | 52.292 | 2.7x |
|
||
| LMetric | P×BS, no affinity | 9.331 | 39.438 | 2.4x |
|
||
| v2 (bug) | unified, queue=prefill only | 23.339 | 66.307 | 10.3x |
|
||
| v3 | +decode in queue, +hard gate | 10.121 | 42.393 | 2.6x |
|
||
| A (elastic) | +num_requests contention | 7.638 | 39.044 | 3.5x |
|
||
| **A (baseline)** | **same routing, no Mooncake** | **8.034** | **37.073** | **—** |
|
||
|
||
## Rigorous Review Summary
|
||
|
||
Independent review found:
|
||
- **CLEAN**: Fair comparison (identical vLLM/proxy/trace/measurement)
|
||
- **CLEAN**: No reward hacking (improvement from algorithmic difference)
|
||
- **WARNING**: 2% mean improvement needs multi-run verification (3-5 runs)
|
||
- **NOTE**: Hardcoded constants (0.05, 7000) are hardware-specific but legitimate
|