# Migration Policy Design: Improving Load Balance in Elastic KV

## Final Result

**Unified routing (baseline mode, no Mooncake)** beats LMetric on E2E mean/p50/p90.
Pending multi-run significance verification.

| Metric | LMetric | Unified | Change |
|--------|---------|---------|--------|
| E2E mean | 18.204 | **17.831** | -2.0% |
| E2E p50 | 6.184 | **6.074** | -1.8% |
| E2E p90 | 39.438 | **37.073** | -6.0% |
| TTFT p90 | 9.331 | **8.034** | -13.9% |
| Errors | 0 | 0 | — |

### Why Unified beats LMetric

1. **Session affinity** preserves KV cache across turns → turn 2+ TTFT much lower
2. **Additive cost model** (`contention + queue + prefill`) avoids LMetric's degenerate
   case when `num_requests = 0` (all instances score 0, tie-break to instance 0)
3. **`num_requests` as contention signal** better captures GPU batch scheduling
   overhead than `ongoing_tokens`

### Why PD-sep offload doesn't help (yet)

Extensive experimentation with offload/migration showed that PD-sep overhead
(C queue + prefill + KV transfer + D scheduling) consistently exceeds load
balance benefit:

| Experiment | Offloads | E2E p90 | vs Baseline |
|-----------|----------|---------|-------------|
| A (old gate, ~5 offloads) | 5 | 39.0 | -25% |
| A (relaxed gate, ~6 offloads) | 6 | 46.0 | -12% |
| A+B2 (forced migration) | 57 | 84.2 | +61% |
| A (relaxed gate v2, both gates removed) | 134 | 81.5 | +56% |

More offloads → worse performance. The offload mechanism itself is the bottleneck.

## Algorithm: Unified Routing

```python
cost(instance_i) = num_requests_i × decode_iteration_s     # contention
                 + pending_prefill_tokens_i / throughput     # prefill queue
                 + max(0, input - cache_hit_i) / throughput  # new prefill

# Session affinity with two gates:
if affinity instance exists:
    gate 1: ongoing_tokens <= avg * overload_factor  (hard gate)
    gate 2: affinity_cost <= global_best * overload_factor  (cost ratio)
    if both pass → use affinity instance
    else → use globally best instance
else:
    use globally best instance
```

Parameters: `decode_iteration_s=0.05` (H20), `throughput=7000` (H20),
`overload_factor=2.0`.

## Evolution of Results

| Version | Description | ALL TTFT p90 | ALL E2E p90 | tok max/min |
|---------|-------------|-------------|-------------|-------------|
| Baseline | linear routing | 16.058 | 52.292 | 2.7x |
| LMetric | P×BS, no affinity | 9.331 | 39.438 | 2.4x |
| v2 (bug) | unified, queue=prefill only | 23.339 | 66.307 | 10.3x |
| v3 | +decode in queue, +hard gate | 10.121 | 42.393 | 2.6x |
| A (elastic) | +num_requests contention | 7.638 | 39.044 | 3.5x |
| **A (baseline)** | **same routing, no Mooncake** | **8.034** | **37.073** | **—** |

## Rigorous Review Summary

Independent review found:
- **CLEAN**: Fair comparison (identical vLLM/proxy/trace/measurement)
- **CLEAN**: No reward hacking (improvement from algorithmic difference)
- **WARNING**: 2% mean improvement needs multi-run verification (3-5 runs)
- **NOTE**: Hardcoded constants (0.05, 7000) are hardware-specific but legitimate