# Unified routing ablation: A (tighter affinity) + B (decode-aware LMetric)

Goal: judge whether `unified` (cache-aware hybrid affinity + LMetric fallback)
has enough headroom to surpass v3 migration-based routing on agentic
workloads, without invoking PD-sep migration.

## Workload / baseline

- Trace: `w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions)
- Hardware: 8 × H100 (dash0), Qwen3-Coder-30B-A3B, TP=1, max_model_len=200000
- Trace replay through `cache_aware_proxy.py` with policy `unified`
- `b3_replay_20260527_0114/unified/` reference

| Metric (ms) | baseline (`overload_factor=2.0`) |
|---|---:|
| TTFT p50 | 520 |
| TTFT p90 | **8781** |
| TTFT p99 | 47647 |
| TPOT p90 | 17.8 |
| E2E p90 | 19989 |
| E2E p99 | 85841 |

Reference points we're trying to beat / match:
- v3 fixed rotation (cache-blind picker): TTFT p90 = 10828
- v3 + Mechanism B (cache-rich picker): TTFT p90 = 9711
- All v3 variants are +10–23% worse than `unified` baseline.

## Tail-source diagnostic on baseline

Decision split, baseline unified:

| Decision | n | TTFT mean | TTFT p90 | TTFT p99 |
|---|---:|---:|---:|---:|
| affinity     | 852 | 3183 | 7011 | 47432 |
| lmetric_fallback | 362 | 4285 | 12083 | 46036 |

Long-tail (>20s, n=65):
- 40 / 65 came from `affinity` decisions
- 25 / 65 came from `lmetric_fallback`

For the 40 slow `affinity` reqs:
- only 12 / 40 were actually overloaded at decision time (`aff_num_req > avg_num_req`)
- overload ratio at decision: mean=0.93, p50=0.87
- **most slow affinity reqs looked fine when the picker stuck — load piled
  on after dispatch**.

This is a snapshot-based-routing limitation. Tightening
`overload_factor` only helps the genuine cases above the new threshold —
expected to be a 5-10% improvement at best.

---

## Direction A — tighten affinity overflow

**Hypothesis.** `overload_factor=2.0` lets the picker stick to affinity
even when `affinity.num_req` is up to 2× the cluster average. Reducing to
1.3 forces earlier overflow to LMetric fallback, escaping busy affinity
hosts before the tail blows up.

**Change.** Single CLI flag: `--overload-factor 1.3`. No code change.

**Run.** `unified_of13_20260527_1532/unified/`.

### A vs baseline

| Metric (ms) | baseline (of=2.0) | A (of=1.3) | Δ |
|---|---:|---:|---:|
| TTFT p50 | 520 | 495 | −5% |
| TTFT p90 | 8781 | 8730 | ≈0 |
| TTFT p99 | 47647 | 43059 | −10% |
| TPOT p50 | 7.9 | 8.0 | ≈0 |
| TPOT p90 | 17.8 | **15.5** | **−13%** |
| E2E p50 | 1761 | 1824 | +4% |
| E2E p90 | 19989 | 18407 | −8% |
| E2E p99 | 85841 | **71396** | **−17%** |

TTFT p90 is essentially unchanged but the **deeper tail (p99) and
TPOT both improved meaningfully**. Net: A alone gives roughly −10% to
−17% on the long tail without hurting medians.

### Decision split, A vs baseline

| Decision | baseline n / p90 | A n / p90 | Δ p90 |
|---|---|---|---|
| affinity | 852 / 7011 | 817 / **5817** | **−17%** ✅ |
| lmetric_fallback | 362 / 12083 | 397 / **15360** | **+27%** ⚠️ |

The picker now sticks to affinity 35 fewer times. The remaining affinity
decisions are higher-quality (no longer "barely-fitting" cases), so their
p90 drops 17%.

But the 35 extra reqs that got pushed into fallback **got slower**:
fallback p90 went from 12083 → 15360. The LMetric scorer is selecting a
worse instance for them.

### Per-worker TTFT under A (of=1.3)

```
port 8000: n=  94  mean=4424  p90=12290    port 8004: n=192  mean=2597  p90=6968
port 8001: n= 135  mean=2779  p90= 5553    port 8005: n=202  mean=3102  p90=6113
port 8002: n=  88  mean=5827  p90=15804    port 8006: n=136  mean=4006  p90=10899
port 8003: n= 217  mean=2674  p90= 4598    port 8007: n=150  mean=3648  p90= 7025
```

Compared to baseline (88..217 reqs/port), A redistributes more evenly
(88..217 still but distribution is fatter in the middle). port 8002
remains slow (p90 15.8s) — its cache pool seems to keep getting cold
work routed there by LMetric.

### Why A alone isn't enough

LMetric scorer (`unified_hybrid` fallback path):

```python
score = (pending_prefill_tokens + new_uncached_tokens) * num_requests
```

This **ignores `ongoing_decode_tokens`** entirely. An instance with no
pending prefill but 200k tokens currently in decode looks "ideal"
(score=0×num_req=0) — yet a new request landing there waits behind
slow decode iters caused by the large batch KV reads.

A pushes more requests into fallback, but fallback can't tell which
instance is actually free. → Direction B is mandatory companion.

---

## Direction B — decode-aware LMetric

**Hypothesis.** Adding a decode-load penalty to the LMetric score lets
fallback distinguish "no prefill queued but heavy decode running" from
"truly idle". Should restore fallback p90 ≤ 12s baseline level.

**Change.**
```python
score = (pending_prefill + new + lmetric_decode_weight * ongoing_decode_tokens) * num_requests
```
- `lmetric_decode_weight=0.0` ⇒ original LMetric (control)
- `lmetric_decode_weight=0.01` ⇒ first experiment (rationale: 1 decode token
  in batch costs ~0.01 prefill-token-equivalent in scheduler iter time
  on H100 + Qwen3-30B-A3B)

CLI: `--lmetric-decode-weight 0.01`. Setting in code:
`cache_aware_proxy.py:Settings.lmetric_decode_weight`.

**Run.** `unified_of13_lmw001_20260527_1628/unified/`.

### A+B vs baseline / A

| Metric (ms) | baseline | A (of=1.3) | A+B (of=1.3, lmw=0.01) | Δ vs baseline |
|---|---:|---:|---:|---:|
| TTFT p50 | 520 | 495 | 514 | −1% |
| **TTFT p90** | 8781 | 8730 | **8421** | **−4%** ✅ |
| TTFT p99 | 47647 | 43059 | 44800 | −6% |
| TPOT p50 | 7.9 | 8.0 | 7.9 | ≈0 |
| TPOT p90 | 17.8 | 15.5 | 15.7 | −12% |
| E2E p50 | 1761 | 1824 | 1870 | +6% |
| E2E p90 | 19989 | 18407 | **21064** | **+5%** ⚠️ |
| E2E p99 | 85841 | 71396 | **64344** | **−25%** ✅ |

Long-tail counts:

```
thresh       baseline      A      A+B   v3 MechB
>  5000ms        170      173      170        177
> 10000ms        105      109      109        119
> 20000ms         65       64       59         78
> 30000ms         41       40       37         50
> 50000ms          8        5        6         14
```

A+B is best on every long-tail-count threshold ≤30s, marginal worse at 50s.

### Decision split (A+B vs A)

| Decision | A (of=1.3) | A+B | Note |
|---|---|---|---|
| affinity p90 | 5817 | 5836 | ≈ same |
| fallback p90 | **15360** | **13501** | B recovered some of A's fallback regression |

B partially fixed fallback's selection (−12% on fallback p90 vs A alone),
but still worse than baseline (12083).

### Per-worker TTFT (A+B)

```
port 8000: n=134  mean=3495  p90=10967      port 8004: n=136  mean=3102  p90= 7906
port 8001: n=143  mean=2981  p90=10189      port 8005: n=179  mean=1624  p90= 2735
port 8002: n=221  mean=2355  p90= 3502      port 8006: n=137  mean=5356  p90= 9628
port 8003: n=146  mean=3932  p90=10729      port 8007: n=118  mean=5210  p90=26798  ← new hotspot
```

A+B trades the baseline's 8002 hotspot (p90=35s) for a new 8007 hotspot
(p90=26.8s). Lower amplitude but hotspot survives.

### Why 8007 became a hotspot under A+B — **found a bug in B**

8007 in A+B: 118 reqs, **53% affinity / 47% fallback** (vs other ports
60–77% affinity), **cache_hit_mean=50.5% (lowest)**.

Top-10 slowest at 8007: all are big-prompt (100k+ tokens) fallback decisions
with `cached_tokens=0` (cold prefill). LMetric is pushing many cold-prefill
fallbacks to 8007.

Looking at the B formula:

```python
decode_pen = lmetric_decode_weight * ongoing_decode_tokens
score = (pending_prefill + new + decode_pen) * num_requests   # ← BUG
```

When `num_requests = 0`, the entire score (including decode penalty) zeros
out. So an idle-but-decoding host (num_req=0 because its last prefill
finished but decode is still running) looks like score=0, beating every
busy host.

**Fix (B'):** multiply by `max(num_requests, 1)`:

```python
score = (pending_prefill + new + decode_pen) * max(num_requests, 1)
```

Now idle hosts with high decode load get score = decode_pen × 1 = real
nonzero penalty, beating zero-load hosts only when decode is small.

### A+B' — re-run with the fix

**Run.** `unified_of13_lmw001_v2_20260527_1724/unified/`.

| Metric (ms) | baseline | A+B (BUG) | A+B' (fix) | Δ vs baseline |
|---|---:|---:|---:|---:|
| TTFT p50 | 520 | 514 | **485** | −7% |
| **TTFT p90** | 8781 | 8421 | **8287** | **−5.6%** ✅ |
| TTFT p99 | 47647 | 44800 | **41876** | **−12%** ✅ |
| TPOT p90 | 17.8 | 15.7 | 17.5 | −2% |
| E2E p90 | 19989 | 21064 | 20625 | +3% |
| E2E p99 | 85841 | 64344 | 77827 | −9% |

A+B' **best of all variants on TTFT p90 (8287) and TTFT p99 (41876)**.
Long-tail counts (>30s, >50s) also best across variants.

vs v3 reference points:
| | TTFT p90 | TPOT p90 | E2E p99 |
|---|---:|---:|---:|
| **A+B'** | **8287** | 17.5 | 77827 |
| v3 fixed (cache-blind) | 10828 | 21.0 | 47610 |
| v3 + Mech B | 9711 | 18.3 | 84492 |

A+B' **beats v3 Mech B by 15% TTFT p90** with no migration overhead.

### Per-worker (A+B' fixed)

```
8000: n=158  p90= 5688      8004: n=189  p90= 4249
8001: n=159  p90= 7323      8005: n=116  p90=14598
8002: n=114  p90= 8726      8006: n=180  p90= 6198
8003: n=173  p90= 6715      8007: n=125  p90=22242   ← still hot
```

A+B' redistributed load more evenly (114..189) but **8007 still has p90=22s**.

### 8007 deep-dive in A+B'

```
8007: n=125, affinity=69 (55%), fallback=56 (45%), cache_hit_mean=lowest
```

Top-15 slow at 8007:
- 7 of them are session **1313181** turns 9–14 (130k+ tokens each, agentic
  long context, ~50% cache hit)
- Several others are cold-start turn-1 of large-prompt sessions
- First two slow reqs arrived **0.7 s apart** — strong hint of concurrent
  picker race

### Iteration 3: race-condition fix

**Diagnosis.** In `_handle_combined`:

```python
chosen, best_idx, decision = pick_instance_unified_hybrid(...)  # sync
# ... sync breakdown updates ...
return await _handle_local_request(...)   # ← await yields here
                                          #   THEN reservation happens
```

`return await async_func(...)` evaluates the async call (creates coroutine)
and yields to the event loop **before** the coroutine body executes. The
reservation (`chosen.pending_prefill_tokens += new`, etc.) lives at the top
of `_handle_local_request`, so between the picker and the reservation there
is a **window where another coroutine can run and re-pick the same instance**.

When two big-prompt reqs arrive within milliseconds, both run pick →
both pick the "free" 8007 → both yield → both reserve. Result: 8007 gets
back-to-back 130k-token cold prefills, each waiting for the other.

**Fix.** Move the reservation **before** the await, inside `_handle_combined`:

```python
# Race fix: reserve atomically with pick, before any await.
chosen.ongoing_tokens += input_length
chosen.pending_prefill_tokens += estimated_new
chosen.num_requests += 1
return await _handle_local_request(..., _pre_reserved=True)
```

`_handle_local_request` skips its own reservation when `_pre_reserved=True`.
PD-sep paths are unaffected (they have their own reservation).

**Run.** Pending — `unified_of13_lmw001_racefix_*`. Hypothesis: 8007 p90
drops to within ±3s of cluster median, since concurrent picks for the
same "free" instance no longer happen.

---

## A+B'+RaceFix — results

**Run.** `unified_of13_lmw001_racefix_20260527_1821/unified/`.

| Metric (ms) | baseline | A+B' | A+B'+RF | Δ vs baseline |
|---|---:|---:|---:|---:|
| TTFT p50 | 520 | 485 | **478** | −8% |
| **TTFT p90** | 8781 | 8287 | **7770** | **−11.5%** ✅ |
| TTFT p99 | 47647 | 41876 | **42447** | −11% |
| TPOT p90 | 17.8 | 17.5 | 18.0 | +1% |
| E2E p90 | 19989 | 20625 | **18418** | −8% |
| E2E p99 | 85841 | 77827 | **71227** | −17% |

vs v3 reference:
- **A+B'+RF TTFT p90 = 7770ms, vs v3 Mech B 9711ms → −20%** ✅

Long-tail counts (best across all variants):
```
> 5s:   170 → 158         > 30s:   41 → 33
>10s:   105 → 103         > 50s:    8 →  4
>20s:    65 →  57         >100s:    0 →  0
```

### Decision split — race fix mainly helped affinity

| Decision | baseline | A+B'+RF |
|---|---:|---:|
| affinity p90 | 7011 | **5042** ✅ (−28%) |
| fallback p90 | 12083 | 13944 (+15%) |

The race-condition was hurting affinity decisions the most. When two
concurrent reqs both stuck to a "free-looking" affinity instance, they
piled up and inflated affinity's tail. Fix removed this collision.

### Per-worker

```
8000: n=86   p90=11541      8004: n=150  p90=11906
8001: n=186  p90= 8307      8005: n=109  p90= 4798
8002: n=105  p90=14540      8006: n=183  p90= 6258
8003: n=264  p90= 3079      8007: n=131  p90=21850   ← still hot
8000 spread now 86..264 — race fix did disperse routing
```

### 8007 still hot — but it's **workload-inherent, not a routing bug**

Top sessions on 8007:
```
session 1279412: n=22  mean= 2208  max=18985  decisions: 91% affinity
session 1313181: n=17  mean=17399  max=49089  decisions: 65% affinity
session 1262354: n=15  mean=  622  max= 2325  decisions: 87% affinity
session 1342921: n= 7  mean=17817  max=55589  decisions: 86% affinity
session 1260327: n= 8  mean= 1636  max= 5382  decisions: 75% affinity
session 1268831: n= 5  mean= 1443  max= 2673  decisions: 80% affinity
```

Sessions 1313181 and 1342921 are **long agentic contexts**: 100k–130k tokens
per turn with ~50% cache hit (i.e. 50k new tokens prefill per turn). Even
on a perfectly load-balanced instance, each turn is 7–15s of pure compute.

Forcing these sessions to spread across instances would mean **cold prefill
every turn (0% cache hit)** → each turn becomes 20–30s instead of 7–15s.
Spreading is **net-negative**.

→ The 8007 p90=22s is the floor imposed by these sessions' structure,
not by routing policy. Unified is at its ceiling for this workload.

---

## Final ranking and take-aways

| Policy | TTFT p90 (ms) | Δ vs baseline | Notes |
|---|---:|---:|---|
| baseline unified (of=2.0) | 8781 | — | reference |
| A (of=1.3) | 8730 | ≈0 | affinity p90 -17%, fallback p90 +27% |
| A+B (of=1.3, lmw=0.01, BUG) | 8421 | −4% | 8007 hotspot from `*num_req` zeroing bug |
| A+B' (formula fix) | 8287 | −5.6% | Bug fixed, still 8007 mild hotspot |
| **A+B'+RaceFix** | **7770** | **−11.5%** ✅ | **Best unified variant** |
| v3 fixed | 10828 | +23% | PD-sep migration, cache-blind picker |
| v3 + Mech B | 9711 | +11% | PD-sep + cache-rich target picker |

### Conclusions

1. **Unified path beats v3 PD-sep on this workload by 20%+ TTFT p90.**
   PD-sep migration's fixed cost (src prefill + dst first-token waiting on
   loaded scheduler) outweighs any decode-time savings for short-output
   agentic turns.

2. **Three orthogonal fixes compound for a 11.5% TTFT p90 win:**
   - A (`overload_factor=1.3`): tighter affinity overflow → −0.6% but
     much cleaner affinity decisions (p90 -17%)
   - B' (`lmetric_decode_weight=0.01` with `max(num_req,1)`): decode-aware
     fallback → −3.5%
   - RaceFix (atomic reserve before await): kills concurrent-pick
     collisions → −5.6%

3. **Race condition was the biggest single hidden bug.** `return await
   async_func(...)` yields to the event loop **before** the body of
   `async_func` runs, so reservations done in the body don't take effect
   in time to deter concurrent picks. This affects ANY async dispatch
   with separate pick/reserve steps — worth checking other routing
   policies.

4. **8007 p90=22s is workload-inherent.** Sessions with 100k+ token turns
   at 50% cache hit cannot finish faster than 7–15s per turn regardless
   of routing. Forcing spread would hurt rather than help.

5. **Migration (v3) is not necessary** when unified routing is tuned
   well. Save the PD-sep mechanism for cases where it can be proven
   net-positive (e.g. very-long-output sessions on extremely overloaded
   prefill hosts) and use unified A+B'+RaceFix as the default.

---

## Direction A+B — run pending

(Will be filled when `unified_of13_lmw001_*/unified/` finishes.)
