# Unified routing ablation: A (tighter affinity) + B (decode-aware LMetric) Goal: judge whether `unified` (cache-aware hybrid affinity + LMetric fallback) has enough headroom to surpass v3 migration-based routing on agentic workloads, without invoking PD-sep migration. ## Workload / baseline - Trace: `w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions) - Hardware: 8 × H100 (dash0), Qwen3-Coder-30B-A3B, TP=1, max_model_len=200000 - Trace replay through `cache_aware_proxy.py` with policy `unified` - `b3_replay_20260527_0114/unified/` reference | Metric (ms) | baseline (`overload_factor=2.0`) | |---|---:| | TTFT p50 | 520 | | TTFT p90 | **8781** | | TTFT p99 | 47647 | | TPOT p90 | 17.8 | | E2E p90 | 19989 | | E2E p99 | 85841 | Reference points we're trying to beat / match: - v3 fixed rotation (cache-blind picker): TTFT p90 = 10828 - v3 + Mechanism B (cache-rich picker): TTFT p90 = 9711 - All v3 variants are +10–23% worse than `unified` baseline. ## Tail-source diagnostic on baseline Decision split, baseline unified: | Decision | n | TTFT mean | TTFT p90 | TTFT p99 | |---|---:|---:|---:|---:| | affinity | 852 | 3183 | 7011 | 47432 | | lmetric_fallback | 362 | 4285 | 12083 | 46036 | Long-tail (>20s, n=65): - 40 / 65 came from `affinity` decisions - 25 / 65 came from `lmetric_fallback` For the 40 slow `affinity` reqs: - only 12 / 40 were actually overloaded at decision time (`aff_num_req > avg_num_req`) - overload ratio at decision: mean=0.93, p50=0.87 - **most slow affinity reqs looked fine when the picker stuck — load piled on after dispatch**. This is a snapshot-based-routing limitation. Tightening `overload_factor` only helps the genuine cases above the new threshold — expected to be a 5-10% improvement at best. --- ## Direction A — tighten affinity overflow **Hypothesis.** `overload_factor=2.0` lets the picker stick to affinity even when `affinity.num_req` is up to 2× the cluster average. Reducing to 1.3 forces earlier overflow to LMetric fallback, escaping busy affinity hosts before the tail blows up. **Change.** Single CLI flag: `--overload-factor 1.3`. No code change. **Run.** `unified_of13_20260527_1532/unified/`. ### A vs baseline | Metric (ms) | baseline (of=2.0) | A (of=1.3) | Δ | |---|---:|---:|---:| | TTFT p50 | 520 | 495 | −5% | | TTFT p90 | 8781 | 8730 | ≈0 | | TTFT p99 | 47647 | 43059 | −10% | | TPOT p50 | 7.9 | 8.0 | ≈0 | | TPOT p90 | 17.8 | **15.5** | **−13%** | | E2E p50 | 1761 | 1824 | +4% | | E2E p90 | 19989 | 18407 | −8% | | E2E p99 | 85841 | **71396** | **−17%** | TTFT p90 is essentially unchanged but the **deeper tail (p99) and TPOT both improved meaningfully**. Net: A alone gives roughly −10% to −17% on the long tail without hurting medians. ### Decision split, A vs baseline | Decision | baseline n / p90 | A n / p90 | Δ p90 | |---|---|---|---| | affinity | 852 / 7011 | 817 / **5817** | **−17%** ✅ | | lmetric_fallback | 362 / 12083 | 397 / **15360** | **+27%** ⚠️ | The picker now sticks to affinity 35 fewer times. The remaining affinity decisions are higher-quality (no longer "barely-fitting" cases), so their p90 drops 17%. But the 35 extra reqs that got pushed into fallback **got slower**: fallback p90 went from 12083 → 15360. The LMetric scorer is selecting a worse instance for them. ### Per-worker TTFT under A (of=1.3) ``` port 8000: n= 94 mean=4424 p90=12290 port 8004: n=192 mean=2597 p90=6968 port 8001: n= 135 mean=2779 p90= 5553 port 8005: n=202 mean=3102 p90=6113 port 8002: n= 88 mean=5827 p90=15804 port 8006: n=136 mean=4006 p90=10899 port 8003: n= 217 mean=2674 p90= 4598 port 8007: n=150 mean=3648 p90= 7025 ``` Compared to baseline (88..217 reqs/port), A redistributes more evenly (88..217 still but distribution is fatter in the middle). port 8002 remains slow (p90 15.8s) — its cache pool seems to keep getting cold work routed there by LMetric. ### Why A alone isn't enough LMetric scorer (`unified_hybrid` fallback path): ```python score = (pending_prefill_tokens + new_uncached_tokens) * num_requests ``` This **ignores `ongoing_decode_tokens`** entirely. An instance with no pending prefill but 200k tokens currently in decode looks "ideal" (score=0×num_req=0) — yet a new request landing there waits behind slow decode iters caused by the large batch KV reads. A pushes more requests into fallback, but fallback can't tell which instance is actually free. → Direction B is mandatory companion. --- ## Direction B — decode-aware LMetric **Hypothesis.** Adding a decode-load penalty to the LMetric score lets fallback distinguish "no prefill queued but heavy decode running" from "truly idle". Should restore fallback p90 ≤ 12s baseline level. **Change.** ```python score = (pending_prefill + new + lmetric_decode_weight * ongoing_decode_tokens) * num_requests ``` - `lmetric_decode_weight=0.0` ⇒ original LMetric (control) - `lmetric_decode_weight=0.01` ⇒ first experiment (rationale: 1 decode token in batch costs ~0.01 prefill-token-equivalent in scheduler iter time on H100 + Qwen3-30B-A3B) CLI: `--lmetric-decode-weight 0.01`. Setting in code: `cache_aware_proxy.py:Settings.lmetric_decode_weight`. **Run.** `unified_of13_lmw001_20260527_1628/unified/`. ### A+B vs baseline / A | Metric (ms) | baseline | A (of=1.3) | A+B (of=1.3, lmw=0.01) | Δ vs baseline | |---|---:|---:|---:|---:| | TTFT p50 | 520 | 495 | 514 | −1% | | **TTFT p90** | 8781 | 8730 | **8421** | **−4%** ✅ | | TTFT p99 | 47647 | 43059 | 44800 | −6% | | TPOT p50 | 7.9 | 8.0 | 7.9 | ≈0 | | TPOT p90 | 17.8 | 15.5 | 15.7 | −12% | | E2E p50 | 1761 | 1824 | 1870 | +6% | | E2E p90 | 19989 | 18407 | **21064** | **+5%** ⚠️ | | E2E p99 | 85841 | 71396 | **64344** | **−25%** ✅ | Long-tail counts: ``` thresh baseline A A+B v3 MechB > 5000ms 170 173 170 177 > 10000ms 105 109 109 119 > 20000ms 65 64 59 78 > 30000ms 41 40 37 50 > 50000ms 8 5 6 14 ``` A+B is best on every long-tail-count threshold ≤30s, marginal worse at 50s. ### Decision split (A+B vs A) | Decision | A (of=1.3) | A+B | Note | |---|---|---|---| | affinity p90 | 5817 | 5836 | ≈ same | | fallback p90 | **15360** | **13501** | B recovered some of A's fallback regression | B partially fixed fallback's selection (−12% on fallback p90 vs A alone), but still worse than baseline (12083). ### Per-worker TTFT (A+B) ``` port 8000: n=134 mean=3495 p90=10967 port 8004: n=136 mean=3102 p90= 7906 port 8001: n=143 mean=2981 p90=10189 port 8005: n=179 mean=1624 p90= 2735 port 8002: n=221 mean=2355 p90= 3502 port 8006: n=137 mean=5356 p90= 9628 port 8003: n=146 mean=3932 p90=10729 port 8007: n=118 mean=5210 p90=26798 ← new hotspot ``` A+B trades the baseline's 8002 hotspot (p90=35s) for a new 8007 hotspot (p90=26.8s). Lower amplitude but hotspot survives. ### Why 8007 became a hotspot under A+B — **found a bug in B** 8007 in A+B: 118 reqs, **53% affinity / 47% fallback** (vs other ports 60–77% affinity), **cache_hit_mean=50.5% (lowest)**. Top-10 slowest at 8007: all are big-prompt (100k+ tokens) fallback decisions with `cached_tokens=0` (cold prefill). LMetric is pushing many cold-prefill fallbacks to 8007. Looking at the B formula: ```python decode_pen = lmetric_decode_weight * ongoing_decode_tokens score = (pending_prefill + new + decode_pen) * num_requests # ← BUG ``` When `num_requests = 0`, the entire score (including decode penalty) zeros out. So an idle-but-decoding host (num_req=0 because its last prefill finished but decode is still running) looks like score=0, beating every busy host. **Fix (B'):** multiply by `max(num_requests, 1)`: ```python score = (pending_prefill + new + decode_pen) * max(num_requests, 1) ``` Now idle hosts with high decode load get score = decode_pen × 1 = real nonzero penalty, beating zero-load hosts only when decode is small. ### A+B' — re-run with the fix **Run.** `unified_of13_lmw001_v2_20260527_1724/unified/`. | Metric (ms) | baseline | A+B (BUG) | A+B' (fix) | Δ vs baseline | |---|---:|---:|---:|---:| | TTFT p50 | 520 | 514 | **485** | −7% | | **TTFT p90** | 8781 | 8421 | **8287** | **−5.6%** ✅ | | TTFT p99 | 47647 | 44800 | **41876** | **−12%** ✅ | | TPOT p90 | 17.8 | 15.7 | 17.5 | −2% | | E2E p90 | 19989 | 21064 | 20625 | +3% | | E2E p99 | 85841 | 64344 | 77827 | −9% | A+B' **best of all variants on TTFT p90 (8287) and TTFT p99 (41876)**. Long-tail counts (>30s, >50s) also best across variants. vs v3 reference points: | | TTFT p90 | TPOT p90 | E2E p99 | |---|---:|---:|---:| | **A+B'** | **8287** | 17.5 | 77827 | | v3 fixed (cache-blind) | 10828 | 21.0 | 47610 | | v3 + Mech B | 9711 | 18.3 | 84492 | A+B' **beats v3 Mech B by 15% TTFT p90** with no migration overhead. ### Per-worker (A+B' fixed) ``` 8000: n=158 p90= 5688 8004: n=189 p90= 4249 8001: n=159 p90= 7323 8005: n=116 p90=14598 8002: n=114 p90= 8726 8006: n=180 p90= 6198 8003: n=173 p90= 6715 8007: n=125 p90=22242 ← still hot ``` A+B' redistributed load more evenly (114..189) but **8007 still has p90=22s**. ### 8007 deep-dive in A+B' ``` 8007: n=125, affinity=69 (55%), fallback=56 (45%), cache_hit_mean=lowest ``` Top-15 slow at 8007: - 7 of them are session **1313181** turns 9–14 (130k+ tokens each, agentic long context, ~50% cache hit) - Several others are cold-start turn-1 of large-prompt sessions - First two slow reqs arrived **0.7 s apart** — strong hint of concurrent picker race ### Iteration 3: race-condition fix **Diagnosis.** In `_handle_combined`: ```python chosen, best_idx, decision = pick_instance_unified_hybrid(...) # sync # ... sync breakdown updates ... return await _handle_local_request(...) # ← await yields here # THEN reservation happens ``` `return await async_func(...)` evaluates the async call (creates coroutine) and yields to the event loop **before** the coroutine body executes. The reservation (`chosen.pending_prefill_tokens += new`, etc.) lives at the top of `_handle_local_request`, so between the picker and the reservation there is a **window where another coroutine can run and re-pick the same instance**. When two big-prompt reqs arrive within milliseconds, both run pick → both pick the "free" 8007 → both yield → both reserve. Result: 8007 gets back-to-back 130k-token cold prefills, each waiting for the other. **Fix.** Move the reservation **before** the await, inside `_handle_combined`: ```python # Race fix: reserve atomically with pick, before any await. chosen.ongoing_tokens += input_length chosen.pending_prefill_tokens += estimated_new chosen.num_requests += 1 return await _handle_local_request(..., _pre_reserved=True) ``` `_handle_local_request` skips its own reservation when `_pre_reserved=True`. PD-sep paths are unaffected (they have their own reservation). **Run.** Pending — `unified_of13_lmw001_racefix_*`. Hypothesis: 8007 p90 drops to within ±3s of cluster median, since concurrent picks for the same "free" instance no longer happen. --- ## A+B'+RaceFix — results **Run.** `unified_of13_lmw001_racefix_20260527_1821/unified/`. | Metric (ms) | baseline | A+B' | A+B'+RF | Δ vs baseline | |---|---:|---:|---:|---:| | TTFT p50 | 520 | 485 | **478** | −8% | | **TTFT p90** | 8781 | 8287 | **7770** | **−11.5%** ✅ | | TTFT p99 | 47647 | 41876 | **42447** | −11% | | TPOT p90 | 17.8 | 17.5 | 18.0 | +1% | | E2E p90 | 19989 | 20625 | **18418** | −8% | | E2E p99 | 85841 | 77827 | **71227** | −17% | vs v3 reference: - **A+B'+RF TTFT p90 = 7770ms, vs v3 Mech B 9711ms → −20%** ✅ Long-tail counts (best across all variants): ``` > 5s: 170 → 158 > 30s: 41 → 33 >10s: 105 → 103 > 50s: 8 → 4 >20s: 65 → 57 >100s: 0 → 0 ``` ### Decision split — race fix mainly helped affinity | Decision | baseline | A+B'+RF | |---|---:|---:| | affinity p90 | 7011 | **5042** ✅ (−28%) | | fallback p90 | 12083 | 13944 (+15%) | The race-condition was hurting affinity decisions the most. When two concurrent reqs both stuck to a "free-looking" affinity instance, they piled up and inflated affinity's tail. Fix removed this collision. ### Per-worker ``` 8000: n=86 p90=11541 8004: n=150 p90=11906 8001: n=186 p90= 8307 8005: n=109 p90= 4798 8002: n=105 p90=14540 8006: n=183 p90= 6258 8003: n=264 p90= 3079 8007: n=131 p90=21850 ← still hot 8000 spread now 86..264 — race fix did disperse routing ``` ### 8007 still hot — but it's **workload-inherent, not a routing bug** Top sessions on 8007: ``` session 1279412: n=22 mean= 2208 max=18985 decisions: 91% affinity session 1313181: n=17 mean=17399 max=49089 decisions: 65% affinity session 1262354: n=15 mean= 622 max= 2325 decisions: 87% affinity session 1342921: n= 7 mean=17817 max=55589 decisions: 86% affinity session 1260327: n= 8 mean= 1636 max= 5382 decisions: 75% affinity session 1268831: n= 5 mean= 1443 max= 2673 decisions: 80% affinity ``` Sessions 1313181 and 1342921 are **long agentic contexts**: 100k–130k tokens per turn with ~50% cache hit (i.e. 50k new tokens prefill per turn). Even on a perfectly load-balanced instance, each turn is 7–15s of pure compute. Forcing these sessions to spread across instances would mean **cold prefill every turn (0% cache hit)** → each turn becomes 20–30s instead of 7–15s. Spreading is **net-negative**. → The 8007 p90=22s is the floor imposed by these sessions' structure, not by routing policy. Unified is at its ceiling for this workload. --- ## Final ranking and take-aways | Policy | TTFT p90 (ms) | Δ vs baseline | Notes | |---|---:|---:|---| | baseline unified (of=2.0) | 8781 | — | reference | | A (of=1.3) | 8730 | ≈0 | affinity p90 -17%, fallback p90 +27% | | A+B (of=1.3, lmw=0.01, BUG) | 8421 | −4% | 8007 hotspot from `*num_req` zeroing bug | | A+B' (formula fix) | 8287 | −5.6% | Bug fixed, still 8007 mild hotspot | | **A+B'+RaceFix** | **7770** | **−11.5%** ✅ | **Best unified variant** | | v3 fixed | 10828 | +23% | PD-sep migration, cache-blind picker | | v3 + Mech B | 9711 | +11% | PD-sep + cache-rich target picker | ### Conclusions 1. **Unified path beats v3 PD-sep on this workload by 20%+ TTFT p90.** PD-sep migration's fixed cost (src prefill + dst first-token waiting on loaded scheduler) outweighs any decode-time savings for short-output agentic turns. 2. **Three orthogonal fixes compound for a 11.5% TTFT p90 win:** - A (`overload_factor=1.3`): tighter affinity overflow → −0.6% but much cleaner affinity decisions (p90 -17%) - B' (`lmetric_decode_weight=0.01` with `max(num_req,1)`): decode-aware fallback → −3.5% - RaceFix (atomic reserve before await): kills concurrent-pick collisions → −5.6% 3. **Race condition was the biggest single hidden bug.** `return await async_func(...)` yields to the event loop **before** the body of `async_func` runs, so reservations done in the body don't take effect in time to deter concurrent picks. This affects ANY async dispatch with separate pick/reserve steps — worth checking other routing policies. 4. **8007 p90=22s is workload-inherent.** Sessions with 100k+ token turns at 50% cache hit cannot finish faster than 7–15s per turn regardless of routing. Forcing spread would hurt rather than help. 5. **Migration (v3) is not necessary** when unified routing is tuned well. Save the PD-sep mechanism for cases where it can be proven net-positive (e.g. very-long-output sessions on extremely overloaded prefill hosts) and use unified A+B'+RaceFix as the default. --- ## Direction A+B — run pending (Will be filled when `unified_of13_lmw001_*/unified/` finishes.)