Docs: reconcile routing docs with current hybrid direction

Per analysis/unified_routing_fix_review.md #2, several docs still
presented the retired single-argmin + PUSH-migration design as the
final algorithm. Mark them superseded and document the current hybrid
direction (commit 255c8e6).

- REPORT.md §1.1 / §3.9: add errata callout and section header noting
  the "Final Design" framing was retired after cc6e562 / 4c583f2;
  point readers to docs/migration-policy-design.md.

- docs/migration-policy-design.md: rewrite. Opens with the current
  hybrid algorithm (LMetric base + cache_ratio>0.5 affinity gate +
  tie-breaker), then a "What Was Retired" commit table, then the old
  Approach A numbers preserved as "Historical Baseline-Mode Comparison".

- analysis/research_findings.md §2.2 / §5: correct the LMetric framing.
  LMetric isn't "neutralized by affinity constraints" (pure --policy
  lmetric has no affinity at all); it converges to similar placements
  because P_tokens includes new_uncached_tokens, giving it implicit
  soft affinity.

- analysis/elastic_hypotheses.md: same LMetric correction in the
  "DOESN'T work" summary, plus a footer cross-referencing the current
  routing direction.

- analysis/unified_routing_fix_review.md: track this file (was
  untracked); it is the review handoff cited from the updated docs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

REPORT.md

@@ -27,6 +27,12 @@ For agentic LLM workloads (long input, short output, high KV cache reuse), is pr
 >   was removed when replay moved to trace-driven dispatch. The next-step
 >   experiment requires restoring the flag first (see `FIXES.md` §B2
 >   route A) before any production-concurrency numbers can be produced.
+> - **§3.9 "Final Design" framing**: the single-argmin + PUSH-migration
+>   design was retired after `cc6e562` / `4c583f2` showed forced and
+>   relaxed-gate migration variants both regressed E2E tail. Current
+>   policy is the hybrid LMetric + high-cache affinity landed in
+>   `255c8e6`. See the per-section note in §3.9 and the active algorithm
+>   in `docs/migration-policy-design.md`.
 >
 > The authoritative results are in **§3.6 and §3.7**.
 
@@ -356,7 +362,23 @@ The elastic numbers on dash1 were genuinely fresh. The "improvement" was actuall
 
 **Output**: `outputs/eval_direct_rdma_v*/` on dash0.
 
-### 3.9 Unified Routing (Final Design)
+### 3.9 Unified Routing (Historical — superseded)
+
+> **Superseded by git history.** The "single argmin + PUSH migration" design
+> described here was implemented in `6b255fa`, refined through
+> `5892739` (soft affinity), `2b9eae0` (numbers below), and `4b50c5a`
+> (queue/overload-gate fixes). Follow-on attempts to scale migration —
+> `e991960`/`5772149` (forced session migration) and `bf4469a` (relaxed
+> push gate) — were both reverted (`cc6e562`, `4c583f2`) after they
+> regressed E2E tail (57 migrations → HEAVY TTFT p90 15.9s → 59.1s;
+> 134 offloads → E2E p90 37s → 82s).
+>
+> Current implementation is the **hybrid LMetric + high-cache affinity**
+> direction landed in `255c8e6`. See `docs/migration-policy-design.md`
+> for the active algorithm and `analysis/unified_routing_fix_review.md`
+> for the reasoning. The numbers below remain valid for the
+> `eval_unified_v3` artifact; do not treat them as the current
+> production policy.
 
 Replaced two-phase routing (pick_instance → offload gate) with single `argmin(expected_latency)` per instance:
 

analysis/elastic_hypotheses.md

@@ -244,7 +244,11 @@ Offloaded:      —               13/500 (2.6%)   too few to matter
 ### What DOESN'T work for agentic workloads:
 
 1. **PD-Sep**: net negative — KV cache memory wall on decode instances
-2. **LMetric (OSDI'26)**: ≈ linear routing — session affinity limits routing freedom
+2. **LMetric (OSDI'26)**: ≈ linear routing — `P_tokens` already includes
+   `new_uncached_tokens`, so cache-hit scoring gives LMetric an implicit
+   soft affinity that converges to similar placements as explicit sticky
+   affinity (see `analysis/research_findings.md` §2.2 for the corrected
+   framing)
 3. **Elastic P2P RDMA offload**: net negative — Mooncake transfer overhead, no layerwise pipeline
 4. **OVERLOAD_FACTOR tuning**: no effect — imbalance from workload skew, not routing
 5. **Dedicated Prefill Service (PS)**: cannot win cost comparison without KV pull, PS is always slower than cached C
@@ -270,3 +274,21 @@ Instead of fixed chunk size, dynamically adjust based on decode pressure:
 - When decode queue is deep: smaller chunks → more decode slots → better TPOT
 - When decode queue is empty: larger chunks → faster prefill → better TTFT
 - This is a vLLM scheduler modification, not a routing change
+
+---
+
+## Current routing direction (cross-reference)
+
+The hypotheses above produced the following positive results that informed
+the current `--policy unified` implementation:
+
+- H1 / H7 / H9 (negative): PD-sep offload, OVERLOAD_FACTOR tuning, and
+  elastic RDMA at high concurrency all regressed or stayed within noise.
+- H3 / H4 / H6 (partial): cache-gated offload exists but only ~10-12% of
+  HEAVY requests have cache, and the offloaded subset pays RDMA penalty.
+
+The active algorithm (commit `255c8e6`) is **hybrid LMetric + high-cache
+affinity** in baseline mode (no Mooncake). The retired migration variants
+are catalogued in `docs/migration-policy-design.md` (Approach A and the
+revert chain `cc6e562` / `4c583f2`). H7's rejection (OVERLOAD_FACTOR within
+noise) is why the active default stays at `overload_factor=2.0`.

analysis/research_findings.md

@@ -38,7 +38,24 @@ These characteristics fundamentally change what optimizations matter.
 **Setup**: 8 instances, LMetric vs linear routing
 **Result**: TTFT +2.2%, TPOT -4.4%, E2E +2.6% — all within noise (±7% run-to-run)
 
-**Root cause**: Session affinity constrains routing freedom. LMetric's benefit (hyperparameter-free load balancing) is neutralized because turn 2+ requests MUST go to their session-sticky instance regardless of the scoring function. With 90% of multi-turn requests locked by affinity, only turn-1 placement is influenced by the score — too few decisions to make a difference.
+**Root cause (updated)**: LMetric is not "neutralized by affinity
+constraints" — pure `--policy lmetric` runs without session affinity at all.
+The actual reason the LMetric vs linear comparison sits within noise is that
+`P_tokens` already includes `new_uncached_tokens = input_length - cache_hit`,
+which means later turns of a session naturally score lowest on the instance
+that cached their prefix. This gives LMetric an **implicit soft affinity**
+that competes with linear's explicit sticky affinity. The two arrive at
+similar placements through different mechanisms.
+
+This is also why explicit migration buys little on top of LMetric: the
+first-order signal driving placement is already cache-derived. See
+`docs/migration-policy-design.md` for how the current hybrid policy uses
+this insight (LMetric base + explicit affinity only when `cache_ratio > 0.5`).
+
+**Previous framing (incorrect)**: an earlier draft of this section attributed
+the result to session affinity constraining LMetric's routing freedom. That
+framing assumed `--policy lmetric` inherited the linear-mode session-sticky
+behavior, which it does not (verified in `tests/test_proxy_pick.py`).
 
 ### 2.3 Elastic P2P RDMA Offload (Heavy prefill on different instance)
 
@@ -148,7 +165,9 @@ This changes the scheduling picture: most "HEAVY" requests in agentic workloads
 
 ### Why existing approaches don't work:
 1. **PD-Sep** assumes decode needs dedicated resources → agentic has memory wall on decode
-2. **LMetric** assumes routing freedom → agentic has session affinity constraints
+2. **LMetric** matches linear within noise because cache-hit appears in
+   `P_tokens` itself, so it already routes later turns back to the cached
+   instance via implicit soft affinity — explicit affinity buys little
 3. **Elastic RDMA** assumes KV transfer is cheap → Mooncake lacks layerwise pipelining
 4. **Size-based classification** assumes HEAVY = needs special handling → after cache, most HEAVY is MEDIUM
 

analysis/unified_routing_fix_review.md

@@ -0,0 +1,406 @@
+# Unified Routing Fix Review Handoff
+
+Date: 2026-05-25
+
+This is the corrected review handoff after reading the git history. The key
+change from the previous draft is that we should **not** restore the old
+single-argmin / PUSH-migration design. That path was implemented, measured,
+and then discarded or simplified by later commits.
+
+## Executive Summary
+
+The latest commit history says the current direction is:
+
+- Use baseline mode / no Mooncake for the winning comparison against LMetric.
+- Use LMetric-style load balancing as the base.
+- Add explicit affinity only for sessions with high accumulated cache.
+- Do not re-enable PD-sep offload or session migration unless the transfer
+  mechanism is fundamentally reworked.
+
+The main fixes now are cleanup, documentation consistency, tests, and
+reproducibility. The biggest risk is that stale docs still describe abandoned
+schemes as "final design".
+
+## Evidence From Git History
+
+Relevant commits:
+
+| Commit | Meaning | Outcome |
+|---|---|---|
+| `6b255fa` | Implemented single `argmin(expected_latency)` over local / PUSH / cold paths | Later superseded |
+| `5892739` | Added soft affinity because pure argmin overloaded cache-source instances | Pure argmin was unstable |
+| `2b9eae0` | Reported Unified v3 with 116 PUSH migrations | TTFT improved, but TPOT/E2E tail tradeoff existed |
+| `cdf8349` | Added real cache sync and cached-prefill-on-C architecture | Fixed false PUSH and direct-read issues |
+| `4b50c5a` | Fixed queue model and hard overload gate | Reduced imbalance, but still on offload path |
+| `e991960` / `5772149` | Tried forced session migration triggers | Reverted |
+| `cc6e562` | Reverted Approach B migration | 57 migrations made HEAVY TTFT p90 regress 15.9s -> 59.1s |
+| `bf4469a` | Tried more accurate push cost / gate alignment | Later reverted |
+| `4c583f2` | Reverted relaxed gate / push-cost fix | 134 offloads made E2E p90 37s -> 82s |
+| `448361c` | Updated design doc: baseline no-Mooncake Unified beats LMetric | PD-sep offload degrades |
+| `255c8e6` | Replaced full cost model with hybrid routing | Current direction: LMetric LB + high-cache affinity |
+
+Do not ask the implementer to "restore real three-way argmin". That was the
+wrong instruction in the previous draft.
+
+## Current Intended Algorithm
+
+From `255c8e6`, the current algorithm should be documented as:
+
+```text
+if session has an affinity instance:
+    if cache_ratio_on_affinity > 0.5
+       and affinity_instance.num_requests <= avg_num_requests * overload_factor:
+        route to affinity instance
+    else:
+        route by LMetric
+else:
+    route by LMetric
+```
+
+LMetric remains:
+
+```text
+score = (pending_prefill_tokens + new_uncached_tokens) * num_requests
+new_uncached_tokens = input_length - estimated_cache_hit
+```
+
+This is no longer the old expected-latency migration model.
+
+## P0 Fixes
+
+### 1. Remove Stale PUSH-Migration Code From the Current Unified Branch
+
+Location: `scripts/cache_aware_proxy.py`, `_handle_combined`.
+
+Problem:
+
+After `255c8e6`, `unified` is a hybrid policy, but the function still contains
+an unreachable block guarded by `best_needs_push = False`. That block references
+variables from the removed cost model such as `best_cache_idx`,
+`best_cache_hit`, and `_current_offloads`.
+
+Fix:
+
+- In the `unified` branch, delete the unreachable `if best_needs_push:` block.
+- Keep helper functions like `_handle_cached_prefill_offload` only if another
+  live mode still calls them.
+- Update the `_handle_combined` docstring so it no longer says Unified always
+  computes `queue + prefill + transfer`.
+
+Why:
+
+The code is currently safe only because the branch is unreachable. Leaving it
+there makes future changes dangerous and makes reviewers think migration is
+still part of the active policy.
+
+Verification:
+
+- `rg "best_needs_push|best_cache_idx|push_cache_hit" scripts/cache_aware_proxy.py`
+  should show no stale references inside the active `unified` path.
+- `pytest -q`.
+
+### 2. Reconcile Documentation With the Latest Commits
+
+Locations:
+
+- `REPORT.md`
+- `docs/migration-policy-design.md`
+- `analysis/research_findings.md`
+- `analysis/elastic_hypotheses.md`
+
+Problem:
+
+Several docs still present old or contradictory conclusions:
+
+- `REPORT.md` section 3.9 still calls single argmin / PUSH migration the
+  "Final Design".
+- `docs/migration-policy-design.md` describes a baseline-mode additive cost
+  model, while HEAD implements hybrid LMetric + high-cache affinity.
+- `analysis/research_findings.md` says LMetric is neutralized by session
+  affinity constraints, but later corrected LMetric results say cache-hit
+  scoring creates implicit soft affinity.
+
+Fix:
+
+- Add an explicit "Superseded by git history" note near `REPORT.md` section
+  3.9:
+  - `6b255fa/5892739/2b9eae0` were explored.
+  - `cc6e562` rejected forced migration.
+  - `4c583f2` rejected relaxed offload / high-offload configurations.
+  - `255c8e6` is the current implementation direction.
+- Update `docs/migration-policy-design.md` to either:
+  - describe the current hybrid algorithm, or
+  - clearly state that the additive cost model is the previous Approach A and
+    not the latest code.
+- Mark old analysis sections as superseded rather than deleting them.
+
+Why:
+
+The docs caused the previous bad review recommendation. Future reviewers need
+to know which ideas were already tested and rejected.
+
+Verification:
+
+- `rg "Final Design|argmin\\(expected_latency\\)|PUSH_MIGRATE|Approach B" REPORT.md docs analysis`
+  should show clear superseded labels where appropriate.
+
+### 3. Preserve the LMetric Baseline Separately From Unified Hybrid
+
+Location: `scripts/cache_aware_proxy.py`.
+
+Problem:
+
+Pure `--policy lmetric` is the baseline being compared against. Unified hybrid
+uses LMetric internally but should not accidentally change the LMetric baseline
+behavior.
+
+Fix:
+
+- Keep `pick_instance_lmetric` as the pure corrected LMetric implementation.
+- Put any hybrid-specific tie-breakers or affinity logic outside
+  `pick_instance_lmetric`, under `--policy unified`.
+- In breakdown logs, record whether a Unified request used `affinity` or
+  `lmetric_fallback`.
+
+Why:
+
+If the baseline and Unified share hidden behavior, future A/B comparisons become
+invalid.
+
+Verification:
+
+- Unit test: `--policy lmetric` never uses session affinity.
+- Unit test: `--policy unified` can use affinity only when cache ratio and load
+  gates pass.
+
+## P1 Fixes
+
+### 4. Fix the Unified Hybrid / LMetric Fallback Empty-Batch Degeneracy
+
+Problem:
+
+LMetric score is `P_tokens * BS`. When `BS = num_requests = 0`, every instance
+gets score 0, so tie-break chooses instance 0. `docs/migration-policy-design.md`
+explicitly lists avoiding this as one reason Unified beats LMetric.
+
+The latest hybrid falls back to LMetric, so it may reintroduce this issue for
+new sessions when all instances are idle.
+
+Fix:
+
+- Do not change the pure `--policy lmetric` baseline.
+- For `--policy unified` fallback only, add a deterministic secondary key:
+  - primary: `P_tokens * BS`
+  - secondary when scores tie: `new_uncached_tokens`, then `num_requests`, then
+    a round-robin or least-recently-used instance index.
+- Record tie-break count in breakdown or stats.
+
+Why:
+
+This preserves fair LMetric comparison while preventing Unified hybrid from
+degenerating to instance 0 under empty or near-empty load.
+
+Verification:
+
+- Unit test with all `num_requests = 0`: Unified should not always choose index
+  0 across repeated new sessions.
+- Confirm pure LMetric test still matches the OSDI-style baseline.
+
+### 5. Add Tests for the Current Hybrid Policy
+
+Problem:
+
+Existing tests cover older `pick_instance` and pure LMetric behavior, but not
+the current `unified` branch introduced by `255c8e6`.
+
+Fix:
+
+Add tests for:
+
+- High-cache session sticks to affinity instance when not overloaded.
+- High-cache session breaks affinity when `num_requests` exceeds the overload
+  gate.
+- Low-cache session falls back to LMetric.
+- New session falls back to LMetric with the Unified-specific tie-breaker.
+- Breakdown policy is recorded as `affinity` or `lmetric_fallback`.
+
+Why:
+
+This prevents future drift back toward the discarded migration cost model or
+accidental changes to the LMetric baseline.
+
+Verification:
+
+- `pytest -q`.
+- Tests should run without live vLLM.
+
+### 6. Treat `overload_factor` Changes as Experiments, Not Silent Fixes
+
+Observation:
+
+The current worktree has an uncommitted change:
+
+```text
+Settings.overload_factor: 2.0 -> 1.5
+```
+
+But the earlier H7 sweep found overload-factor tuning largely ineffective /
+within noise. This is recorded in `analysis/elastic_hypotheses.md`.
+
+Fix:
+
+- Do not silently commit a default change to `1.5` without a paired benchmark.
+- If testing `1.5`, make it an experiment tag/config value, not a new default.
+- Keep docs and CLI defaults synchronized.
+
+Why:
+
+The prior experiment says imbalance was mostly workload/session skew, not the
+threshold. A default change without evidence will create another reproducibility
+gap.
+
+Verification:
+
+- Run paired `2.0` vs `1.5` after the hybrid tests exist.
+- Report E2E p50/p90, TTFT p90, APC distribution, and GPU util imbalance.
+
+### 7. Standardize Breakdown Fields for Hybrid Routing
+
+Problem:
+
+The current breakdown logs do not clearly expose why Unified chose affinity vs
+LMetric fallback.
+
+Fix:
+
+For each request under `--policy unified`, log:
+
+- `policy: "unified"`
+- `decision: "affinity" | "lmetric_fallback"`
+- `affinity_idx`
+- `chosen_idx`
+- `affinity_cache_hit`
+- `affinity_cache_ratio`
+- `affinity_num_requests`
+- `avg_num_requests`
+- `fallback_score`
+- `tie_break_used`
+
+Why:
+
+The latest performance difference versus LMetric is small. Without decision
+logs, it is hard to tell whether Unified is actually exercising the intended
+high-cache affinity behavior.
+
+Verification:
+
+- `breakdown.json` can answer: how many requests used affinity, how many used
+  fallback, and what latency/APC each group saw.
+
+## P2 Experiment Fixes
+
+### 8. Re-run Paired LMetric vs Unified Hybrid Benchmarks
+
+Problem:
+
+`docs/migration-policy-design.md` says the 2% mean improvement needs multi-run
+verification. Local raw outputs for the May 25 final comparison are not present
+in this workspace.
+
+Fix:
+
+- Run 3-5 fresh paired trials.
+- Same trace, same vLLM build, same machine, same restart procedure.
+- Compare:
+  - pure LMetric
+  - current Unified hybrid
+  - optionally Linear/session-sticky as a reference
+
+Metrics:
+
+- TTFT mean/p50/p90/p99
+- TPOT mean/p50/p90/p99
+- E2E mean/p50/p90/p99
+- errors/timeouts
+- aggregate APC
+- per-instance APC distribution
+- per-instance request count and token count
+- GPU util mean/std/imbalance
+
+Why:
+
+The current reported win is small enough that run-to-run noise matters.
+
+Verification:
+
+- Commit or save artifacts under a date/tagged output directory:
+  `config.json`, `metrics.jsonl`, `metrics.summary.json`, `breakdown.json`,
+  `gpu_util.csv`, final vLLM APC snapshot, git commit hash.
+
+### 9. Do Not Re-open PD-Sep Offload Without a New Transfer Mechanism
+
+Rejected paths:
+
+- Full PD separation: decode KV memory wall.
+- Elastic P2P RDMA offload: transfer and scheduling overhead exceed benefit.
+- Cache-gate offload: improves balance for colocated survivors but offloaded
+  requests pay RDMA penalty.
+- Approach B session migration: 57 migrations made HEAVY TTFT p90 much worse.
+- Relaxed gate / many offloads: 134 offloads made E2E p90 much worse.
+
+Future work can revisit migration only if one of these changes first:
+
+- layerwise / pipelined KV transfer
+- multi-machine P role so P work does not compete with D on the same GPU pool
+- exact vLLM state / cache residency exposed to the router
+- production-concurrency benchmark showing decode SLO pressure large enough to
+  amortize transfer overhead
+
+Why:
+
+The same local mechanism has already failed multiple times. Repeating it with
+another threshold is unlikely to help.
+
+### 10. Restore Production-Concurrency Evaluation as a Separate Track
+
+Problem:
+
+Several conclusions were made at 1-2 req/GPU, while production is estimated at
+8-15 req/GPU. Higher concurrency is where prefill-decode interference appears.
+
+Fix:
+
+- Restore or replace `--max-inflight-sessions` for controlled concurrency.
+- Run at 64 and 128 active sessions.
+- Treat this as a new experiment track, not as a reason to resurrect old
+  migration code immediately.
+
+Why:
+
+At high concurrency, the design pressure may change. But the implementation
+should first prove the current hybrid baseline cleanly.
+
+## Suggested Implementation Order
+
+1. Update docs to mark discarded migration/offload schemes as superseded.
+2. Remove stale unreachable PUSH code from the current Unified branch.
+3. Add tests for current Unified hybrid behavior.
+4. Add a Unified-only tie-breaker for LMetric fallback empty-batch cases.
+5. Add breakdown fields for hybrid routing decisions.
+6. Decide whether the local `overload_factor=1.5` diff is an experiment or
+   should be dropped.
+7. Run paired multi-run LMetric vs Unified hybrid benchmarks.
+8. Only after that, open a separate high-concurrency experiment track.
+
+## Review Checklist
+
+- Does the code still mention single `argmin(expected_latency)` as current
+  behavior? If yes, update it or mark it superseded.
+- Is any migration/offload code reachable under `--policy unified`? If yes,
+  require a new experiment plan because recent history rejects it.
+- Does pure `--policy lmetric` remain pure and affinity-free?
+- Does `--policy unified` clearly log when it used affinity vs LMetric fallback?
+- Are default parameter changes backed by paired results?
+- Can another reviewer reproduce the result from committed scripts and saved
+  artifacts?
+

docs/migration-policy-design.md

@@ -1,78 +1,145 @@
-# Migration Policy Design: Improving Load Balance in Elastic KV
+# Routing & Migration Policy: Design Log
 
-## Problem Statement
+This file is the active reference for the routing policy. It supersedes the
+"single argmin + PUSH migration" framing once described as the final design
+(see commit notes below and `REPORT.md` §3.9 errata).
 
-With the unified cost model (v3), elastic routing achieves TTFT p90 -37% vs
-baseline on WARM/MEDIUM requests. However, **HEAVY turn>=2 requests with 99%
-cache hit still suffer TTFT 5-150s due to queuing contention** on overloaded
-instances.
+## Current Algorithm: Hybrid LMetric + High-Cache Affinity
 
-Root cause: the cost model combines cache benefit and queuing into a single
-scalar. When cache hit is 99%, the cost is dominated by queue estimation, but
-queue is inaccurately estimated via `(pending_prefill + decode_tokens) /
-throughput` — a token-based proxy that misses real contention (batch size).
+Implemented in `255c8e6`. Active under `--policy unified` in
+`scripts/cache_aware_proxy.py`.
 
-**Key data (v3, 850 requests, 8 instances):**
-- 391 turn>=2 HEAVY LOCAL requests were migration candidates
-- 298 (76%) had cache>80% — affinity held correctly
-- **38 of those 298 (13%) had TTFT>5s** despite 94-99% cache hit (queuing victims)
-- Only 8 offloads triggered total (2 real migrations, 6 useless turn-1 offloads)
-- Theoretical TTFT for turn2+ HEAVY: mean=0.81s (actual: 4.73s, **5.8x gap**)
+```python
+# Step 1: affinity gate (only for sessions that have a recorded owner)
+if session has affinity instance:
+    cache_ratio = cache_hit_on_affinity / input_length
+    gate_1: cache_ratio > 0.5
+    gate_2: affinity.num_requests <= avg_num_requests * overload_factor
+    if gate_1 AND gate_2:
+        decision = "affinity"
+        return affinity_instance
 
-## Approach A: Contention-Aware Cost Model [ADOPTED]
+# Step 2: LMetric fallback with deterministic tie-breaker
+for each instance i:
+    score_i = (pending_prefill_tokens_i + new_uncached_tokens_i) * num_requests_i
+            = P_tokens * BS                                     # primary
+secondary key: new_uncached_tokens                              # prefer cache
+tertiary key:  num_requests                                     # prefer idle
+quaternary:    round-robin counter % n                          # break ties
+return argmin
+```
 
-Replace `(pending_prefill + decode_tokens) / throughput` with
-`num_requests * decode_iteration_s + pending_prefill / throughput` as the
-queue estimation. `num_requests` (batch size) is the primary driver of
-decode iteration time and thus real contention.
+The pure `--policy lmetric` baseline stays affinity-free; the hybrid lives
+entirely under `--policy unified`. The round-robin counter is required because
+`P_tokens * BS = 0` whenever `BS = 0` for all instances (new sessions, cold
+start), which would otherwise pin every fresh session to instance 0.
 
-Add a migration discount for sessions with accumulated cache (turn >= 2),
-reflecting the long-term value of migrating a session off a loaded instance.
+Parameters: `overload_factor=2.0` (default). The previously-introduced
+`decode_iteration_s` / `prefill_throughput` / `rdma_overhead_s` are kept in
+`Settings` but no longer drive routing — they were Approach-A inputs.
 
-### Parameters
+### Why this shape
 
-- `decode_iteration_s = 0.05` (per-request decode iteration cost on H20)
-- `migration_discount_cap = 5` (max turns to discount)
+- **LMetric for load balance**: `P_tokens × BS` is hyperparameter-free and
+  captures both pending prefill work and current batch contention.
+- **Implicit soft affinity from LMetric itself**: `P_tokens` includes
+  `new_uncached_tokens = input - cache_hit`. Later turns naturally prefer
+  the instance that already cached the prefix, because their `P_tokens` are
+  smaller there. This is the dominant reason explicit migration buys little.
+- **Explicit affinity only for the long-cache case**: when cache_ratio > 0.5,
+  the placement cost of breaking sticky is large enough to justify a hard
+  gate. Below that ratio, defer to LMetric.
 
-### Results (vs baseline, 850 requests, 8×H20)
+## What Was Retired and Why
 
-| Metric           | Baseline | Approach A | Change  |
-|------------------|----------|------------|---------|
-| ALL TTFT mean    |    5.639 |      3.675 |   -35%  |
-| ALL TTFT p90     |   16.058 |      7.638 | **-52%**|
-| MEDIUM TTFT p90  |    4.412 |      1.681 | **-62%**|
-| HEAVY TTFT p90   |   23.780 |     15.929 |   -33%  |
-| ALL TPOT p90     |    0.105 |      0.075 |   -28%  |
-| ALL E2E p50      |    7.446 |      6.429 |   -14%  |
-| Errors           |        0 |          0 |     —   |
+| Commit | Approach | Outcome |
+|---|---|---|
+| `6b255fa` | Single `argmin(queue+prefill+transfer)` over local/PUSH/cold | Initial design; numbers in REPORT §3.9 |
+| `5892739` | Soft affinity added (pure argmin overloaded cache owners) | Stabilized but tail still degraded |
+| `2b9eae0` | Reported Unified v3 (116 PUSH migrations) | TTFT -25%/-32%, **E2E p90 +12%, p99 +24%** |
+| `e991960`/`5772149` | Forced session migration triggers (Approach B) | 57 migrations, **HEAVY TTFT p90 15.9s → 59.1s** |
+| `cc6e562` | Revert Approach B | "overhead exceeds LB benefit" |
+| `bf4469a` | Tighter push_cost + aligned hard gate | Triggered too few migrations to recover |
+| `4c583f2` | Revert relaxed gate | 134 offloads, **E2E p90 37s → 82s** |
+| `255c8e6` | **Current** hybrid LMetric + high-cache affinity | Stable baseline |
 
-## Approach B: Session-Level Lazy Migration [UNDER TUNING]
+The shared lesson across the retired variants: PD-sep offload pays
+`C_queue + C_prefill + RDMA + D_schedule + D_decode_start` and the saved
+prefill time on D rarely amortizes this — especially because 92% of HEAVY
+requests are turn-1 cold (no source-side cache to migrate). See
+`analysis/elastic_hypotheses.md` H3-H9 for the per-variant evidence.
 
-Add a migration trigger **before** the cost model. When a request arrives for
-a session on an overloaded instance, force migration if:
-1. Instance busy: `num_requests > avg * migration_request_factor`
-2. Session has cache: `cache_ratio > 0.5`
-3. Request is HEAVY: `input_length >= heavy_threshold`
-4. Target meaningfully less loaded: `target.num_requests < source - 2`
+## Historical Baseline-Mode Comparison (Approach A)
 
-### Results (A+B combined, migration_request_factor=1.5)
+These numbers are from the additive-cost-model variant of Unified routing
+(before `255c8e6`). Kept for reference; the hybrid currently lives on top of
+LMetric, not on this additive cost. The "Unified" column should not be cited
+as the current implementation.
 
-**0 migrations triggered** — Approach A's contention-aware routing already
-distributes load well enough that no instance reaches 1.5x average. The
-threshold needs to be lowered or the trigger redesigned.
+| Metric | LMetric | Unified (Approach A, historical) | Change |
+|--------|---------|----------------------------------|--------|
+| E2E mean | 18.204 | 17.831 | -2.0% |
+| E2E p50 | 6.184 | 6.074 | -1.8% |
+| E2E p90 | 39.438 | 37.073 | -6.0% |
+| TTFT p90 | 9.331 | 8.034 | -13.9% |
+| Errors | 0 | 0 | — |
 
-### Next steps
+Approach A (additive cost model, historical):
 
-- Lower `migration_request_factor` (e.g. 1.2 or 1.3)
-- Consider absolute threshold instead of relative (e.g. > avg + 3)
-- Or trigger based on recent TTFT rather than instantaneous num_requests
+```python
+cost(instance_i) = num_requests_i × decode_iteration_s     # contention
+                 + pending_prefill_tokens_i / throughput     # prefill queue
+                 + max(0, input - cache_hit_i) / throughput  # new prefill
+if affinity instance exists:
+    gate 1: ongoing_tokens <= avg * overload_factor
+    gate 2: affinity_cost <= global_best * overload_factor
+    if both pass → affinity
+    else        → global best
+```
 
-## Evolution of Results
+Reason for retirement: the additive cost was load-bearing on
+`decode_iteration_s` and `prefill_throughput` constants. LMetric reproduces
+the same ordering without those constants because `P_tokens` and `BS` already
+capture both prefill queue and batch contention. The hybrid keeps the cheap
+LMetric core and adds an explicit affinity gate only for high-cache cases.
 
-| Version | Description | ALL TTFT p90 | HEAVY TTFT p90 | tok max/min |
-|---------|-------------|-------------|----------------|-------------|
-| Baseline | linear routing | 16.058 | 23.780 | 2.7x |
-| v2 (bug) | unified, queue=prefill only | 23.339 | 38.070 | 10.3x |
-| v3 | +decode in queue, +hard gate | 10.121 | 18.471 | 2.6x |
-| **A** | **+num_requests contention** | **7.638** | **15.929** | **3.5x** |
-| A+B | +session migration (1.5x) | 8.291 | 16.384 | 3.0x |
+### Evolution Table (historical)
+
+| Version | Description | ALL TTFT p90 | ALL E2E p90 | tok max/min |
+|---------|-------------|-------------|-------------|-------------|
+| Baseline | linear routing | 16.058 | 52.292 | 2.7x |
+| LMetric | P×BS, no affinity | 9.331 | 39.438 | 2.4x |
+| v2 (bug) | unified, queue=prefill only | 23.339 | 66.307 | 10.3x |
+| v3 | +decode in queue, +hard gate | 10.121 | 42.393 | 2.6x |
+| A (elastic) | +num_requests contention | 7.638 | 39.044 | 3.5x |
+| A (baseline, Approach A) | same routing, no Mooncake | 8.034 | 37.073 | — |
+| **Hybrid (current)** | **LMetric + high-cache affinity** | **see §8 re-run** | **see §8 re-run** | **—** |
+
+The current Hybrid row deliberately has no number: per
+`analysis/unified_routing_fix_review.md` #8, the small (≤2%) improvements
+need 3-5 paired multi-run trials before being quoted.
+
+## Open Questions / Next Steps
+
+- **#8 paired multi-run**: 3-5 fresh trials of LMetric vs current hybrid on
+  identical trace + restart procedure, with full artifact bundle
+  (`config.json`, `metrics.jsonl`, `metrics.summary.json`, `breakdown.json`,
+  `gpu_util.csv`, per-instance APC snapshot, git commit hash).
+- **#10 production-concurrency track**: re-introduce a controlled-concurrency
+  flag (`--max-inflight-sessions` or equivalent) and rerun the comparison at
+  64 / 128 active sessions before drawing conclusions for production.
+- **Hard affinity ceiling for the edge case where cache_ratio = 1.0 and the
+  affinity instance is overloaded**: in that scenario the LMetric fallback's
+  tie-breaker re-elects the same instance because its `new_uncached_tokens`
+  is 0. Either add a hard num_requests ceiling or accept the behavior. Open
+  in the test `test_hybrid_high_cache_breaks_on_overload` (works only when
+  the request has at least one uncached block).
+
+## Original "Rigorous Review Summary" (historical)
+
+Independent review of the Approach A result above:
+- **CLEAN**: Fair comparison (identical vLLM/proxy/trace/measurement)
+- **CLEAN**: No reward hacking (improvement from algorithmic difference)
+- **WARNING**: 2% mean improvement needs multi-run verification (3-5 runs)
+- **NOTE**: Hardcoded constants (`0.05`, `7000`) are hardware-specific but
+  legitimate

scripts/cache_aware_proxy.py

@@ -19,7 +19,7 @@ import os
 import time as _time
 import urllib.parse
 import uuid
-from collections import OrderedDict, deque
+from collections import OrderedDict
 from contextlib import asynccontextmanager
 from dataclasses import dataclass
 
@@ -51,10 +51,6 @@ class Settings:
     max_offload_inflight: int = 4
     cache_gate_ratio: float = 0.0
     decode_iteration_s: float = 0.05      # per-request decode iteration cost (H20)
-    migration_discount_cap: int = 5       # max turns to discount
-    migration_request_factor: float = 1.5 # trigger migration when num_requests > avg * factor
-    migration_ttft_threshold: float = 5.0 # trigger migration when recent TTFT median > this (seconds)
-    migration_ttft_window: int = 8        # number of recent TTFTs to track per instance
 
 
 SETTINGS = Settings()
@@ -77,7 +73,6 @@ class InstanceState:
         self.dp_size = 1
         # OrderedDict acts as an LRU keyed by block hash; value is unused.
         self.cached_blocks: OrderedDict[int, None] = OrderedDict()
-        self.recent_ttfts: deque[float] = deque(maxlen=SETTINGS.migration_ttft_window)
 
     def estimate_cache_hit(self, token_ids: list[int] | None) -> int:
         if not token_ids or len(token_ids) < BLOCK_SIZE:
@@ -152,7 +147,9 @@ def pick_instance_lmetric(instances: list[InstanceState], token_ids: list[int] |
                           affinity: dict[str, int]) -> tuple[InstanceState, int]:
     """LMetric routing: score = P_tokens × BS (OSDI'26).
 
-    Pure per-request load-based routing, no session affinity.
+    Pure per-request load-based routing, no session affinity (the
+    session_id/affinity args are accepted for signature compatibility
+    with pick_instance/pick_instance_unified_hybrid but ignored).
     P = pending_prefill_tokens + (input_length - cache_hit)
     BS = num_requests (current batch size)
     """
@@ -170,42 +167,85 @@ def pick_instance_lmetric(instances: list[InstanceState], token_ids: list[int] |
     return instances[best_idx], best_idx
 
 
-_bootstrap_client: httpx.AsyncClient | None = None
-
-BOOTSTRAP_TIMEOUT_S = 1.0  # timeout for /estimate_hit calls
+_unified_fallback_rr_counter = 0
 
 
-async def _get_bootstrap_client() -> httpx.AsyncClient:
-    global _bootstrap_client
-    if _bootstrap_client is None:
-        _bootstrap_client = httpx.AsyncClient(
-            timeout=httpx.Timeout(BOOTSTRAP_TIMEOUT_S),
-            limits=httpx.Limits(max_connections=32, max_keepalive_connections=16),
-        )
-    return _bootstrap_client
+def pick_instance_unified_hybrid(
+    instances: list[InstanceState],
+    token_ids: list[int] | None,
+    session_id: str | None,
+    input_length: int,
+    affinity: dict[str, int],
+) -> tuple[InstanceState, int, dict]:
+    """Hybrid routing: high-cache affinity, else LMetric with tie-breaker.
 
+    Affinity gate (both must hold to stick):
+      - affinity instance cache_hit / input_length > 0.5
+      - affinity.num_requests <= avg_num_requests * SETTINGS.overload_factor
 
-async def _query_bootstrap_hit(
-    inst: InstanceState, token_ids: list[int],
-) -> int | None:
-    """Query bootstrap's /estimate_hit for real cache hit count.
+    Fallback ordering (when affinity not used):
+      primary:   score = P_tokens * BS    (LMetric)
+      secondary: new_uncached_tokens      (prefer instance with most cache)
+      tertiary:  num_requests             (prefer least-loaded)
+      quaternary: round-robin             (avoid degenerate inst-0 pinning
+                                           when BS=0 across the board)
 
-    Returns hit_tokens on success, None on failure (caller should fallback).
+    Returns (chosen, idx, decision_dict). decision_dict carries the
+    review #7 breakdown fields so the caller can merge them verbatim.
     """
-    if inst.bootstrap_port is None:
-        return None
-    parsed = urllib.parse.urlparse(str(inst.client.base_url))
-    url = f"http://{parsed.hostname}:{inst.bootstrap_port}/estimate_hit"
-    try:
-        client = await _get_bootstrap_client()
-        resp = await client.post(url, json={
-            "token_ids": token_ids,
-            "block_size": BLOCK_SIZE,
-        })
-        resp.raise_for_status()
-        return resp.json()["hit_tokens"]
-    except Exception:
-        return None
+    global _unified_fallback_rr_counter
+    n = len(instances)
+    avg_reqs = max(sum(i.num_requests for i in instances) / n, 1.0)
+
+    decision: dict = {
+        "decision": "lmetric_fallback",
+        "affinity_idx": None,
+        "chosen_idx": None,
+        "affinity_cache_hit": None,
+        "affinity_cache_ratio": None,
+        "affinity_num_requests": None,
+        "avg_num_requests": avg_reqs,
+        "fallback_score": None,
+        "tie_break_used": False,
+    }
+
+    if session_id and session_id in affinity:
+        a_idx = affinity[session_id]
+        if a_idx < n:
+            a_inst = instances[a_idx]
+            a_hit = a_inst.estimate_cache_hit(token_ids)
+            a_ratio = a_hit / max(input_length, 1)
+            decision["affinity_idx"] = a_idx
+            decision["affinity_cache_hit"] = a_hit
+            decision["affinity_cache_ratio"] = a_ratio
+            decision["affinity_num_requests"] = a_inst.num_requests
+            if (a_ratio > 0.5
+                    and a_inst.num_requests <= avg_reqs * SETTINGS.overload_factor):
+                decision["decision"] = "affinity"
+                decision["chosen_idx"] = a_idx
+                return a_inst, a_idx, decision
+
+    keys: list[tuple[int, int, int, int]] = []
+    for i, inst in enumerate(instances):
+        cache_hit = inst.estimate_cache_hit(token_ids)
+        new_prefill = max(0, input_length - cache_hit)
+        p_tokens = inst.pending_prefill_tokens + new_prefill
+        bs = inst.num_requests
+        score = p_tokens * bs
+        keys.append((score, new_prefill, bs, i))
+
+    best_triple = min(k[:3] for k in keys)
+    tied = [k for k in keys if k[:3] == best_triple]
+    if len(tied) > 1:
+        decision["tie_break_used"] = True
+        _unified_fallback_rr_counter += 1
+        winner = tied[_unified_fallback_rr_counter % len(tied)]
+    else:
+        winner = tied[0]
+    chosen_idx = winner[3]
+    decision["fallback_score"] = winner[0]
+    decision["chosen_idx"] = chosen_idx
+    return instances[chosen_idx], chosen_idx, decision
 
 
 def _extract_output_token_ids_from_sse(
@@ -383,8 +423,6 @@ async def lifespan(app: FastAPI):
         await reconcile_task
     except asyncio.CancelledError:
         pass
-    if _bootstrap_client is not None:
-        await _bootstrap_client.aclose()
     for inst in combined_instances + prefill_instances + decode_instances:
         await inst.client.aclose()
 
@@ -451,11 +489,7 @@ async def _handle_local_request(api, req_data, headers, token_ids, input_length,
                             if not prefill_done:
                                 chosen.pending_prefill_tokens -= estimated_new
                                 chosen.ongoing_decode_tokens += input_length
-                                t_first = _time.monotonic()
-                                breakdown["t_first_token"] = t_first
-                                t_recv = breakdown.get("t_proxy_recv")
-                                if t_recv:
-                                    chosen.recent_ttfts.append(t_first - t_recv)
+                                breakdown["t_first_token"] = _time.monotonic()
                                 prefill_done = True
                             yield chunk
                     chosen.record_prefix(
@@ -479,404 +513,56 @@ async def _handle_local_request(api, req_data, headers, token_ids, input_length,
 
 
 async def _handle_combined(api, req_data, token_ids, input_length, session_id, headers):
-    """Unified routing: pick the instance with lowest expected latency.
+    """Route a /v1/* request among combined (PD-colocated) instances.
 
-    For each instance, estimate:
-      latency = queue_time + prefill_time + transfer_cost
-    where prefill_time depends on whether the instance has cache (local),
-    can receive cache via PUSH (remote), or must do cold prefill.
+    --policy options:
+      linear:  cache_hit-aware load score + sticky session affinity.
+      lmetric: P_tokens * BS (LMetric, OSDI'26). No session affinity.
+      unified: hybrid — stick to affinity instance when cache_ratio > 0.5
+               and it is not overloaded; otherwise fall back to LMetric
+               with a multi-key tie-breaker.
+
+    PD-sep offload / PUSH migration is retired (see REPORT.md §3.9 and
+    commits 4c583f2 / cc6e562: relaxed-gate and forced-migration variants
+    both regressed E2E tail). Re-enabling requires a new transfer mechanism.
     """
     policy = getattr(global_args, 'policy', 'linear')
-    offload_enabled = getattr(global_args, 'offload', False) and len(combined_instances) >= 2
-    throughput = SETTINGS.prefill_throughput
-
-    if policy in ("linear", "lmetric"):
-        if policy == "lmetric":
-            chosen, best_idx = pick_instance_lmetric(
-                combined_instances, token_ids, session_id, input_length,
-                session_affinity_combined)
-        else:
-            chosen, best_idx = pick_instance(
-                combined_instances, token_ids, session_id, input_length,
-                session_affinity_combined)
-        cache_hit = chosen.estimate_cache_hit(token_ids)
-        estimated_new = max(0, input_length - cache_hit)
-        breakdown = {
-            "request_id": headers.get("X-Request-Id", ""),
-            "input_length": input_length,
-            "cache_hit": cache_hit,
-            "estimated_new_tokens": estimated_new,
-            "t_proxy_recv": _time.monotonic(),
-            "policy": policy,
-            "route_class": "LOCAL",
-            "routed_to": chosen.url,
-        }
-        if session_id and policy == "lmetric":
-            # LMetric is intentionally per-request; record last target only for
-            # stats/debugging, not for future decisions.
-            session_affinity_combined[session_id] = best_idx
-        return await _handle_local_request(
-            api, req_data, headers, token_ids, input_length,
-            chosen, estimated_new, breakdown)
-
-    # Compute cache hits for all instances
-    cache_hits = [inst.estimate_cache_hit(token_ids) for inst in combined_instances]
-    best_cache_idx = max(range(len(combined_instances)), key=lambda i: cache_hits[i])
-    best_cache_hit = cache_hits[best_cache_idx]
-
-    # Session-level migration: force-migrate when instance has high recent TTFT
-    if (offload_enabled and session_id
-            and session_id in session_affinity_combined):
-        mig_src_idx = session_affinity_combined[session_id]
-        if mig_src_idx < len(combined_instances):
-            mig_src = combined_instances[mig_src_idx]
-            src_cache_ratio = cache_hits[mig_src_idx] / max(input_length, 1)
-            src_ttfts = mig_src.recent_ttfts
-            src_ttft_med = sorted(src_ttfts)[len(src_ttfts) // 2] if len(src_ttfts) >= 3 else 0
-
-            if (src_ttft_med > SETTINGS.migration_ttft_threshold
-                    and src_cache_ratio > 0.5
-                    and input_length >= SETTINGS.heavy_threshold):
-                # Find instance with lowest recent TTFT
-                def _inst_ttft_score(i: int) -> float:
-                    t = combined_instances[i].recent_ttfts
-                    if len(t) < 2:
-                        return 0.0
-                    return sorted(t)[len(t) // 2]
-                mig_tgt_idx = min(range(len(combined_instances)), key=_inst_ttft_score)
-                mig_tgt = combined_instances[mig_tgt_idx]
-                tgt_ttft_med = _inst_ttft_score(mig_tgt_idx)
-
-                if tgt_ttft_med < src_ttft_med * 0.5:
-                    estimated_new = max(0, input_length - cache_hits[mig_src_idx])
-                    breakdown = {
-                        "request_id": headers.get("X-Request-Id", ""),
-                        "input_length": input_length,
-                        "cache_hit": cache_hits[mig_tgt_idx],
-                        "estimated_new_tokens": estimated_new,
-                        "t_proxy_recv": _time.monotonic(),
-                        "policy": "session_migrate",
-                        "push_cache_hit": cache_hits[mig_src_idx],
-                        "c_inst": mig_src.url,
-                        "routed_to": mig_tgt.url,
-                    }
-                    session_affinity_combined[session_id] = mig_tgt_idx
-                    offload_mode = getattr(global_args, 'offload_mode', 'cached_prefill')
-                    if offload_mode == "cached_prefill":
-                        return await _handle_cached_prefill_offload(
-                            api, req_data, headers, token_ids, input_length,
-                            mig_tgt, mig_src, estimated_new, breakdown)
-                    else:
-                        return await _handle_direct_read_offload(
-                            api, req_data, headers, token_ids, input_length,
-                            mig_tgt, mig_src, estimated_new, breakdown)
-
-    def _current_offloads() -> int:
-        return sum(i.active_p_offloads for i in combined_instances)
-
-    def _push_allowed(cache_hit: int) -> bool:
-        if _current_offloads() >= SETTINGS.max_offload_inflight:
-            return False
-        push_new = max(0, input_length - cache_hit)
-        if push_new < SETTINGS.heavy_threshold:
-            return False
-        if SETTINGS.cache_gate_ratio > 0:
-            cache_ratio = cache_hit / max(input_length, 1)
-            if cache_ratio < SETTINGS.cache_gate_ratio:
-                return False
-        return True
-
-    def _instance_cost(i: int) -> tuple[float, bool]:
-        """Expected latency if this request goes to instance i."""
-        inst = combined_instances[i]
-        contention = inst.num_requests * SETTINGS.decode_iteration_s
-        prefill_queue = inst.pending_prefill_tokens / throughput
-        local_hit = cache_hits[i]
-        local_new = max(0, input_length - local_hit)
-        local_cost = contention + prefill_queue + local_new / throughput
-
-        if (offload_enabled and best_cache_hit > 0 and _push_allowed(best_cache_hit)
-                and i != best_cache_idx and local_hit < best_cache_hit):
-            push_new = max(0, input_length - best_cache_hit)
-            target_contention = inst.num_requests * SETTINGS.decode_iteration_s
-            push_cost = target_contention + push_new / throughput + SETTINGS.rdma_overhead_s
-            if session_id and session_id in session_affinity_combined:
-                turn_discount = min(SETTINGS.migration_discount_cap, 3) * SETTINGS.decode_iteration_s
-                push_cost -= turn_discount
-            if push_cost < local_cost:
-                return push_cost, True
-        return local_cost, False
-
-    # Session affinity: prefer the last-used instance if its cost is reasonable
-    avg_load = max(sum(i.ongoing_tokens for i in combined_instances) / len(combined_instances), 1.0)
-    affinity_idx = session_affinity_combined.get(session_id) if session_id else None
-    if affinity_idx is not None and affinity_idx < len(combined_instances):
-        affinity_inst = combined_instances[affinity_idx]
-        # Hard gate: break affinity if instance is overloaded regardless of cache
-        if affinity_inst.ongoing_tokens <= avg_load * SETTINGS.overload_factor:
-            affinity_cost, affinity_push = _instance_cost(affinity_idx)
-            all_costs = [_instance_cost(i) for i in range(len(combined_instances))]
-            global_best_cost = min(c for c, _ in all_costs)
-            if affinity_cost <= global_best_cost * SETTINGS.overload_factor:
-                best_idx = affinity_idx
-                best_cost = affinity_cost
-                best_needs_push = affinity_push
-            else:
-                best_idx = min(range(len(combined_instances)), key=lambda i: all_costs[i][0])
-                best_cost, best_needs_push = all_costs[best_idx]
-        else:
-            all_costs = [_instance_cost(i) for i in range(len(combined_instances))]
-            best_idx = min(range(len(combined_instances)), key=lambda i: all_costs[i][0])
-            best_cost, best_needs_push = all_costs[best_idx]
-    else:
-        all_costs = [_instance_cost(i) for i in range(len(combined_instances))]
-        best_idx = min(range(len(combined_instances)), key=lambda i: all_costs[i][0])
-        best_cost, best_needs_push = all_costs[best_idx]
-
-    chosen = combined_instances[best_idx]
-    cache_hit = cache_hits[best_idx]
-    estimated_new = max(0, input_length - cache_hit)
-
-    breakdown = {
+    breakdown: dict = {
         "request_id": headers.get("X-Request-Id", ""),
         "input_length": input_length,
-        "cache_hit": cache_hit,
-        "estimated_new_tokens": estimated_new,
         "t_proxy_recv": _time.monotonic(),
         "policy": policy,
-        "chosen_cost": round(best_cost, 2),
     }
 
-    if session_id:
-        session_affinity_combined[session_id] = best_idx
+    if policy == "lmetric":
+        chosen, best_idx = pick_instance_lmetric(
+            combined_instances, token_ids, session_id, input_length,
+            session_affinity_combined)
+    elif policy == "unified":
+        chosen, best_idx, decision = pick_instance_unified_hybrid(
+            combined_instances, token_ids, session_id, input_length,
+            session_affinity_combined)
+        breakdown.update(decision)
+        if session_id:
+            session_affinity_combined[session_id] = best_idx
+    else:  # linear (default)
+        chosen, best_idx = pick_instance(
+            combined_instances, token_ids, session_id, input_length,
+            session_affinity_combined)
 
-    if best_needs_push:
-        c_inst = combined_instances[best_cache_idx]
-        d_inst = chosen
-
-        # Query real cache hit from bootstrap (shadow cache is inaccurate)
-        real_hit = await _query_bootstrap_hit(c_inst, token_ids)
-        breakdown["shadow_cache_hit"] = best_cache_hit
-        breakdown["real_cache_hit"] = real_hit
-
-        if real_hit is not None:
-            push_cache_hit = real_hit
-        else:
-            push_cache_hit = best_cache_hit  # fallback to shadow estimate
-
-        # If real hit > 0, proceed with offload
-        if push_cache_hit > 0:
-            push_new = max(0, input_length - push_cache_hit)
-            cache_ratio = push_cache_hit / max(input_length, 1)
-
-            if _current_offloads() >= SETTINGS.max_offload_inflight:
-                breakdown["push_downgraded"] = "cap_reached"
-                return await _handle_local_request(
-                    api, req_data, headers, token_ids, input_length,
-                    chosen, estimated_new, breakdown)
-            if push_new < SETTINGS.heavy_threshold:
-                breakdown["push_downgraded"] = "below_heavy_threshold"
-                return await _handle_local_request(
-                    api, req_data, headers, token_ids, input_length,
-                    chosen, estimated_new, breakdown)
-            if SETTINGS.cache_gate_ratio > 0 and cache_ratio < SETTINGS.cache_gate_ratio:
-                breakdown["push_downgraded"] = "cache_gate"
-                return await _handle_local_request(
-                    api, req_data, headers, token_ids, input_length,
-                    chosen, estimated_new, breakdown)
-
-            offload_mode = getattr(global_args, 'offload_mode', 'cached_prefill')
-            breakdown["c_inst"] = c_inst.url
-            breakdown["d_inst"] = d_inst.url
-            breakdown["push_cache_hit"] = push_cache_hit
-
-            if offload_mode == "cached_prefill":
-                c_inst.ongoing_tokens += input_length
-                c_inst.pending_prefill_tokens += push_new
-                c_inst.num_requests += 1
-                c_inst.active_p_offloads += 1
-                breakdown["route_class"] = "CACHED_PREFILL_OFFLOAD"
-                return await _handle_cached_prefill_offload(
-                    api, req_data, headers, token_ids, input_length,
-                    c_inst, d_inst, push_cache_hit, push_new, breakdown)
-            else:
-                d_inst.ongoing_tokens += input_length
-                d_inst.pending_prefill_tokens += push_new
-                d_inst.num_requests += 1
-                c_inst.active_p_offloads += 1
-                breakdown["route_class"] = "PUSH_MIGRATE"
-                return await _handle_direct_read_offload(
-                    api, req_data, headers, token_ids, input_length,
-                    c_inst, d_inst, push_cache_hit, push_new, breakdown)
-
-        # Real hit is 0 — downgrade to LOCAL
-        breakdown["push_downgraded"] = True
-
-    # LOCAL path (also handles downgraded PUSH)
-    breakdown["route_class"] = "LOCAL"
-    breakdown["routed_to"] = chosen.url
+    cache_hit = chosen.estimate_cache_hit(token_ids)
+    estimated_new = max(0, input_length - cache_hit)
+    breakdown.update({
+        "cache_hit": cache_hit,
+        "estimated_new_tokens": estimated_new,
+        "route_class": "LOCAL",
+        "routed_to": chosen.url,
+    })
     return await _handle_local_request(
         api, req_data, headers, token_ids, input_length,
         chosen, estimated_new, breakdown)
 
 
-PREFILL_TIMEOUT_S = 120  # max seconds to wait for P-instance prefill
-
-
-async def _handle_cached_prefill_offload(api, req_data, headers, token_ids,
-                                          input_length, c_inst, d_inst,
-                                          cache_hit, estimated_new, breakdown):
-    """C does fast cached prefill → KV to Mooncake → D pulls KV and decodes.
-
-    Unlike direct_read (D pulls blocks from C), here C's scheduler IS
-    involved: C prefills (fast, because prefix is cached), pushes KV to
-    Mooncake store, then D pulls and decodes.  This avoids the broken
-    PUSH path where D waits for RDMA transfer while occupying KV blocks.
-    """
-    request_id = headers.get("X-Request-Id", "")
-
-    # Step 1: send blocking prefill to C
-    prefill_data = req_data.copy()
-    prefill_data["kv_transfer_params"] = {
-        "do_remote_decode": True,
-        "do_remote_prefill": False,
-        "transfer_id": f"xfer-{request_id}",
-    }
-    prefill_data["stream"] = False
-    prefill_data["max_tokens"] = 1
-    prefill_data["min_tokens"] = 1
-    prefill_data.pop("max_completion_tokens", None)
-    prefill_data.pop("stream_options", None)
-    p_headers = {**headers, "X-data-parallel-rank": "0"}
-
-    breakdown["t_prefill_sent"] = _time.monotonic()
-
-    try:
-        resp = await c_inst.client.post(api, json=prefill_data, headers=p_headers)
-        breakdown["t_prefill_done"] = _time.monotonic()
-        resp.raise_for_status()
-        await resp.aclose()
-        c_inst.record_prefix(token_ids)
-    except Exception as e:
-        breakdown["t_prefill_done"] = _time.monotonic()
-        breakdown["prefill_error"] = True
-        _breakdown_log.append(breakdown)
-        c_inst.active_p_offloads = max(0, c_inst.active_p_offloads - 1)
-        c_inst.ongoing_tokens -= input_length
-        c_inst.pending_prefill_tokens -= estimated_new
-        c_inst.num_requests -= 1
-        raise HTTPException(status_code=502, detail=f"Prefill on C failed: {e}")
-
-    c_inst.ongoing_tokens -= input_length
-    c_inst.pending_prefill_tokens -= estimated_new
-    c_inst.num_requests -= 1
-    c_inst.active_p_offloads = max(0, c_inst.active_p_offloads - 1)
-
-    # Step 2: send decode to D (pull KV from C via Mooncake)
-    d_inst.ongoing_tokens += input_length
-    d_inst.num_requests += 1
-
-    parsed = urllib.parse.urlparse(str(c_inst.client.base_url))
-    bootstrap_addr = f"http://{parsed.hostname}:{c_inst.bootstrap_port}"
-
-    decode_data = req_data.copy()
-    decode_data["kv_transfer_params"] = {
-        "do_remote_decode": False,
-        "do_remote_prefill": True,
-        "remote_bootstrap_addr": bootstrap_addr,
-        "remote_engine_id": c_inst.engine_id.get(0, ""),
-        "transfer_id": f"xfer-{request_id}",
-    }
-
-    breakdown["t_decode_sent"] = _time.monotonic()
-
-    async def generate():
-        first_token = True
-        sse_buffer = ""
-        output_token_ids: list[int] = []
-        try:
-            async with d_inst.client.stream("POST", api, json=decode_data, headers=headers) as resp:
-                resp.raise_for_status()
-                async for chunk in resp.aiter_bytes():
-                    sse_buffer, new_output_ids = _extract_output_token_ids_from_sse(
-                        sse_buffer, chunk)
-                    output_token_ids.extend(new_output_ids)
-                    if first_token:
-                        d_inst.ongoing_decode_tokens += input_length
-                        breakdown["t_first_token"] = _time.monotonic()
-                        first_token = False
-                    yield chunk
-            d_inst.record_prefix(_realized_tokens(token_ids, output_token_ids))
-        finally:
-            if not first_token:
-                d_inst.ongoing_decode_tokens -= input_length
-            d_inst.ongoing_tokens -= input_length
-            d_inst.num_requests -= 1
-            breakdown["t_done"] = _time.monotonic()
-            _breakdown_log.append(breakdown)
-
-    return StreamingResponse(generate(), media_type="text/event-stream")
-
-
-async def _handle_direct_read_offload(api, req_data, headers, token_ids,
-                                       input_length, c_inst, d_inst,
-                                       cache_hit, estimated_new, breakdown):
-    """HEAVY request: D direct-RDMA-reads cached KV from C_s, then does
-    local prefill for new tokens + decode.  C_s's scheduler is NOT involved.
-    """
-    request_id = headers.get("X-Request-Id", "")
-
-    # Align cache_hit to block boundary for remote_num_tokens
-    cached_tokens = (cache_hit // BLOCK_SIZE) * BLOCK_SIZE
-    breakdown["t_offload_sent"] = _time.monotonic()
-
-    parsed = urllib.parse.urlparse(str(c_inst.client.base_url))
-    bootstrap_addr = "http://%s:%s" % (parsed.hostname, c_inst.bootstrap_port)
-
-    # Send full prompt to D with direct_read flag
-    decode_data = req_data.copy()
-    decode_data["kv_transfer_params"] = {
-        "do_remote_decode": False,
-        "do_remote_prefill": True,
-        "direct_read": True,
-        "remote_bootstrap_addr": bootstrap_addr,
-        "remote_engine_id": c_inst.engine_id.get(0, ""),
-        "transfer_id": "xfer-" + request_id,
-        "remote_num_tokens": cached_tokens,
-    }
-
-    async def generate():
-        first_token = True
-        sse_buffer = ""
-        output_token_ids: list[int] = []
-        try:
-            async with d_inst.client.stream("POST", api, json=decode_data, headers=headers) as resp:
-                resp.raise_for_status()
-                async for chunk in resp.aiter_bytes():
-                    sse_buffer, new_output_ids = _extract_output_token_ids_from_sse(
-                        sse_buffer, chunk)
-                    output_token_ids.extend(new_output_ids)
-                    if first_token:
-                        d_inst.pending_prefill_tokens -= estimated_new
-                        d_inst.ongoing_decode_tokens += input_length
-                        breakdown["t_first_token"] = _time.monotonic()
-                        first_token = False
-                    yield chunk
-            d_inst.record_prefix(_realized_tokens(token_ids, output_token_ids))
-        finally:
-            if first_token:
-                d_inst.pending_prefill_tokens -= estimated_new
-            else:
-                d_inst.ongoing_decode_tokens -= input_length
-            d_inst.ongoing_tokens -= input_length
-            d_inst.num_requests -= 1
-            c_inst.active_p_offloads = max(0, c_inst.active_p_offloads - 1)
-            breakdown["t_done"] = _time.monotonic()
-            _breakdown_log.append(breakdown)
-
-    return StreamingResponse(generate(), media_type="text/event-stream")
-
-
 async def _handle_pd_sep(api, req_data, request_id, token_ids, input_length,
                           session_id, headers):
     """PD-Sep mode with per-stage breakdown profiling."""
@@ -999,24 +685,23 @@ def parse_args():
                    help="Comma-separated bootstrap ports for combined instances (for offload mode)")
     p.add_argument("--policy", type=str, default="linear",
                    choices=["linear", "lmetric", "unified"],
-                   help="Routing policy: linear, lmetric (P_tokens × BS), or unified cost model")
+                   help="Routing policy: linear (cache-aware), lmetric (P_tokens × BS), "
+                        "or unified (hybrid affinity + LMetric fallback)")
     p.add_argument("--overload-factor", type=float, default=2.0,
                    help="Break session affinity when instance load > factor * avg")
+    # The four flags below are accepted for bench.sh backward compatibility but
+    # have no effect after the PD-sep offload path was retired (REPORT §3.9,
+    # commits 4c583f2 / cc6e562). Removing them would break scripts/bench.sh and
+    # scripts/legacy/*.sh which still pass them through.
     p.add_argument("--max-offload-inflight", type=int, default=4,
-                   help="Global cap on concurrent P-role offloads (M3)")
+                   help="[DEPRECATED] PUSH offload retired; no effect")
     p.add_argument("--offload-mode", type=str, default="cached_prefill",
                    choices=["direct_read", "cached_prefill"],
-                   help="direct_read: D pulls KV from C (PUSH). "
-                        "cached_prefill: C prefills then D decodes (PD-sep style).")
+                   help="[DEPRECATED] PUSH offload retired; no effect")
     p.add_argument("--cache-gate-ratio", type=float, default=0.0,
-                   help="Min cache_hit/input ratio to allow offload "
-                        "(0.0 disables gate, 1.0 disables offload entirely)")
+                   help="[DEPRECATED] PUSH offload retired; no effect")
     p.add_argument("--decode-iteration-s", type=float, default=0.05,
-                   help="Estimated per-request decode iteration time in seconds")
-    p.add_argument("--migration-request-factor", type=float, default=1.5,
-                   help="Trigger session migration when num_requests > avg * factor")
-    p.add_argument("--migration-ttft-threshold", type=float, default=5.0,
-                   help="Trigger migration when instance median TTFT > this (seconds)")
+                   help="[DEPRECATED] PUSH offload retired; no effect")
     args = p.parse_args()
 
     args.prefill = []
@@ -1039,8 +724,6 @@ if __name__ == "__main__":
     SETTINGS.max_offload_inflight = global_args.max_offload_inflight
     SETTINGS.cache_gate_ratio = global_args.cache_gate_ratio
     SETTINGS.decode_iteration_s = getattr(global_args, 'decode_iteration_s', 0.05)
-    SETTINGS.migration_request_factor = getattr(global_args, 'migration_request_factor', 1.5)
-    SETTINGS.migration_ttft_threshold = getattr(global_args, 'migration_ttft_threshold', 5.0)
     print("SETTINGS: throughput=%.0f rdma_overhead=%.2f offload=%s" % (
         SETTINGS.prefill_throughput, SETTINGS.rdma_overhead_s,
         getattr(global_args, 'offload', False)))

tests/test_proxy_pick.py

@@ -180,6 +180,107 @@ def test_pick_instance_lmetric_picks_lowest_score(proxy):
     assert idx == 0 and chosen is insts[0]
 
 
+def test_pick_instance_lmetric_ignores_session_affinity(proxy):
+    """Review #3: pure --policy lmetric must remain affinity-free."""
+    insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b")]
+    # Make inst[1] look much busier than inst[0]; LMetric must still pick 0
+    # even though affinity points at 1.
+    insts[0].pending_prefill_tokens = 0
+    insts[0].num_requests = 0
+    insts[1].pending_prefill_tokens = 5000
+    insts[1].num_requests = 4
+    affinity = {"sess1": 1}
+    chosen, idx = proxy.pick_instance_lmetric(insts, None, "sess1", 1000, affinity)
+    assert idx == 0
+    # Picker must not mutate the affinity dict either.
+    assert affinity == {"sess1": 1}
+
+
+def _record_n_blocks(proxy, inst, n: int) -> list[int]:
+    """Record n distinct one-block prefixes on inst; return token_ids covering them."""
+    block_size = proxy.BLOCK_SIZE
+    tokens: list[int] = []
+    for b in range(n):
+        tokens.extend([1000 + b] * block_size)
+    inst.record_prefix(tokens)
+    return tokens
+
+
+def test_hybrid_high_cache_session_sticks_to_affinity(proxy):
+    """Hybrid: affinity instance with cache_ratio > 0.5 and no overload → stick."""
+    insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b")]
+    tokens = _record_n_blocks(proxy, insts[1], 2)  # 2 blocks cached on inst[1]
+    affinity = {"sess1": 1}
+    chosen, idx, decision = proxy.pick_instance_unified_hybrid(
+        insts, tokens, "sess1", len(tokens), affinity)
+    assert idx == 1 and chosen is insts[1]
+    assert decision["decision"] == "affinity"
+    assert decision["affinity_idx"] == 1
+    assert decision["chosen_idx"] == 1
+    assert decision["affinity_cache_ratio"] > 0.5
+    assert decision["tie_break_used"] is False
+
+
+def test_hybrid_high_cache_breaks_on_overload(proxy):
+    """Hybrid: affinity num_requests > avg * overload_factor → fall back to LMetric,
+    and with realistic new-token tail the LMetric fallback steers off the hot instance."""
+    insts = [
+        _make_inst(proxy, "http://a"),
+        _make_inst(proxy, "http://b"),
+        _make_inst(proxy, "http://c"),
+    ]
+    cached = _record_n_blocks(proxy, insts[1], 2)
+    # Append one more uncached block so LMetric sees a real prefill cost on the
+    # cached instance too (BS multiplier becomes visible). Without this, the
+    # cached instance scores 0 * BS = 0 regardless of how loaded it is.
+    tokens = cached + [999_999] * proxy.BLOCK_SIZE
+    insts[1].num_requests = 300  # avg = 100; 300 > 100 * 2.0 ✓ breaks the gate
+    affinity = {"sess1": 1}
+    chosen, idx, decision = proxy.pick_instance_unified_hybrid(
+        insts, tokens, "sess1", len(tokens), affinity)
+    assert decision["decision"] == "lmetric_fallback"
+    assert decision["affinity_idx"] == 1
+    assert idx != 1, "affinity instance is overloaded; fallback should steer away"
+
+
+def test_hybrid_low_cache_falls_back(proxy):
+    """Hybrid: cache_ratio <= 0.5 on affinity → fall back to LMetric."""
+    insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b")]
+    tokens = [1] * (proxy.BLOCK_SIZE * 2)  # 1024 tokens, nothing cached anywhere
+    affinity = {"sess1": 1}
+    chosen, idx, decision = proxy.pick_instance_unified_hybrid(
+        insts, tokens, "sess1", len(tokens), affinity)
+    assert decision["decision"] == "lmetric_fallback"
+    assert decision["affinity_cache_ratio"] == 0.0
+
+
+def test_hybrid_new_session_tie_break_does_not_always_pick_index_0(proxy):
+    """Review #4: when all instances tie (e.g. BS=0), tie-break must rotate."""
+    insts = [_make_inst(proxy, "http://a") for _ in range(3)]
+    seen = set()
+    for _ in range(12):
+        # No session_id, all empty → score = 0 for everyone → ties → rotate.
+        chosen, idx, decision = proxy.pick_instance_unified_hybrid(
+            insts, None, None, 100, {})
+        seen.add(idx)
+        assert decision["decision"] == "lmetric_fallback"
+        assert decision["tie_break_used"] is True
+    assert seen == {0, 1, 2}, f"tie-breaker did not rotate; only saw {seen}"
+
+
+def test_hybrid_decision_fields_populated(proxy):
+    """Review #7: decision dict must carry the breakdown fields."""
+    insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b")]
+    _, _, decision = proxy.pick_instance_unified_hybrid(
+        insts, None, None, 100, {})
+    expected_keys = {
+        "decision", "affinity_idx", "chosen_idx",
+        "affinity_cache_hit", "affinity_cache_ratio", "affinity_num_requests",
+        "avg_num_requests", "fallback_score", "tie_break_used",
+    }
+    assert expected_keys.issubset(decision.keys())
+
+
 def test_settings_has_runtime_knobs(proxy):
     """D5/B4/M3: Settings dataclass exposes the previously-hardcoded knobs."""
     s = proxy.SETTINGS