Docs: reconcile routing docs with current hybrid direction

Per analysis/unified_routing_fix_review.md #2, several docs still
presented the retired single-argmin + PUSH-migration design as the
final algorithm. Mark them superseded and document the current hybrid
direction (commit 255c8e6).

- REPORT.md §1.1 / §3.9: add errata callout and section header noting
  the "Final Design" framing was retired after cc6e562 / 4c583f2;
  point readers to docs/migration-policy-design.md.

- docs/migration-policy-design.md: rewrite. Opens with the current
  hybrid algorithm (LMetric base + cache_ratio>0.5 affinity gate +
  tie-breaker), then a "What Was Retired" commit table, then the old
  Approach A numbers preserved as "Historical Baseline-Mode Comparison".

- analysis/research_findings.md §2.2 / §5: correct the LMetric framing.
  LMetric isn't "neutralized by affinity constraints" (pure --policy
  lmetric has no affinity at all); it converges to similar placements
  because P_tokens includes new_uncached_tokens, giving it implicit
  soft affinity.

- analysis/elastic_hypotheses.md: same LMetric correction in the
  "DOESN'T work" summary, plus a footer cross-referencing the current
  routing direction.

- analysis/unified_routing_fix_review.md: track this file (was
  untracked); it is the review handoff cited from the updated docs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-25 10:47:14 +08:00
parent ac6534c3ff
commit 6a27f75337
5 changed files with 591 additions and 53 deletions

View File

@@ -38,7 +38,24 @@ These characteristics fundamentally change what optimizations matter.
**Setup**: 8 instances, LMetric vs linear routing
**Result**: TTFT +2.2%, TPOT -4.4%, E2E +2.6% — all within noise (±7% run-to-run)
**Root cause**: Session affinity constrains routing freedom. LMetric's benefit (hyperparameter-free load balancing) is neutralized because turn 2+ requests MUST go to their session-sticky instance regardless of the scoring function. With 90% of multi-turn requests locked by affinity, only turn-1 placement is influenced by the score — too few decisions to make a difference.
**Root cause (updated)**: LMetric is not "neutralized by affinity
constraints" — pure `--policy lmetric` runs without session affinity at all.
The actual reason the LMetric vs linear comparison sits within noise is that
`P_tokens` already includes `new_uncached_tokens = input_length - cache_hit`,
which means later turns of a session naturally score lowest on the instance
that cached their prefix. This gives LMetric an **implicit soft affinity**
that competes with linear's explicit sticky affinity. The two arrive at
similar placements through different mechanisms.
This is also why explicit migration buys little on top of LMetric: the
first-order signal driving placement is already cache-derived. See
`docs/migration-policy-design.md` for how the current hybrid policy uses
this insight (LMetric base + explicit affinity only when `cache_ratio > 0.5`).
**Previous framing (incorrect)**: an earlier draft of this section attributed
the result to session affinity constraining LMetric's routing freedom. That
framing assumed `--policy lmetric` inherited the linear-mode session-sticky
behavior, which it does not (verified in `tests/test_proxy_pick.py`).
### 2.3 Elastic P2P RDMA Offload (Heavy prefill on different instance)
@@ -148,7 +165,9 @@ This changes the scheduling picture: most "HEAVY" requests in agentic workloads
### Why existing approaches don't work:
1. **PD-Sep** assumes decode needs dedicated resources → agentic has memory wall on decode
2. **LMetric** assumes routing freedom → agentic has session affinity constraints
2. **LMetric** matches linear within noise because cache-hit appears in
`P_tokens` itself, so it already routes later turns back to the cached
instance via implicit soft affinity — explicit affinity buys little
3. **Elastic RDMA** assumes KV transfer is cheap → Mooncake lacks layerwise pipelining
4. **Size-based classification** assumes HEAVY = needs special handling → after cache, most HEAVY is MEDIUM