Docs: reconcile routing docs with current hybrid direction
Per analysis/unified_routing_fix_review.md #2, several docs still presented the retired single-argmin + PUSH-migration design as the final algorithm. Mark them superseded and document the current hybrid direction (commit255c8e6). - REPORT.md §1.1 / §3.9: add errata callout and section header noting the "Final Design" framing was retired aftercc6e562/ 4c583f2; point readers to docs/migration-policy-design.md. - docs/migration-policy-design.md: rewrite. Opens with the current hybrid algorithm (LMetric base + cache_ratio>0.5 affinity gate + tie-breaker), then a "What Was Retired" commit table, then the old Approach A numbers preserved as "Historical Baseline-Mode Comparison". - analysis/research_findings.md §2.2 / §5: correct the LMetric framing. LMetric isn't "neutralized by affinity constraints" (pure --policy lmetric has no affinity at all); it converges to similar placements because P_tokens includes new_uncached_tokens, giving it implicit soft affinity. - analysis/elastic_hypotheses.md: same LMetric correction in the "DOESN'T work" summary, plus a footer cross-referencing the current routing direction. - analysis/unified_routing_fix_review.md: track this file (was untracked); it is the review handoff cited from the updated docs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -38,7 +38,24 @@ These characteristics fundamentally change what optimizations matter.
|
||||
**Setup**: 8 instances, LMetric vs linear routing
|
||||
**Result**: TTFT +2.2%, TPOT -4.4%, E2E +2.6% — all within noise (±7% run-to-run)
|
||||
|
||||
**Root cause**: Session affinity constrains routing freedom. LMetric's benefit (hyperparameter-free load balancing) is neutralized because turn 2+ requests MUST go to their session-sticky instance regardless of the scoring function. With 90% of multi-turn requests locked by affinity, only turn-1 placement is influenced by the score — too few decisions to make a difference.
|
||||
**Root cause (updated)**: LMetric is not "neutralized by affinity
|
||||
constraints" — pure `--policy lmetric` runs without session affinity at all.
|
||||
The actual reason the LMetric vs linear comparison sits within noise is that
|
||||
`P_tokens` already includes `new_uncached_tokens = input_length - cache_hit`,
|
||||
which means later turns of a session naturally score lowest on the instance
|
||||
that cached their prefix. This gives LMetric an **implicit soft affinity**
|
||||
that competes with linear's explicit sticky affinity. The two arrive at
|
||||
similar placements through different mechanisms.
|
||||
|
||||
This is also why explicit migration buys little on top of LMetric: the
|
||||
first-order signal driving placement is already cache-derived. See
|
||||
`docs/migration-policy-design.md` for how the current hybrid policy uses
|
||||
this insight (LMetric base + explicit affinity only when `cache_ratio > 0.5`).
|
||||
|
||||
**Previous framing (incorrect)**: an earlier draft of this section attributed
|
||||
the result to session affinity constraining LMetric's routing freedom. That
|
||||
framing assumed `--policy lmetric` inherited the linear-mode session-sticky
|
||||
behavior, which it does not (verified in `tests/test_proxy_pick.py`).
|
||||
|
||||
### 2.3 Elastic P2P RDMA Offload (Heavy prefill on different instance)
|
||||
|
||||
@@ -148,7 +165,9 @@ This changes the scheduling picture: most "HEAVY" requests in agentic workloads
|
||||
|
||||
### Why existing approaches don't work:
|
||||
1. **PD-Sep** assumes decode needs dedicated resources → agentic has memory wall on decode
|
||||
2. **LMetric** assumes routing freedom → agentic has session affinity constraints
|
||||
2. **LMetric** matches linear within noise because cache-hit appears in
|
||||
`P_tokens` itself, so it already routes later turns back to the cached
|
||||
instance via implicit soft affinity — explicit affinity buys little
|
||||
3. **Elastic RDMA** assumes KV transfer is cheap → Mooncake lacks layerwise pipelining
|
||||
4. **Size-based classification** assumes HEAVY = needs special handling → after cache, most HEAVY is MEDIUM
|
||||
|
||||
|
||||
Reference in New Issue
Block a user