Docs: reconcile routing docs with current hybrid direction
Per analysis/unified_routing_fix_review.md #2, several docs still presented the retired single-argmin + PUSH-migration design as the final algorithm. Mark them superseded and document the current hybrid direction (commit255c8e6). - REPORT.md §1.1 / §3.9: add errata callout and section header noting the "Final Design" framing was retired aftercc6e562/ 4c583f2; point readers to docs/migration-policy-design.md. - docs/migration-policy-design.md: rewrite. Opens with the current hybrid algorithm (LMetric base + cache_ratio>0.5 affinity gate + tie-breaker), then a "What Was Retired" commit table, then the old Approach A numbers preserved as "Historical Baseline-Mode Comparison". - analysis/research_findings.md §2.2 / §5: correct the LMetric framing. LMetric isn't "neutralized by affinity constraints" (pure --policy lmetric has no affinity at all); it converges to similar placements because P_tokens includes new_uncached_tokens, giving it implicit soft affinity. - analysis/elastic_hypotheses.md: same LMetric correction in the "DOESN'T work" summary, plus a footer cross-referencing the current routing direction. - analysis/unified_routing_fix_review.md: track this file (was untracked); it is the review handoff cited from the updated docs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -244,7 +244,11 @@ Offloaded: — 13/500 (2.6%) too few to matter
|
||||
### What DOESN'T work for agentic workloads:
|
||||
|
||||
1. **PD-Sep**: net negative — KV cache memory wall on decode instances
|
||||
2. **LMetric (OSDI'26)**: ≈ linear routing — session affinity limits routing freedom
|
||||
2. **LMetric (OSDI'26)**: ≈ linear routing — `P_tokens` already includes
|
||||
`new_uncached_tokens`, so cache-hit scoring gives LMetric an implicit
|
||||
soft affinity that converges to similar placements as explicit sticky
|
||||
affinity (see `analysis/research_findings.md` §2.2 for the corrected
|
||||
framing)
|
||||
3. **Elastic P2P RDMA offload**: net negative — Mooncake transfer overhead, no layerwise pipeline
|
||||
4. **OVERLOAD_FACTOR tuning**: no effect — imbalance from workload skew, not routing
|
||||
5. **Dedicated Prefill Service (PS)**: cannot win cost comparison without KV pull, PS is always slower than cached C
|
||||
@@ -270,3 +274,21 @@ Instead of fixed chunk size, dynamically adjust based on decode pressure:
|
||||
- When decode queue is deep: smaller chunks → more decode slots → better TPOT
|
||||
- When decode queue is empty: larger chunks → faster prefill → better TTFT
|
||||
- This is a vLLM scheduler modification, not a routing change
|
||||
|
||||
---
|
||||
|
||||
## Current routing direction (cross-reference)
|
||||
|
||||
The hypotheses above produced the following positive results that informed
|
||||
the current `--policy unified` implementation:
|
||||
|
||||
- H1 / H7 / H9 (negative): PD-sep offload, OVERLOAD_FACTOR tuning, and
|
||||
elastic RDMA at high concurrency all regressed or stayed within noise.
|
||||
- H3 / H4 / H6 (partial): cache-gated offload exists but only ~10-12% of
|
||||
HEAVY requests have cache, and the offloaded subset pays RDMA penalty.
|
||||
|
||||
The active algorithm (commit `255c8e6`) is **hybrid LMetric + high-cache
|
||||
affinity** in baseline mode (no Mooncake). The retired migration variants
|
||||
are catalogued in `docs/migration-policy-design.md` (Approach A and the
|
||||
revert chain `cc6e562` / `4c583f2`). H7's rejection (OVERLOAD_FACTOR within
|
||||
noise) is why the active default stays at `overload_factor=2.0`.
|
||||
|
||||
Reference in New Issue
Block a user