Docs: reconcile routing docs with current hybrid direction

Per analysis/unified_routing_fix_review.md #2, several docs still
presented the retired single-argmin + PUSH-migration design as the
final algorithm. Mark them superseded and document the current hybrid
direction (commit 255c8e6).

- REPORT.md §1.1 / §3.9: add errata callout and section header noting
  the "Final Design" framing was retired after cc6e562 / 4c583f2;
  point readers to docs/migration-policy-design.md.

- docs/migration-policy-design.md: rewrite. Opens with the current
  hybrid algorithm (LMetric base + cache_ratio>0.5 affinity gate +
  tie-breaker), then a "What Was Retired" commit table, then the old
  Approach A numbers preserved as "Historical Baseline-Mode Comparison".

- analysis/research_findings.md §2.2 / §5: correct the LMetric framing.
  LMetric isn't "neutralized by affinity constraints" (pure --policy
  lmetric has no affinity at all); it converges to similar placements
  because P_tokens includes new_uncached_tokens, giving it implicit
  soft affinity.

- analysis/elastic_hypotheses.md: same LMetric correction in the
  "DOESN'T work" summary, plus a footer cross-referencing the current
  routing direction.

- analysis/unified_routing_fix_review.md: track this file (was
  untracked); it is the review handoff cited from the updated docs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-25 10:47:14 +08:00
parent ac6534c3ff
commit 6a27f75337
5 changed files with 591 additions and 53 deletions

View File

@@ -244,7 +244,11 @@ Offloaded: — 13/500 (2.6%) too few to matter
### What DOESN'T work for agentic workloads:
1. **PD-Sep**: net negative — KV cache memory wall on decode instances
2. **LMetric (OSDI'26)**: ≈ linear routing — session affinity limits routing freedom
2. **LMetric (OSDI'26)**: ≈ linear routing — `P_tokens` already includes
`new_uncached_tokens`, so cache-hit scoring gives LMetric an implicit
soft affinity that converges to similar placements as explicit sticky
affinity (see `analysis/research_findings.md` §2.2 for the corrected
framing)
3. **Elastic P2P RDMA offload**: net negative — Mooncake transfer overhead, no layerwise pipeline
4. **OVERLOAD_FACTOR tuning**: no effect — imbalance from workload skew, not routing
5. **Dedicated Prefill Service (PS)**: cannot win cost comparison without KV pull, PS is always slower than cached C
@@ -270,3 +274,21 @@ Instead of fixed chunk size, dynamically adjust based on decode pressure:
- When decode queue is deep: smaller chunks → more decode slots → better TPOT
- When decode queue is empty: larger chunks → faster prefill → better TTFT
- This is a vLLM scheduler modification, not a routing change
---
## Current routing direction (cross-reference)
The hypotheses above produced the following positive results that informed
the current `--policy unified` implementation:
- H1 / H7 / H9 (negative): PD-sep offload, OVERLOAD_FACTOR tuning, and
elastic RDMA at high concurrency all regressed or stayed within noise.
- H3 / H4 / H6 (partial): cache-gated offload exists but only ~10-12% of
HEAVY requests have cache, and the offloaded subset pays RDMA penalty.
The active algorithm (commit `255c8e6`) is **hybrid LMetric + high-cache
affinity** in baseline mode (no Mooncake). The retired migration variants
are catalogued in `docs/migration-policy-design.md` (Approach A and the
revert chain `cc6e562` / `4c583f2`). H7's rejection (OVERLOAD_FACTOR within
noise) is why the active default stays at `overload_factor=2.0`.