Gahow Wang
cef914ecd4
Why the LMetric → load_only APC gap is only +3.3pp despite LMetric explicitly being "cache-aware load routing": P = pending_prefill_tokens + (input_length - cache_hit) score = P × num_requests <-- multiplicative cache_hit appears only as a reduction inside P. Because score is multiplicative in num_requests, a session-affinity instance whose num_requests has climbed will lose argmin to a cold instance even when cache_hit on the warm one is ~90%. Worked example: warm: P=2500, num_req=5 -> score 12500 cold: P=10000, num_req=1 -> score 10000 <-- LMetric picks cold load_only 53.9% APC (pure num_requests) LMetric 57.2% +3.3pp (cache as additive cost term) sticky 77.7% +23.8pp (cache as hard constraint) unified 78.7% +24.8pp (cache as hard+soft hybrid) Lesson worth stating explicitly in §3.1: cache awareness folded into a multiplicative load cost-model is structurally insufficient. Affinity must be a separate routing branch (sticky / unified hybrid), not a correction term inside a load score. PAPER_OUTLINE.md §3.1 gets the design analysis + the new APC table; MEETING.md gets a one-paragraph version of the same point. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>