2 Commits

Author	SHA1	Message	Date
Gahow Wang	3594f7dce0	Fix LMetric routing: remove session affinity, align with OSDI'26 spec ... LMetric was incorrectly sharing session-sticky logic with Linear policy. Fixed to pure per-request routing: score = P_tokens × BS where P = pending_prefill + (input - cache_hit), BS = num_requests. Experiment result (200 req, fresh restart): Linear vs corrected LMetric show <2% difference on all metrics — LMetric's cache-hit estimation provides implicit soft affinity that preserves locality without explicit session stickiness. Also fix bench.sh missing cd (replayer module not found from non-project cwd) and rewrite run_lmetric_ab.sh as thin wrapper around bench.sh to eliminate duplicated launch/cleanup logic that broke under set -euo. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-23 11:56:58 +08:00
Gahow Wang	e4fa56cb1e	LMetric routing policy (OSDI'26) + A/B results vs linear baseline ... Implement LMetric (P_tokens × BS multiplication score) from "Simple is Better" (Zhang et al., OSDI'26) as alternative routing policy for combined mode. Key changes: - cache_aware_proxy.py: add --policy {linear,lmetric} flag, track pending_prefill_tokens and num_requests per instance, /stats endpoint - run_lmetric_ab.sh: automated A/B script for fair comparison Results (200 req, fresh restart, same trace): Linear: TTFT50=1.086 TPOT90=0.077 E2E50=5.423 LMetric: TTFT50=1.099 TPOT90=0.073 E2E50=5.205 Delta: TTFT +1.2% TPOT -5.9% E2E -4.0% LMetric improves TPOT/E2E modestly through better load balancing, but routing policy headroom is limited vs elastic P2P offload (-44% E2E). TODO: vLLM → Redis → router pipeline for exact state ablation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-22 16:57:32 +08:00