Gahow Wang 012d73f596 Hybrid routing: session-sticky + load-aware override achieves best results ...

Session affinity for KV reuse, with load-aware override when pinned
instance has ongoing_tokens > 2x average. Combines APC of sticky
routing with latency of load-based routing.

Results (1000 req, TP=1 DP=8 combined):
                              TTFT50  TPOT90  E2E50   APC
  Old cache-aware              0.731   0.073   4.480  44.7%
  Balanced session-sticky      0.953   0.079   5.520  48.7%
  Hybrid (sticky+load-aware)   0.737   0.072   4.487  49.4%  <- BEST

Hybrid achieves +4.7pp APC improvement with zero latency regression.
Session-sticky provides KV reuse; load-aware override prevents hotspots.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-22 02:53:44 +08:00

..

ab_gpu_test.sh

Add GPU utilization A/B test and fix cache-aware proxy bugs

2026-05-21 22:13:38 +08:00

analyze_3way.py

Add P/D ratio ablation: 6P+2D vs 4P+4D vs Combined

2026-05-21 22:42:20 +08:00

analyze_ablations.py

Ablation 2: fire-and-forget vs await-prefill scheduling

2026-05-21 23:02:42 +08:00

analyze_agentic_patterns.py

Balanced session-sticky routing + agentic workload pattern analysis

2026-05-22 01:50:27 +08:00

analyze_breakdown.py

Add per-request breakdown profiling, identify KV cache memory bottleneck

2026-05-22 00:13:50 +08:00

analyze_cache_hit.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

analyze_eviction.py

KV cache lifecycle design + eviction loss analysis

2026-05-22 01:27:22 +08:00

analyze_gpu_ab.py

Add GPU utilization A/B test and fix cache-aware proxy bugs

2026-05-21 22:13:38 +08:00

analyze_trace.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

cache_aware_proxy.py

Hybrid routing: session-sticky + load-aware override achieves best results

2026-05-22 02:53:44 +08:00

compare_adaptive.py

Adaptive prefill offload v1: implementation + experiment

2026-05-22 01:00:10 +08:00

compare_balanced.py

Balanced routing result: APC +4pp but latency +23% (cache-load tradeoff)

2026-05-22 02:13:15 +08:00

compare_results.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

compute_roofline.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

final_comparison.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

gpu_monitor.sh

Add GPU utilization A/B test and fix cache-aware proxy bugs

2026-05-21 22:13:38 +08:00

launch_pd_mooncake.sh

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

launch_pd_separated.sh

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

launch_vllm.sh

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

profile_fnf.py

Add per-request breakdown profiling, identify KV cache memory bottleneck

2026-05-22 00:13:50 +08:00

run_benchmark.sh

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

run_experiments.sh

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

sample_trace.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

simulate_cache_policies.py

Cache policy simulation: routing quality dominates, not eviction policy

2026-05-22 01:28:53 +08:00