Gahow Wang 3bc37cc6d5 PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix ...

Experiments run:
- Phase 0: kv_both has zero idle overhead (TPOT +1.3%, noise)
- PS V1 (cold prefill): REJECTED — PS always slower than cached C
- PS V1+flexD: 92.5% OK, HEAVY TTFT 7.8s (baseline 5.0s) — PS bottleneck
- V2 (C_s prefill + flexible D): E2E -9% but 6 errors, RDMA bimodal
- H4 (cache-gate): 198/200 OK, GPU imbalance 4.0x→2.0x, but HEAVY_OFFLOAD
  TTFT=11.5s due to RDMA. HEAVY_COLO improved 10.5% from better balance.
- H5: Mooncake RDMA transfer R²=0.095, bimodal (0.6s or 18-30s)

Key findings:
- Mooncake lacks layerwise KV transfer → RDMA is pure sequential overhead
- 92% of HEAVY are turn-1 cold → offloading cold requests always loses
- GPU balance improvement from routing IS real (-10.5% HEAVY_COLO TTFT)
- RDMA transfer negates the routing benefit for offloaded requests

Code changes:
- bench.sh: add GPU timeline monitoring (gpu_monitor.sh during benchmark)
- cache_aware_proxy.py: H4 cache-gate, flexible D, PS routing
- mooncake_connector.py: elif→if fix (allow dual prefill+decode flags)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-23 02:14:37 +08:00

..

ab_gpu_test.sh

Add GPU utilization A/B test and fix cache-aware proxy bugs

2026-05-21 22:13:38 +08:00

analyze_3way.py

Add P/D ratio ablation: 6P+2D vs 4P+4D vs Combined

2026-05-21 22:42:20 +08:00

analyze_ablations.py

Ablation 2: fire-and-forget vs await-prefill scheduling

2026-05-21 23:02:42 +08:00

analyze_agentic_patterns.py

Balanced session-sticky routing + agentic workload pattern analysis

2026-05-22 01:50:27 +08:00

analyze_aggregation.py

P2P prefill offload: TTFT p50 -13% but p90 +59% (median-vs-tail tradeoff)

2026-05-22 12:28:24 +08:00

analyze_breakdown.py

Add per-request breakdown profiling, identify KV cache memory bottleneck

2026-05-22 00:13:50 +08:00

analyze_cache_hit.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

analyze_eviction.py

KV cache lifecycle design + eviction loss analysis

2026-05-22 01:27:22 +08:00

analyze_gpu_ab.py

Add GPU utilization A/B test and fix cache-aware proxy bugs

2026-05-21 22:13:38 +08:00

analyze_h4_results.py

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

analyze_h5_rdma.py

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

analyze_p2p_cache.py

P2P cache analysis: external KV correctly registered in prefix cache

2026-05-22 13:25:34 +08:00

analyze_trace.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

bench.sh

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

cache_aware_proxy.py

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

compare_ab_final.py

Fair A/B: Elastic P2P wins on ALL metrics vs baseline (fresh restart)

2026-05-22 15:48:51 +08:00

compare_adaptive.py

Adaptive prefill offload v1: implementation + experiment

2026-05-22 01:00:10 +08:00

compare_aggregation.py

P2P prefill offload: TTFT p50 -13% but p90 +59% (median-vs-tail tradeoff)

2026-05-22 12:28:24 +08:00

compare_balanced.py

Balanced routing result: APC +4pp but latency +23% (cache-load tradeoff)

2026-05-22 02:13:15 +08:00

compare_elastic_v4.py

Elastic P2P v4: error rate 25% -> 4%, TTFT p50 -12% (median-tail tradeoff)

2026-05-22 15:08:16 +08:00

compare_p2p.py

P2P prefill offload: TTFT p50 -13% but p90 +59% (median-vs-tail tradeoff)

2026-05-22 12:28:24 +08:00

compare_results.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

compute_roofline.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

final_all_comparison.py

TP=2 DP=4 + hybrid routing: best TTFT at cost of TPOT

2026-05-22 10:35:18 +08:00

final_comparison.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

final_gpu_comparison.py

Final GPU comparison: hybrid routing matches baseline latency with better APC

2026-05-22 03:14:05 +08:00

gpu_monitor.sh

Add GPU utilization A/B test and fix cache-aware proxy bugs

2026-05-21 22:13:38 +08:00

launch_elastic_p2p.sh

Phase 1 milestone: system-level analysis + reproducible report

2026-05-22 16:17:41 +08:00

launch_pd_mooncake.sh

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

launch_pd_separated.sh

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

launch_phase1_ps.sh

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

launch_vllm.sh

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

plot_gpu_timeline.py

Fair A/B: Elastic P2P wins on ALL metrics vs baseline (fresh restart)

2026-05-22 15:48:51 +08:00

profile_fnf.py

Add per-request breakdown profiling, identify KV cache memory bottleneck

2026-05-22 00:13:50 +08:00

profile_why_pdsep_loses.py

System profile: 4 mechanisms why PD-Sep loses to session-sticky combined

2026-05-22 10:58:59 +08:00

run_benchmark.sh

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

run_elastic_stability_test.sh

Invalidate prior A/B results + add proper experiment harness

2026-05-22 17:54:21 +08:00

run_experiments.sh

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

run_h4_cache_gate.sh

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

run_lmetric_ab.sh

LMetric routing policy (OSDI'26) + A/B results vs linear baseline

2026-05-22 16:57:32 +08:00

run_ps_ablation.sh

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

run_ps_flexd.sh

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

run_ps_remaining.sh

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

run_v2_offload.sh

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

sample_trace.py

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

simulate_cache_policies.py

Cache policy simulation: routing quality dominates, not eviction policy

2026-05-22 01:28:53 +08:00