Gahow Wang f5e45afd4e Fix 4 elastic PS bugs: D accounting, offload cap, cache migration, prefix sync ...

Bug 1+5: D instance had no accounting during prefill phase (7-11s window).
Router saw D as idle, routing extra traffic that caused KV allocation failures.
Fix: reserve D's ongoing_tokens+num_requests at offload decision time.

Bug 7: No cap on concurrent offloads despite REPORT claiming MAX_OFFLOAD=4.
Fix: add MAX_OFFLOAD_INFLIGHT=4 check before offloading.

Bug 6: Session affinity migrated to D but proxy cache estimator wasn't
updated for D. Future turns scored D as cache-cold.
Fix: call d_inst.record_prefix(token_ids) after successful decode.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-23 15:55:11 +08:00

analysis

Add comprehensive research findings document

2026-05-23 07:16:31 +08:00

patches

Add vLLM patches directory for version-controlled patch management

2026-05-22 00:26:14 +08:00

replayer

Fix replay methodology: trace-driven dispatch, no artificial limits

2026-05-23 12:43:41 +08:00

scripts

Fix 4 elastic PS bugs: D accounting, offload cap, cache migration, prefix sync

2026-05-23 15:55:11 +08:00

third_party/vllm

PS experiments + H4 cache-gate + GPU profiling + Mooncake elif→if fix

2026-05-23 02:14:37 +08:00

.gitignore

Phase 1 milestone: system-level analysis + reproducible report

2026-05-22 16:17:41 +08:00

pyproject.toml

Agentic workload PD separation analysis with trace-driven benchmarks

2026-05-21 21:21:57 +08:00

REPORT.md

Production-realistic baseline: APC 67.5%, TPOT +139% from interference

2026-05-23 15:44:34 +08:00

TODO.md

LMetric routing policy (OSDI'26) + A/B results vs linear baseline

2026-05-22 16:57:32 +08:00

Description

No description provided

48 MiB

Languages

Python 82.9%

Shell 17.1%