Gahow Wang 5c66f500fc Fix offload gate: remove cache_gate for direct RDMA read, fix cost model ...

The cache_gate_ratio=0.3 check blocked 83/112 HEAVY requests (75%)
because they were cold (cache_ratio=0). But with direct RDMA read,
D reads C's cached blocks via RDMA regardless of cache ratio — the
gate was protecting against the OLD flow (C does prefill + push).

Also fixed cost model: offload_cost now reflects direct read reality:
  OLD: P_queue + P_full_prefill + RDMA (P has no cache → expensive)
  NEW: D_queue + RDMA_read + D_local_prefill(new_tokens)

Offload wins when C_s queue > RDMA_overhead (~2s).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-23 22:01:43 +08:00

analysis

Add comprehensive research findings document

2026-05-23 07:16:31 +08:00

experiments

Add elastic PS evaluation plan for production-realistic trace

2026-05-23 15:56:05 +08:00

patches

Add vLLM patches directory for version-controlled patch management

2026-05-22 00:26:14 +08:00

replayer

replayer: wire --max-inflight-sessions cap into replay loop (B2)

2026-05-23 21:04:09 +08:00

scripts

Fix offload gate: remove cache_gate for direct RDMA read, fix cost model

2026-05-23 22:01:43 +08:00

tests

proxy: Settings dataclass + cache-ratio gate + P-pick offload penalty (B4, M2, M3, D5)

2026-05-23 21:11:17 +08:00

third_party/vllm

Fix: import field from dataclasses for PullReqMeta

2026-05-23 21:29:24 +08:00

.gitignore

Phase 1 milestone: system-level analysis + reproducible report

2026-05-22 16:17:41 +08:00

FIXES.md

Add FIXES.md with prioritized repo cleanup checklist

2026-05-23 20:35:56 +08:00

pyproject.toml

tests: add minimal coverage for percentile + proxy routing (S1)

2026-05-23 21:07:14 +08:00

REPORT.md

REPORT: add §1.1 errata flagging superseded sections (S3)

2026-05-23 20:58:38 +08:00

TODO.md

LMetric routing policy (OSDI'26) + A/B results vs linear baseline

2026-05-22 16:57:32 +08:00

Description

No description provided

48 MiB

Languages

Python 82.9%

Shell 17.1%