gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix A+C: real cache sync + cached-prefill-on-C architecture

Browse Source 
A: Add /estimate_hit endpoint to bootstrap server for real-time cache
   probing. Proxy queries this before committing to PUSH, eliminating
   24% zero-match PUSH requests (shadow cache divergence).

C: Add _handle_cached_prefill_offload: C (cache source) does fast
   cached prefill → KV to Mooncake → D pulls and decodes.
   Replaces broken direct_read PUSH where D waited for RDMA transfer
   while occupying KV blocks without doing compute.

Also: update §3.9 baseline to plain vLLM with full mean/p50/p90/p99.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-24 11:22:38 +08:00
parent 2b9eae0d54
commit cdf83493ab
3 changed files with 252 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
REPORT.md
View File

@@ -373,20 +373,28 @@ Only 2 measured parameters: `prefill_throughput=7000 tok/s`, `rdma_overhead=0.1s
**Results (eval_unified_v3, 850/850, 0 errors):**
| Metric | Baseline | **Unified v3** | Delta |
|--------|----------|---------------|-------|
| TTFT mean | 4.35s | **3.24s** | **-25.5%** |
| TTFT p50 | 0.95s | **0.78s** | **-17.9%** |
| TTFT p90 | 12.47s | **7.79s** | **-37.5%** |
| TPOT p90 | 0.177 | 0.204 | +14.9% |
| E2E mean | 19.10s | **17.69s** | **-7.4%** |
| E2E p50 | 6.44s | **5.48s** | **-14.9%** |
Baseline = `eval_baseline_linear` (plain vLLM, no Mooncake, linear policy, 850 req, same trace).
| Metric | Baseline (plain) | **Unified v3 (kv_both)** | Delta |
|--------|-----------------|-------------------------|-------|
| TTFT mean | 4.348s | **3.277s** | **-24.6%** |
| TTFT p50 | 0.945s | **0.793s** | **-16.1%** |
| TTFT p90 | 12.468s | **8.472s** | **-32.1%** |
| TTFT p99 | 48.149s | **41.587s** | **-13.6%** |
| TPOT mean | 0.116s | 0.112s | -3.1% |
| TPOT p50 | 0.071s | 0.077s | +8.9% |
| TPOT p90 | 0.177s | 0.198s | +11.7% |
| TPOT p99 | 1.018s | **0.816s** | **-19.9%** |
| E2E mean | 19.10s | 19.81s | +3.7% |
| E2E p50 | 6.443s | **5.599s** | **-13.1%** |
| E2E p90 | 42.27s | 47.48s | +12.3% |
| E2E p99 | 192.2s | 238.0s | +23.8% |
**Routing**: 723 LOCAL + 116 PUSH_MIGRATE (13.8%). All 116 pushes had cache (avg 25k tokens) — no cold offloads. The unified cost model naturally avoids cold migration because `cold + RDMA > cold` (RDMA adds overhead without reducing prefill).
**Tradeoff**: TPOT p90 +15% from kv_both background threads + PUSH operations. In exchange: TTFT -38%, E2E -15% at p50.
**Tradeoff**: TTFT uniformly improves (p50 -16%, p90 -32%). TPOT mixed: p50/p90 worse (+9%/+12%), but p99 improves (-20%) — PUSH migration relieves the heaviest tail prefills. **E2E tail degrades significantly** (p90 +12%, p99 +24%): kv_both always-on overhead + PUSH transfer latency on migrated requests inflates E2E for long requests, offsetting the TTFT gain. The p50 benefit (-13%) comes from the majority of LOCAL requests getting faster prefill due to reduced queue contention.
**Output**: `outputs/eval_unified_v3/` on dash0.
**Output**: `outputs/eval_unified_v3/` on dash0, baseline from `outputs/eval_baseline_linear/`.
## 4. System-Level Analysis