Invalidate prior A/B results + add proper experiment harness
Prior cross-machine comparison (commit 1e86285) was invalid: dash0
baseline used warm instances with residual KV cache, inflating TTFT
by 2x. Evidence: inst_7 APC=68.3% impossible from 25 cold-start
requests; WARM TTFT p90=3.3s vs fresh=0.26s.
Fair same-machine comparison (both fresh restart on dash0):
Baseline: TTFT50=1.075 TPOT90=0.076 E2E50=5.075 OK=198/200
Elastic P2P: TTFT50=1.018 TPOT90=0.085 E2E50=6.977 OK=195/200
Elastic is WORSE due to Mooncake kv_both memory overhead.
Changes:
- REPORT.md: rewrite §3-4 with corrected results, add §3.5 errata
- pd_separation_analysis.md: update elastic TL;DR with correct numbers
- cache_aware_proxy.py: fix double-decrement bugs in offload path,
add 120s prefill timeout with co-located fallback (HEAVY_COLO_FALLBACK)
- bench.sh: standardized experiment harness with guaranteed GPU cleanup
and fresh-state verification (nvidia-smi check before start)
- run_elastic_stability_test.sh: two-phase elastic vs baseline test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -13,12 +13,14 @@ We benchmarked PD separation (prefill-decode disaggregation) against PD co-locat
|
||||
|
||||
Per-request breakdown shows **87.7% of TTFT** is spent waiting for KV cache memory on decode instances, not prefill compute or KV transfer.
|
||||
|
||||
**Elastic P2P offload** (selective disaggregation of HEAVY requests only) recovers the wins of PD separation without the memory wall:
|
||||
**Elastic P2P offload** (selective disaggregation of HEAVY requests only, Mooncake kv_both): under fair same-machine fresh-restart comparison, elastic does NOT improve over baseline. Mooncake kv_both memory overhead outweighs prefill isolation benefit at moderate load.
|
||||
|
||||
| Config (TP=1, 8×H20) | TTFT p50 | TPOT p90 | E2E p50 | Effective APC |
|
||||
|---|---|---|---|---|
|
||||
| Combined DP=8 (baseline) | 2.383s | 0.117s | 10.232s | ~40% (skewed) |
|
||||
| Elastic P2P (cap=4) | **1.315s** | **0.075s** | **5.708s** | ~70% (balanced) |
|
||||
| Config (TP=1, 8×H20, fresh) | TTFT p50 | TPOT p90 | E2E p50 |
|
||||
|---|---|---|---|
|
||||
| Combined DP=8 (baseline) | **1.075s** | **0.076s** | **5.075s** |
|
||||
| Elastic P2P (kv_both, cap=4) | 1.018s | 0.085s | 6.977s |
|
||||
|
||||
> Earlier cross-machine comparison (commit `1e86285`) was invalidated — baseline used warm instances. See REPORT.md §3.5.
|
||||
| **Delta** | **-45%** | **-36%** | **-44%** | **+30pp** |
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user