Invalidate prior A/B results + add proper experiment harness

Prior cross-machine comparison (commit 1e86285) was invalid: dash0
baseline used warm instances with residual KV cache, inflating TTFT
by 2x. Evidence: inst_7 APC=68.3% impossible from 25 cold-start
requests; WARM TTFT p90=3.3s vs fresh=0.26s.

Fair same-machine comparison (both fresh restart on dash0):
  Baseline:    TTFT50=1.075  TPOT90=0.076  E2E50=5.075  OK=198/200
  Elastic P2P: TTFT50=1.018  TPOT90=0.085  E2E50=6.977  OK=195/200
Elastic is WORSE due to Mooncake kv_both memory overhead.

Changes:
- REPORT.md: rewrite §3-4 with corrected results, add §3.5 errata
- pd_separation_analysis.md: update elastic TL;DR with correct numbers
- cache_aware_proxy.py: fix double-decrement bugs in offload path,
  add 120s prefill timeout with co-located fallback (HEAVY_COLO_FALLBACK)
- bench.sh: standardized experiment harness with guaranteed GPU cleanup
  and fresh-state verification (nvidia-smi check before start)
- run_elastic_stability_test.sh: two-phase elastic vs baseline test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-22 17:54:21 +08:00
parent e4fa56cb1e
commit fc92410ec9
5 changed files with 775 additions and 116 deletions

View File

@@ -13,12 +13,14 @@ We benchmarked PD separation (prefill-decode disaggregation) against PD co-locat
Per-request breakdown shows **87.7% of TTFT** is spent waiting for KV cache memory on decode instances, not prefill compute or KV transfer.
**Elastic P2P offload** (selective disaggregation of HEAVY requests only) recovers the wins of PD separation without the memory wall:
**Elastic P2P offload** (selective disaggregation of HEAVY requests only, Mooncake kv_both): under fair same-machine fresh-restart comparison, elastic does NOT improve over baseline. Mooncake kv_both memory overhead outweighs prefill isolation benefit at moderate load.
| Config (TP=1, 8×H20) | TTFT p50 | TPOT p90 | E2E p50 | Effective APC |
|---|---|---|---|---|
| Combined DP=8 (baseline) | 2.383s | 0.117s | 10.232s | ~40% (skewed) |
| Elastic P2P (cap=4) | **1.315s** | **0.075s** | **5.708s** | ~70% (balanced) |
| Config (TP=1, 8×H20, fresh) | TTFT p50 | TPOT p90 | E2E p50 |
|---|---|---|---|
| Combined DP=8 (baseline) | **1.075s** | **0.076s** | **5.075s** |
| Elastic P2P (kv_both, cap=4) | 1.018s | 0.085s | 6.977s |
> Earlier cross-machine comparison (commit `1e86285`) was invalidated — baseline used warm instances. See REPORT.md §3.5.
| **Delta** | **-45%** | **-36%** | **-44%** | **+30pp** |
---