gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v2: LMetric PD-colo vs PD-disagg on the real agentic trace

Browse Source 
Anchor experiment for the clean-stack PD comparison using the canonical
cache-aware proxy with --policy lmetric (scripts/bench.sh harness). Two
traces x four arms = eight runs on dash1.

Headline: with the right routing baseline (LMetric), PD-colo holds 100%
completion on both traces while every static PD-disagg ratio fails
(14-65% completion), and the failure mode rotates with the split --
no static partition has a working operating point on this workload.
LMetric improves colo dramatically (TTFT p50 1.0s vs original §3 RR
7.0s; 7x) but does NOT rescue PD-disagg, confirming the bottleneck is
structural (D-pool admission + multi-turn KV accumulation), not routing.

Completion matrix:
                    first600s  full
  colo                 100%    100%
  pd6 (6:2)            58.7%   65.3%   (decode-bound)
  pd4 (4:4)            43.1%   43.9%   (both bottlenecks)
  pd2 (2:6)            22.3%   13.9%   (prefill-bound)

The original §3 RR "100% PD completion" appears to be a measurement
artifact of e13391e: producer-KV eviction acted as a relief valve,
letting more requests squeeze under the 600s timeout at the (uncosted)
price of cross-turn re-prefill. With the eviction off, PD-disagg is
worse than §3 advertised, not better.

Artifacts:
  analysis/v2/fig4l_lmetric.json     -- 8-arm summary data
  analysis/v2/PD_DISAGG_LMETRIC.md   -- writeup + reproduce recipe
  figs/v2/fig4_lmetric_pd_vs_colo.png -- 4-panel comparison figure
  microbench/fresh_setup/plot_fig4l_lmetric.py -- plot script
This commit is contained in:
Gahow Wang
2026-05-31 20:15:10 +08:00
parent fafc44da79
commit 7529284cee
4 changed files with 222 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

108
analysis/v2/PD_DISAGG_LMETRIC.md Normal file
View File

@@ -0,0 +1,108 @@
# PD-colo vs PD-disagg on the real agentic trace — LMetric (cache-aware) clean-stack anchor
**Figure:** [`figs/v2/fig4_lmetric_pd_vs_colo.png`](../../figs/v2/fig4_lmetric_pd_vs_colo.png)
**Data:** [`analysis/v2/fig4l_lmetric.json`](fig4l_lmetric.json)
**Date:** 2026-05-31 · Hardware: dash1, 8×H20 · Model: Qwen3-Coder-30B-A3B-Instruct
· vLLM 0.18.1 (V1, chunked-prefill on, `e13391e` eviction gated **off**)
· Mooncake 0.3.11 · Routing: cache-aware proxy with **`--policy lmetric`**
· Replayer per-request timeout 600 s.
## TL;DR
On the production agentic trace with the *right* routing baseline (LMetric, cache-aware),
**PD-colo (8× kv_both) keeps 100 % completion on both traces** and matches the daily-bench
expectation (~17 min for the high-load first600s, ~50 min for the full trace, with E2E p50
~3 s and TTFT p50 ~1 s — **3.5–7× better than the original §3 round-robin baseline at the
same wall-clock**). Every static **PD-disagg ratio fails** (14–65 % completion), and the
failure mode rotates predictably with the split — **no static partition has a working
operating point on this workload**. LMetric improves colo dramatically; it does *not*
rescue PD-disagg, confirming the bottleneck is structural (D-pool admission capacity +
multi-turn KV accumulation), not routing.
## Setup
- Trace: `w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions, agentic multi-turn,
contexts up to ~112 k tokens; "first600s" variant = same heavy sessions compressed
into 600 s → 807 reqs at 3.2× higher arrival rate).
- 8 instances on 8 GPUs.
- `--mode baseline` for colo (plain vLLM); `--mode pdsep --pd-ratio P:D` for the three PD
splits, all with Mooncake KV transfer.
- Cache-aware proxy with LMetric scoring (`P_tokens × num_requests`) + session affinity
for multi-turn (the colleague's canonical baseline).
## Results
### first600s (1.35 req/s, high-load stress)
| arm | success | E2E mean / p50 / p90 / p99 | TTFT p90 | TPOT p99 | TPS | wall |
|---|---|---|---|---|---|---|
| **colo (8C)** | **807/807 = 100 %** | 11.1 / 3.27 / 28.6 / 95.9 s | 14.5 s | 388 ms | 226 | 17.0 min |
| pd6 (6:2) | 474/807 = **58.7 %** | 83.2 / 6.75 / 382 / 542 s | 380 s | 19 ms | 40 | 55 min |
| pd4 (4:4) | 348/807 = **43.1 %** | 203 / 215 / 477 / 575 s | 475 s | 25 ms | 15 | 114 min |
| pd2 (2:6) | 180/807 = **22.3 %** | 380 / 536 / 579 / 602 s | 577 s | 18 ms | 34 | 321 min* |
### Full trace (0.42 req/s, original §3 anchor load)
| arm | success | E2E mean / p50 / p90 / p99 | TTFT p90 | TPOT p99 | TPS | wall |
|---|---|---|---|---|---|---|
| **colo (8C)** | **1214/1214 = 100 %** | 10.9 / 3.13 / 29.6 / 93.7 s | 16.9 s | 254 ms | 125 | 49.9 min |
| pd6 (6:2) | 793/1214 = **65.3 %** | 61.9 / 3.70 / 307 / 477 s | 300 s | 18 ms | 46 | 94 min |
| pd4 (4:4) | 533/1214 = **43.9 %** | 131 / 8.22 / 468 / 531 s | 467 s | 21 ms | 13 | 231 min |
| pd2 (2:6) | 169/1214 = **13.9 %** | 195 / 6.82 / 552 / 593 s | 549 s | 13 ms | 1 | 563 min |
\* The pd2 wall-clock is dominated by per-request timeouts (`request_timeout=600 s`)
draining concurrently behind the multi-turn session causality.
## Five clean findings
1. **LMetric+colo is the right baseline.** Full-trace colo wall **49.9 min ≈ the original
§3 RR's 49.9 min**, but E2E p50 **3.13 s vs §3's 10.8 s (3.5×)** and TTFT p50
**1.02 s vs §3's 7.0 s (7×)**. Same throughput envelope, far better latency — by virtue
of cache-aware routing concentrating each session's turns onto one instance for
prefix-cache reuse. The original §3 RR was an *unfairly weak* colo baseline.
2. **Every static PD-disagg ratio fails on the agentic workload.** Completion drops to
14–65 %, on *both* traces. The drop is not a high-load artifact (it holds at the
original §3 arrival rate of 0.42 req/s); it is structural.
3. **Failure mode rotates predictably with the P:D split:**
- **pd2 (2 producers)** → prefill-bound → 78–86 % TTFT timeouts.
- **pd6 (2 decode)** → decode-admission-bound → 35–41 % TTFT timeouts.
- **pd4 (4P+4D)** → both bottlenecks hit → 57 % TTFT timeouts.
- **No static ratio works.** Colo's elastic 8-GPU pool absorbs whichever phase is
hot at the moment.
4. **Decode isolation works, but doesn't matter under failure.** TPOT p99 on every PD
arm is **13–25 ms** — an order of magnitude better than colo's 254–388 ms — but the
win applies only to the 14–65 % of requests that get admitted. The other 35–86 %
time out before ever seeing a first token, so the TPOT win is invisible to the end user.
5. **The §3 RR "100 % PD completion" was a measurement artifact.** Original §3
(contaminated stack, RR routing) reported 100 % completion for pd6/pd4. LMetric on
the clean stack shows 44–65 %. Most plausible mechanism: `e13391e`'s eviction of
producer KV on every transfer acted as a **relief valve**, reducing producer-pool
pressure and letting more requests squeeze under the 600 s timeout — at the (uncosted)
price of cross-turn re-prefill. With the eviction off, producers retain prefix
correctly → cache works on PD too → but the cache itself contends for producer
pool capacity, and the decode-pool admission ceiling tips earlier. **PD-disagg is
worse on agentic than §3 advertised, not better.**
## Reproduce
```bash
# On dash1, from the main repo /home/admin/cpfs/wjh/agentic-kv:
for TR in w600_r0.0015_st30.jsonl w600_r0.0015_st30_first600s.jsonl; do
TRACE=traces/$TR bash scripts/bench.sh --tag fig4l_lmetric_colo_${TR%.*} \
--mode baseline --policy lmetric
for r in 6:2 4:4 2:6; do
TRACE=traces/$TR bash scripts/bench.sh --tag fig4l_lmetric_${r/:/p}_${TR%.*} \
--mode pdsep --pd-ratio $r --policy lmetric
done
done
python microbench/fresh_setup/plot_fig4l_lmetric.py
```
Source `bench.sh` cleans GPUs before each arm and writes `metrics.jsonl` +
`metrics.summary.json` per tag. Aggregation script: see the inline JSON dump used
to build `analysis/v2/fig4l_lmetric.json`.