gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Paper section: system analysis + workload figures + KV-wall model

Browse Source 
Adds the system-level argument resolving the roofline/PD-sep paradox.
Even at 95% cache reuse prefill stays compute-bound (the C6 roofline
fact), yet PD separation regresses TTFT 72%. The new system_analysis.md
walks through six layers showing why the roofline claim is necessary
but not sufficient, with the falsifiable condition being decode-side
KV memory budget: concurrent_decode * KV_per_req / (N_D * HBM_pool).

For chatbot this ratio is << 1 at any layout; for agentic at p90+
context it goes >> 1 under 4P+4D and 6P+2D, predicting the empirical
97% decode KV occupancy. fig_kv_memory_wall.pdf visualizes the model
with audit-able constants; fig_c1a/b ground the per-request KV-size
inputs in the actual sampled trace (input p50=33.5k, p90=101k,
intra-session reuse 79.2%).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-25 11:41:31 +08:00
parent d71a111099
commit 4028c587b1
7 changed files with 404 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
analysis/pd_sep_paper_section/README.md
View File

@@ -9,28 +9,46 @@ not the whole paper.
```
analysis/pd_sep_paper_section/
├── README.md # this file
├── system_analysis.md # why PD-sep loses despite compute-bound prefill (6 layers)
├── scripts/
│ ├── plot_workload.py # C1: input/output CDF + KV reuse decomposition
│ ├── plot_roofline.py # C6: prefill roofline at varying cache reuse
│ └── plot_routing_lever.py # C7: routing vs PD-sep as design levers
│ ├── plot_routing_lever.py # C7: routing vs PD-sep as design levers
│ └── plot_kv_memory_wall.py # KV mem-wall: the system-level explanation
└── figures/
├── fig_c6_roofline.pdf # rendered locally (analytical, no trace needed)
├── fig_c7_routing_lever.pdf # rendered locally (from REPORT.md §3.1)
└── (fig_c1a_io_cdf.pdf, # produced on dash0 when trace is available
fig_c1b_reuse.pdf)
├── fig_c1a_io_cdf.pdf # input/output token CDF (from traces/w600_r0.0015_st30.jsonl)
├── fig_c1b_reuse.pdf # KV reuse decomposition: 79% intra-session
├── fig_c6_roofline.pdf # analytical roofline
├── fig_c7_routing_lever.pdf # routing vs PD-sep (legacy data, footer caveat)
└── fig_kv_memory_wall.pdf # the explanatory figure for system_analysis.md
```
## Candidate claims -> figures (status)
| Claim | Figure | Status |
|---|---|---|
| C1: 98% prefill share + 91% intra-session KV reuse | `figures/fig_c1a_io_cdf.pdf`, `figures/fig_c1b_reuse.pdf` | **needs trace on dash0** |
| C1a: agentic input distribution (p50=33.5k, p90=101k, p99=132k); I/O = 142x | `figures/fig_c1a_io_cdf.pdf` | **rendered** |
| C1b: 79% intra-session reuse + 0.8% cross-session | `figures/fig_c1b_reuse.pdf` | **rendered** |
| C2: PD-sep vs Combined headline numbers | (not yet) | **needs re-run without --enforce-eager on `traces/w600_r0.0015_st30.jsonl`** |
| C3: decode KV cache memory wall (time-series) | (not yet) | needs step-level vLLM telemetry during PD-sep run |
| C4: TTFT stacked breakdown (prefill / KV pull / decode wait) | (not yet) | needs per-request breakdown.json from PD-sep run |
| C5: cuda-graph ablation (eager vs cudagraph × Combined vs PD-sep) | (not yet) | needs the 2×2 matrix |
| C6: prefill stays compute-bound at 95% reuse | `figures/fig_c6_roofline.pdf` | **rendered** |
| C7: cache-aware routing is a larger lever than PD-sep | `figures/fig_c7_routing_lever.pdf` | **rendered** (legacy data, footer caveat) |
| KV-WALL: per-D-instance KV demand vs PD layout (system mechanism) | `figures/fig_kv_memory_wall.pdf` | **rendered** (analytical, audit constants in script) |
## System-level argument (`system_analysis.md`)
The doc answers: *if prefill stays compute-bound even at 95% reuse, why
does PD separation not help?* Six layers, each pointing to a figure in
this directory:
1. compute-bound is a *kernel* property, not a system claim
2. absolute prefill work after cache hit is small (~hundreds of ms savings ceiling)
3. PD separation relocates compute; it doesn't accelerate it
4. PD separation's costs (KV transfer, decode-side concentration) scale with workload size
5. **decode-side KV memory wall** — quantified in `fig_kv_memory_wall.pdf`
6. the DistServe / Splitwise assumption that silently breaks: `concurrent × KV/req / (N_D × HBM)` is ≪ 1 for chatbot but ≥ 1 for agentic at p90+ context
## In-place edits made for this task
@@ -58,7 +76,7 @@ old `--enforce-eager` setup. Update them once the new runs land.
From repo root:
```bash
# C1 (needs sampled trace on dash0)
# C1 (needs traces/w600_r0.0015_st30.jsonl; ~1.2 MB, pull from dash0 if missing)
.venv/bin/python analysis/pd_sep_paper_section/scripts/plot_workload.py \
--trace traces/w600_r0.0015_st30.jsonl
@@ -67,9 +85,12 @@ From repo root:
# C7 (hardcoded REPORT.md §3.1 numbers; no inputs)
.venv/bin/python analysis/pd_sep_paper_section/scripts/plot_routing_lever.py
# KV mem-wall (analytical; audit constants at top of the script)
.venv/bin/python analysis/pd_sep_paper_section/scripts/plot_kv_memory_wall.py
```
All three default `--outdir` to `analysis/pd_sep_paper_section/figures`.
All four default `--outdir` to `analysis/pd_sep_paper_section/figures`.
## Caveats / open items