gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Paper section: system analysis + workload figures + KV-wall model

Browse Source 
Adds the system-level argument resolving the roofline/PD-sep paradox.
Even at 95% cache reuse prefill stays compute-bound (the C6 roofline
fact), yet PD separation regresses TTFT 72%. The new system_analysis.md
walks through six layers showing why the roofline claim is necessary
but not sufficient, with the falsifiable condition being decode-side
KV memory budget: concurrent_decode * KV_per_req / (N_D * HBM_pool).

For chatbot this ratio is << 1 at any layout; for agentic at p90+
context it goes >> 1 under 4P+4D and 6P+2D, predicting the empirical
97% decode KV occupancy. fig_kv_memory_wall.pdf visualizes the model
with audit-able constants; fig_c1a/b ground the per-request KV-size
inputs in the actual sampled trace (input p50=33.5k, p90=101k,
intra-session reuse 79.2%).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-25 11:41:31 +08:00
parent d71a111099
commit 4028c587b1
7 changed files with 404 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
analysis/pd_sep_paper_section/scripts/plot_workload.py
View File

@@ -160,29 +160,39 @@ def plot_reuse(rows, out_path):
d = reuse_decomposition(rows)
total = sum(d.values())
parts = [
("intra-session reuse", d["intra_session_reuse_tokens"], "#2ca02c"),
("cross-session reuse", d["cross_session_reuse_tokens"], "#1f77b4"),
("intra-session reuse", d["intra_session_reuse_tokens"], "#2ca02c"),
("cross-session reuse", d["cross_session_reuse_tokens"], "#1f77b4"),
("first emission (reused later)", d["first_emission_will_reuse_tokens"], "#ff7f0e"),
("unique (never reused)", d["unique_no_reuse_tokens"], "#d62728"),
]
fig, ax = plt.subplots(figsize=(8.5, 1.9))
fig, ax = plt.subplots(figsize=(9.0, 2.2))
left = 0
handles = []
for label, val, color in parts:
frac = val / total
ax.barh(0, frac, left=left, color=color, edgecolor="white", height=0.6, label=label)
if frac > 0.025:
ax.text(left + frac / 2, 0,
f"{label}\n{frac*100:.1f}%",
ha="center", va="center", fontsize=8.5, color="white")
b = ax.barh(0, frac, left=left, color=color, edgecolor="white",
height=0.55, label=f"{label} ({frac*100:.1f}%)")
handles.append(b)
if frac > 0.04:
ax.text(left + frac / 2, 0, f"{frac*100:.1f}%",
ha="center", va="center", fontsize=10,
color="white", fontweight="bold")
left += frac
ax.set_xlim(0, 1)
ax.set_ylim(-0.6, 0.6)
ax.set_yticks([])
ax.set_xlabel("share of total cacheable tokens (block-aligned, 512 tok blocks)")
ax.set_title("Where do prefix cache hits come from? "
f"(N={len(rows)} requests, sampled trace)")
ax.legend(loc="upper center", bbox_to_anchor=(0.5, -0.45), ncol=4, fontsize=8, frameon=False)
ax.set_xticks([0, 0.25, 0.5, 0.75, 1.0])
ax.set_xticklabels(["0%", "25%", "50%", "75%", "100%"])
ax.set_title(
f"Where do prefix cache hits come from? (N={len(rows)} requests; "
"block-aligned 512-tok blocks)",
pad=8,
)
ax.legend(loc="upper center", bbox_to_anchor=(0.5, -0.20),
ncol=2, fontsize=9, frameon=False, handlelength=1.5,
columnspacing=2.5)
for spine in ("top", "right", "left"):
ax.spines[spine].set_visible(False)
fig.tight_layout()