docs(kvc): redesign gpu_utilization figure to lead with system-total compute
Reviewer feedback: the original gpu_utilization figure was confusing.
"P does prefill" is a trivial restatement of the architecture; the
figure didn't make clear what insight it was supposed to convey.
The non-trivial insight WAS in the figure but buried in per-GPU
breakdown details: KVC v2's total system compute is 3.47M tokens
vs DP's 5.17M -- a 33% reduction for the same 4449-request workload.
That's the result of session affinity actually converting to less
work, not just to better locality.
Redesigned the figure to lead with that finding:
Left panel (NEW): system-wide compute as two stacked bars
- KVC: P heavy prefill (1.07M) + D append-prefill (1.39M) + decode (1.01M)
- DP: full prefill (4.17M) + decode (1.00M)
- Big "-33% total compute" badge bracketed by an arrow between the
bar tops makes the headline number unmissable
Right panel (kept, simplified): per-GPU work distribution
- Same color coding as the left panel, so the architecture story
flows from "what work the system does" to "where it happens"
- In-panel annotation boxes describe the two architectural shapes
(specialized P + light D vs uniform fused workers)
- Removed the second legend that was overlapping bars
Doc §4.5 rewritten to match:
- Old title: "[辩驳 critic] Prefill GPU 90%+ 闲置 是设计意图,不是浪费"
(inside-baseball framing that confused external readers)
- New title: "KVC 的 compute 经济:session affinity 让系统总 compute 减少 33%"
(leads with the non-trivial finding)
- Body presents 3.47M vs 5.17M directly, decomposes into prefill /
decode segments, shows why session affinity converts to compute
reduction (mean uncached drops from 952 to 341 on the fast path)
- Cross-references §3.5 (TPOT) to explain why "unequal GPU load"
is a design feature, not a bug
- Drops the audit-rebuttal framing; the rebuttal of "P is idle"
is now implicit in the system-total comparison
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -367,33 +367,38 @@ Critic 的 framing:
|
||||
|
||||
→ 论文里这是 **contribution**,不是 caveat:KVC 的 mechanism 让 27% 更少的总池子产生了更高的 retention 效率。
|
||||
|
||||
### 4.5 [辩驳 critic] "Prefill GPU 90%+ 闲置" 是设计意图,不是浪费
|
||||
### 4.5 KVC 的 compute 经济:session affinity 让系统总 compute 减少 33%
|
||||
|
||||
Critic 的 framing:
|
||||
> KVC 1P3D 中 prefill GPU 只在 8.3% 请求时被激活;实际工作 GPU 只有 ~3.08 个,对比 4DP CA 的 4 个 fused GPU 不公平。
|
||||
**头条事实**:在同样 4449 个请求的 workload 上,KVC v2 整个系统消耗的 compute tokens 比 4DP CA 少 33%。
|
||||
|
||||
**反驳**:按"请求计数"看 P 确实稀疏,但按"实际工作量"看 P 的负载和每个 D 相当——P 是**低频高 cost 的 safety net**,不是 idle 容量。
|
||||

|
||||
|
||||

|
||||
**左图 — 系统总 compute(堆叠条形图)**:
|
||||
- KVC 1P3D v2 总 compute = **3.47M tokens**
|
||||
- P-side 重 prefill(reseed/seed 路径,8.3% 请求):1.07M
|
||||
- D-side append-prefill(91.6% direct-to-D 路径,每个请求平均仅 341 token):1.39M
|
||||
- Decode:1.01M
|
||||
- DP 4-way CA 总 compute = **5.17M tokens**
|
||||
- Full prefill(每个请求都是 mean 952 uncached token):4.17M
|
||||
- Decode:1.00M
|
||||
|
||||
**左图 — 请求计数视图**:KVC P GPU 仅处理 328 个请求(7.4%),而 KVC D 各处理 ~1450 个(33%),DP 各处理 ~1100 个(25%)。**乍看像 critic 说的"P 闲着"**。
|
||||
差异的根因**完全在 prefill 段**:DP 每个请求做 mean 952 token 的 uncached prefill,KVC 91.6% 请求只做 mean 341 token 的 append-prefill(剩 8.3% 走 P 做平均 5455 token 的重 prefill)。session affinity 让 91.6% 请求的 prefix KV **已经在目标 D 上 resident**,下次 turn 只需算 append delta——**这就是 cache 复用直接折算成 compute 减少的过程**。
|
||||
|
||||
**右图 — 工作量视图(compute tokens)**:
|
||||
- KVC P GPU:**1.07M tokens 的 prefill 工作**(仅 prefill,无 decode)
|
||||
- KVC D GPU 每个:~0.80M tokens(小量 append-prefill + 全部 decode)
|
||||
- DP 每个 worker:~1.30M tokens(全套 prefill + decode)
|
||||
**右图 — per-GPU 工作分布(同样 8 个 GPU)**:
|
||||
- KVC 把 compute **不均匀分配**:P 专门承担 1.07M 的重 prefill(不做 decode),3 个 D 各自只承担 ~0.80M 的轻 append + decode 混合。
|
||||
- DP 把 compute **均匀分配**:每个 fused worker ~1.25M(full prefill + decode 必须在同 GPU 上交替)。
|
||||
|
||||
→ **KVC P GPU 的 per-GPU 工作量与每个 KVC D GPU 相当**——只是分布在少数(328)个高强度请求上(每个 reseed 5K-90K tokens)。它不是空转,是 **low-frequency, high-cost safety net**。
|
||||
这种"不均匀分配"是 KVC 的设计意图,不是 load imbalance bug:
|
||||
1. **重 prefill 被隔离**——P 的 prefill kernel 不会插队进 D 的 decode batch,decode 端 batching 几乎无 jitter(详见 §3.5 TPOT 双方完全重合)
|
||||
2. **D 端只做小 append**(mean 341 token vs DP 的 952 token),prefill kernel 占的 GPU 时间从 ~10ms 降到 ~1ms,对 decode batching 的干扰从主导变为可忽略
|
||||
3. **总 compute 不依赖每个 GPU 满载** —— "P 闲着但当它工作时承担全部重活" 是合理的分工
|
||||
|
||||
**总工作量对比**:
|
||||
- KVC 4 个 GPU 合计 ~3.47M tokens 工作
|
||||
- DP 4 个 GPU 合计 ~5.17M tokens 工作(**KVC 减少 33% compute**——这是 session affinity 带来的 cache 复用收益)
|
||||
**Paper 论述角度**:这张图证明 session affinity 不是只产生 locality 收益,而是直接把 locality **折算成系统层面的 compute 减少**。具体地:
|
||||
- 91.6% 请求的 uncached_tokens 从 mean 952(DP)降到 mean 341(KVC direct-to-D)= 工作量减少 64%
|
||||
- 8.3% 请求的 uncached_tokens 在 KVC 里上升(mean 5455 reseed vs DP 全部 mean 952)但请求数小
|
||||
- 加权平均后 KVC 系统总 prefill compute 减少 67%(1.07M+1.39M vs 4.17M),加上不变的 decode 后总 compute 减少 33%
|
||||
|
||||
这两点综合:KVC 用 **同样 4 个 GPU、更少总 KV pool、更少总 compute**,做到了 latency / TTFT mean/p50/p90 全胜。
|
||||
|
||||
**论文应当把这条作为 architectural rationale 写出来:KVC 用 P 的低频专用化换 D 端的 TTFT 稳定性。**
|
||||
|
||||
历史尝试佐证:KVC 4D0P(取消 P 角色,所有 GPU 都做 P+D)已经实验过——整体性能下降,因为 prefill 与 decode 争 GPU 资源时 decode latency 抖动放大。
|
||||
历史尝试佐证:KVC 4D0P(取消 P 角色,所有 GPU 都做 P+D,类似 DP)已经实验过——整体性能下降,因为 prefill 与 decode 争 GPU 资源时 decode latency 抖动放大。这反过来印证 "P 专门化" 的设计价值:它让 D 的 decode 路径**永不与重 prefill 在同 GPU 上争资源**。
|
||||
|
||||
### 4.6 v2 N=1 + 新代码路径未验证确定性 — **MINOR,方法学待办**
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 216 KiB After Width: | Height: | Size: 244 KiB |
Reference in New Issue
Block a user