gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Layerwise under load: overlap benefit survives (bg=16)

Browse Source 
mb7 with background decode load (8/instance). Critical-path transfer overhead
stays ~constant ~90ms for layerwise vs 158/239/749ms baseline (up to 7.9x at
32k), prefill not slowed, KV correct. Confirms the overlap holds on busy
instances. DESIGN.md updated with idle-vs-load table + the two blockers
(chunk-safety, concurrent-transfer safety) that the full 1200-req trace needs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-28 16:30:14 +08:00
parent fec50fa45d
commit e77bdcac5a
5 changed files with 387 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
microbench/connector_tax/layerwise/DESIGN.md
View File

@@ -114,11 +114,39 @@ Key signatures:
(`_handle_combined_pd_sep_v2`) dispatches sequentially and would need the
write-mode (concurrent) restructure.
## Results under LOAD (bg=16 background decode streams, 8 per instance)
Critical-path transfer overhead (ms), `total − prefill_only`:
| KV size | idle base | idle LW | **load base** | **load LW** |
|--------:|----------:|--------:|--------------:|------------:|
| 8k | 123 | 58 | 158 | **94** |
| 16k | 202 | 58 | 239 | **83** |
| 32k | 529 | 57 | **749** | **95** |
The overlap **survives load**: layerwise overhead stays ~constant (~90 ms)
under load while baseline grows to 749 ms at 32k (7.9× reduction). Prefill did
not slow (load LW `t_A` == load `prefill_only`); the transfer (0.56/1.46/4.37 s,
producer logs) ran inside the prefill window even with 16 concurrent decodes.
Correctness PASS under load.
## Verdict
The mechanism **works and delivers the predicted benefit**: layer-wise push
turns migration's KV-transfer cost from O(KV size) on the critical path into a
near-constant tail, by overlapping it with prefill compute — exactly what
MoRIIO's write mode does on AMD, now demonstrated on NVIDIA/Mooncake. Whether
it flips agentic *migration* to net-positive still depends on the busy-instance
behavior (caveat 1) and is the next experiment.
The mechanism **works and the benefit holds under load**: layer-wise push turns
migration's KV-transfer cost from O(KV size) on the critical path into a
near-constant ~90 ms tail, by overlapping it with prefill compute — what
MoRIIO's write mode does on AMD, now demonstrated on NVIDIA/Mooncake.
**BUT this is single-transfer, non-chunked.** Running the actual 1200-req trace
correctly needs two more pieces this PoC does NOT have:
1. **Chunk-safe tracking** — long agentic prompts force chunked prefill;
`save_kv_layer` then fires per-chunk and the monotonic counter would ship
uncomputed blocks. Needs slot-mapping-aware per-(request,chunk) tracking.
2. **Concurrent-transfer safety** — the global counter assumes one producer at
a time; the trace migrates from busy instances running other forwards.
Also: even with those fixed, layer-wise only removes the **transfer half** of
the measured migration overhead. The b3_v3_fullbreak profile showed dst-side
`T_kv_pull` = ~55% RDMA + ~45% control-plane GIL-dispatch stalls; layer-wise
hides the RDMA half but the control-plane half is orthogonal. So a trace
re-profile would show roughly the transfer half collapse, not the whole thing.