Gahow Wang
ad754cfe0b
Sweeps GPU KV-cache capacity (--num-gpu-blocks-override) under a closed-loop replay (concurrency 4) of a controlled multi-turn workload (cumulative intra-session prefix, gen_synth_trace.py), measuring realized APC (prefix_cache hits/queries delta) and latency per capacity. Result: a sharp knee at 3.6 GB = exactly the active working set (4 sessions x 0.91 GB). APC rises 7->12->36->80% then saturates at the ~71% intra-session ceiling; TTFT p90 collapses 13.0 s -> 0.53 s at the same point; dead flat to 14.5 GB, 100% completion throughout. So only the active working set needs HBM; capacity beyond it -- and the CPU/storage tier built to chase the reuse tail -- buys ~0. Knee scales linearly with concurrency = cluster GPU count. README.md ties exp(a)+exp(b) into the section-2.2 GPU-hit-first argument with tables, conclusions, and caveats. Raw per-request dumps gitignored; summary/m0/m1 deltas kept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
98 lines
2.2 KiB
JSON
98 lines
2.2 KiB
JSON
[
|
|
{
|
|
"blocks": 768,
|
|
"gb": 1.207959552,
|
|
"apc": 0.07353854948550977,
|
|
"completion": 1.0,
|
|
"n_ok": 144,
|
|
"n": 144,
|
|
"ttft_p50": 8.315758996002842,
|
|
"ttft_p90": 13.000879739003722,
|
|
"e2e_p50": 11.904735280026216,
|
|
"e2e_p90": 16.53674147298443
|
|
},
|
|
{
|
|
"blocks": 1024,
|
|
"gb": 1.610612736,
|
|
"apc": 0.12170482411635379,
|
|
"completion": 1.0,
|
|
"n_ok": 144,
|
|
"n": 144,
|
|
"ttft_p50": 4.015194748993963,
|
|
"ttft_p90": 8.895869197003776,
|
|
"e2e_p50": 7.799231034005061,
|
|
"e2e_p90": 12.4102137539885
|
|
},
|
|
{
|
|
"blocks": 1536,
|
|
"gb": 2.415919104,
|
|
"apc": 0.36322752074570874,
|
|
"completion": 1.0,
|
|
"n_ok": 144,
|
|
"n": 144,
|
|
"ttft_p50": 0.46762072801357135,
|
|
"ttft_p90": 4.615992321021622,
|
|
"e2e_p50": 4.144864278001478,
|
|
"e2e_p90": 8.661657008022303
|
|
},
|
|
{
|
|
"blocks": 2304,
|
|
"gb": 3.623878656,
|
|
"apc": 0.8029661016949152,
|
|
"completion": 1.0,
|
|
"n_ok": 144,
|
|
"n": 144,
|
|
"ttft_p50": 0.4056103950133547,
|
|
"ttft_p90": 0.532125736004673,
|
|
"e2e_p50": 4.129167931008851,
|
|
"e2e_p90": 4.328828729019733
|
|
},
|
|
{
|
|
"blocks": 3072,
|
|
"gb": 4.831838208,
|
|
"apc": 0.7291666666666666,
|
|
"completion": 1.0,
|
|
"n_ok": 144,
|
|
"n": 144,
|
|
"ttft_p50": 0.4871154689753894,
|
|
"ttft_p90": 0.6493310299993027,
|
|
"e2e_p50": 4.035265229002107,
|
|
"e2e_p90": 4.273102787992684
|
|
},
|
|
{
|
|
"blocks": 4608,
|
|
"gb": 7.247757312,
|
|
"apc": 0.7291666666666666,
|
|
"completion": 1.0,
|
|
"n_ok": 144,
|
|
"n": 144,
|
|
"ttft_p50": 0.4874342739931308,
|
|
"ttft_p90": 0.6399849629960954,
|
|
"e2e_p50": 4.077990949008381,
|
|
"e2e_p90": 4.249602819007123
|
|
},
|
|
{
|
|
"blocks": 6144,
|
|
"gb": 9.663676416,
|
|
"apc": 0.7291666666666666,
|
|
"completion": 1.0,
|
|
"n_ok": 144,
|
|
"n": 144,
|
|
"ttft_p50": 0.4956600739969872,
|
|
"ttft_p90": 0.649673483974766,
|
|
"e2e_p50": 4.049805466987891,
|
|
"e2e_p90": 4.187004164006794
|
|
},
|
|
{
|
|
"blocks": 9216,
|
|
"gb": 14.495514624,
|
|
"apc": 0.7291666666666666,
|
|
"completion": 1.0,
|
|
"n_ok": 144,
|
|
"n": 144,
|
|
"ttft_p50": 0.49285231801331975,
|
|
"ttft_p90": 0.6484746419882867,
|
|
"e2e_p50": 4.013530449010432,
|
|
"e2e_p90": 4.254351082985522
|
|
}
|
|
] |