From 21ffb3d4f77956d008b1815a3c0d46e0188ac390 Mon Sep 17 00:00:00 2001 From: Gahow Wang Date: Mon, 25 May 2026 11:47:33 +0800 Subject: [PATCH] PD-sep matrix infrastructure: bench.sh pdsep mode + matrix driver MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the experiment harness that gates the empirical claims (C2/C3/C4/C5) in the PD-sep paper section. Three pieces: 1. scripts/bench.sh: new --mode pdsep with --pd-ratio P:D, and an --eager flag to re-enable --enforce-eager for the cuda-graph ablation. pdsep reuses the elastic-mode Mooncake kv_both launch and swaps the proxy command from --combined to --prefill/--decode. baseline and elastic flows are unchanged. 2. analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh: matrix driver that runs {combined-ca, pdsep-4p4d, pdsep-6p2d} x cudagraph x 3 seeds by default (~2 h on dash0). --with-rr adds combined-rr; --with-eager doubles to ~5 h with the cuda-graph ablation. Skips completed runs, captures per-instance vLLM logs (needed for C3 step-level KV-utilization mining). 3. fig_kv_memory_wall.pdf: empirical anchor (star) at REPORT.md §3.3's observed 6P+2D 97% KV utilization. The marker lands on the model's predicted curve at p90 input, confirming the steady-state analysis. README updated with the run command, output layout, and the followup plotters that consume outputs/pd_matrix/. Co-Authored-By: Claude Opus 4.7 --- analysis/pd_sep_paper_section/README.md | 53 +++++- .../figures/fig_kv_memory_wall.pdf | Bin 23568 -> 26614 bytes .../scripts/bench_pd_matrix.sh | 160 ++++++++++++++++++ .../scripts/plot_kv_memory_wall.py | 16 ++ scripts/bench.sh | 77 +++++++-- 5 files changed, 279 insertions(+), 27 deletions(-) create mode 100755 analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh diff --git a/analysis/pd_sep_paper_section/README.md b/analysis/pd_sep_paper_section/README.md index 0fd3fa5..27e21bd 100644 --- a/analysis/pd_sep_paper_section/README.md +++ b/analysis/pd_sep_paper_section/README.md @@ -14,7 +14,8 @@ analysis/pd_sep_paper_section/ │ ├── plot_workload.py # C1: input/output CDF + KV reuse decomposition │ ├── plot_roofline.py # C6: prefill roofline at varying cache reuse │ ├── plot_routing_lever.py # C7: routing vs PD-sep as design levers -│ └── plot_kv_memory_wall.py # KV mem-wall: the system-level explanation +│ ├── plot_kv_memory_wall.py # KV mem-wall model + empirical anchor +│ └── bench_pd_matrix.sh # orchestrates the C2/C3/C4/C5 experiment matrix on dash0 └── figures/ ├── fig_c1a_io_cdf.pdf # input/output token CDF (from traces/w600_r0.0015_st30.jsonl) ├── fig_c1b_reuse.pdf # KV reuse decomposition: 79% intra-session @@ -92,17 +93,51 @@ From repo root: All four default `--outdir` to `analysis/pd_sep_paper_section/figures`. +## Running the experiment matrix (gating C2/C3/C4/C5) + +`bench_pd_matrix.sh` orchestrates the experiments that the unrendered +claims depend on. It uses the extended `scripts/bench.sh` (now supports +`--mode pdsep --pd-ratio 4:4|6:2` and `--eager` for the cuda-graph +ablation; all launchers no longer pin `--enforce-eager`). + +On dash0: + +```bash +cd ~/agentic-kv +# minimal set: 3 configs (combined-ca, pdsep-4p4d, pdsep-6p2d) x 3 seeds +# = 9 runs ~= 2 h +bash analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh + +# full matrix (adds combined-rr and the eager ablation) +bash analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh \ + --with-rr --with-eager +``` + +Each run writes to `outputs/pd_matrix/__seed/` with +`metrics.summary.json`, `breakdown.json`, `apc.txt`, `gpu_util.csv`, and +per-instance vLLM logs (the latter contain the step-level +`KV cache: X%` lines needed for the C3 time-series figure). + ## Caveats / open items - **C7 uses legacy data**. The footer of `fig_c7_routing_lever.pdf` says so: - PD-sep numbers come from the random-sampled trace + `--enforce-eager`. Re-run - on `traces/w600_r0.0015_st30.jsonl` with cuda-graphs on before paper-grade - citation. The plotting code keeps the source numbers in a single `ROWS` - table (top of `plot_routing_lever.py`) for a one-line swap. -- **C2/C3/C4/C5 figures are not produced** because the experiments have not - been re-run. The 4h matrix proposed in the prior conversation turn - (Combined + RR, Combined + cache-aware, PD-sep 4P+4D, PD-sep 6P+2D, plus - eager-vs-cudagraph ablation, ×3 seeds) is the prerequisite. + PD-sep numbers come from the random-sampled trace + `--enforce-eager`. + After `pd_matrix` lands, swap the four numbers in `plot_routing_lever.py`'s + `ROWS` table and re-render. +- **C2/C3/C4/C5 figures depend on `pd_matrix` outputs**. Followup plotters + (TBD) will read `outputs/pd_matrix/*/metrics.summary.json`, + `breakdown.json`, and the `KV cache: X%` lines from per-instance logs to + produce: bar chart with error bars (C2), KV utilization time-series (C3), + TTFT stacked breakdown (C4), 2x2 cuda-graph ablation (C5). +- **C8 (mined logs)**: rejected — existing PD-sep `outputs/exp3_*` directories + have per-request metrics but no per-stage breakdown and no step-level KV + utilization. C3/C4 require fresh runs with proxy `/breakdown` collection + (already automatic in `bench.sh collect_artifacts()`). - **C6 is analytical**, so it is independent of any re-run. The numbers match `scripts/compute_roofline.py` (constants are duplicated; if one changes, the other must change too). +- **fig_kv_memory_wall.pdf** is analytical with one empirical anchor (the + star marker for REPORT.md §3.3 6P+2D @ 97 % KV utilization). It does not + need a re-run, but the empirical anchor's *pinpoint* would be more + rigorous from a `pd_matrix` 6P+2D log (KV-utilization time-series rather + than the single snapshot). diff --git a/analysis/pd_sep_paper_section/figures/fig_kv_memory_wall.pdf b/analysis/pd_sep_paper_section/figures/fig_kv_memory_wall.pdf index 323ca7c2e265553ae782e184696860b34bec2464..533cc14eb75ce91d13bb593415eb3b26d8af46c2 100644 GIT binary patch delta 9246 zcmZuzcOcdO*LUr%?OrMKV`tvI>`gXVWn|runU&cUiKxgY6_Hsep{|nbkeTcvQA(&J zBn|mK_xk!h&rhEBKllAP_nh-O`@AzALB2nSR9}NooL>bxbqd639YrA^UhP89|JW30 zzdI0q$<$9R@E5NjiyjKwLi})8<+oRGf}d(7vCVXS)Cxr+mV{@cxS_ev?%lh!_r>t1rbXA3+I^jSDU{qnq6d!qvR;UImg~fW#K)!V zf{eLsqbsBwBIL16&z0cvY-ep=|e?ApWPY$(w+Zcz@*{VJ+-0q_B#ZZ zjFcQZm5e}xbdmB=;eI(59-ml#G3-XKiGdf_S@De-yY&R~jn|TH0hd*eee!=-c|;x0 zxNcR4VH~5Bj3(mSzttPGh#g?;_XL*AoCdPOD*jnlm|TdP3yZQ?*mnW>aIFby@kgs+ ziTnK@<;cfeKu)=;0k=|hR-w=3F4+G#GzQ_o zFF6ZeJfc4F=8po8Z}qYbIV<^X*?+{D)Dlk~tHxQUZP~BG)o=CYP-^1%8qMP^hx0&E z-AKq86IIH@WE_;o*FVID&BuK4JL(*fH(af;HB|Vob4y;T#fXfvWfkoPoYa<~-k5mc=T<$He@@zN{5Iu*t4zOI&t;@78sV z@^Jc_8$x`k^@#UktKr-&wn4R;i~Vqhk=Eg+=iEG!rs4@47tl7>tKUAM0VL4?lJ535 zs`MQ^Qs0}%!^F~MYcVBL$P)*iQKDuSAv2P(S0NAfpOF6Nx%78>rhl{vVN z47Qwvo1~VX>%WtG*>5_EbUyfG!$drnNTSIo6YiP{+QN4?T<%k{ zJ5-7`z|!HHN{M4rj{HV?YO$_PZ+Sbtl!usSDrDetqQ>V9F0ZkA+>vyFw5ysB)p
5Nt^ez%-=^=TV+ z%-o*!T@6eno>=M-R(^zV~HsQn`bHNVzC@x#6a} z#b-Xpz4ssHq+Sgy2YmdcZwnthNFg+0MPbJxbMuYPD8K$7N&2i4tlkTI&21{-RCuAy zi!&r=wnqGnfAXZq+ncd(6m<4W1v(b1c3Ep2nVp$5GZ64;ynV@WfJL@%1;{fM`*}-4 zsaq_&NAPe~39`5%Wpo4tK_b17Ty74p^Vd1+ZaWSxF@VarHD$Z~^HIe8ZV;dleI?o~Etu5pFqn+W* zt<1wIZciD0W$vfwp06tq@JL*z@H%Q=Lm}g<7Uqfc@-@4ML~lY zV%-`&xZ;!inY6){PR}n2wLWZLbuJd#JASsliuZ(H_RtqIJf&KgDu8}XITyctIfNXc z0cC|=Hyqf%lw#)?vY&rnoHL8z!e^NbhEswP5ZlHNv0-23=lLHT)q7@8x>uO*hBI0; zgXy_%!Y5;wHlsF{v&SD!@s$;M4LZadD;b8gAPZpk*)mRN20$>@4`QB;CbhH;<~Z_x z6d+L5H0TP>8(1 zWt^(cV9U%@m*&kHZ*q=7MR=IO3e#0>QdpB9X2qrOG}3g_has!7l8K^zcUZYVXhmE| z-8+n>Y~HIzpg46kMq7qE;f6?^x}vrWKV#WB>)Lpeb;A~&FeU9Yb*`=e#cO9PnhUDU zHwPhh!x64JwjL)1YKm<3*^%xnWKwldKDxZo5cy?IuFH#G496=5EZE8C!VeGBcOCc? zHj@&9*AIfdD>rBg(I!zI-^c{pt;>fh+|Qh`?nW2u9gdF3FU+(CFxl-42=RT^mp+!A zcMjg`Sx|^RP+WXxpCn8pStiyC#mnf(Jf*9h_VK$J^L&p`U&&c+x0(ZCh8)%CKIQ{U zdwfx)EzD{5tlGi}Bk^X8U2smh)G|on6Fvt0XEqAGoQcYifRQ5fs6 zm6Tbi27 z)+}+822`n!zPus2TxT947jLM)v&ORLAM#~A`&A%C`oQ(ULi>!{WwQmJNVlT)dp)VC z$Z-^FFGpIyy^#%W43JA@%+1GP*Am$tG2K&kPIo=*D5fxN6-r@153jJEn~61j^w!j4 zatiY(TRJK{{>ZI)wi>k?nwJ^5tD_lT@>?Iu5#^jnh<`%Pey)zb-?6;f?#r)y68qq; zmz_lM#imb*lCZN@D&_ZM89AR!Uqi=rCx2vvXL_eAv7Tz}P3RdRBQZA`OBeNRrQaKy zG6&;?i_8i_zAXZjja2`U4vvv8?ou{FgR-{Cvr3&DKcKQ9%PhQQPa0c=2KpHbCBoVA zMYPi2>#4A99(QCZfAwCR=Xgg<#fQNG>032V?IqvLde7ZL^7caH3HYR#pGLU{a@5zE zH`2Nvzp8sen3bD4LC)cA=YJKnL|C6u!v}Dq?TXI$ENjQNQtpO>tHwGlyBp!K`%>e{fnIzEssxW6~DH;GU{}} zddQk1Log-lQ-DrQd&JSN9ufB*ZQ3>D=jNxqF#4zaBiK1JSJk{}IH@G)K=25OMjcg| zyXrj?^4)b{p}CL}r_h47CO&2kZWBK9Iy#Zp&0@N$WBU1zC2I7mOdrHvvx1r$Cf4~< zCoulSv9pUCx3tW!XenLsSXtfw?D>XMAEIhX`ISrearTc-3pVyvAPU3X)-G}{QO7l( zy!Vf&%e8cna@#Nd{eq55YA5paNwtB6CDKzn`+ZEWqn3DW9Vo5j3g&?n!wVU{%rUja zVMnFWmBq%t#eU(-7So=^@f=Y|$R2!>*MAq;%zndG#bNMDgHGM*tX1S5=Ig&=3+^HY z63qORl^n-UJEHiHz5x2(7>ap$_sE}Wj&jKls4Wmy?{+NJeZzI2qzWeiXK(P^tFPPA zu5J9|8)3!7;_Eu4GCu^u-3&FA%Ykq}W}X*F0KR-eGyS)(yEs3qnj7x^DXt&6Sx(}k zFrM5fd6sln%Iznk)5sS-SMkP;nV-|Hnnh}RziI`P8VIxxDlQ+jD(LheW#^0MtEhaq z3eJA6dB)!{oq^~STYF*g^G>K+&gnqD!vbS3Yi37I-#UT5PEoP&?j)=p(saG1qEgy5 z{HZ&n=@Fc26nsxw|MSIDb*lM&h@keB((K4nA+eZxejHY!bQm#%Fw|($xanAEp`{d% z6n|jkXkZ5?v+5+Z)bfVLKfgUkIL+U@UR!)LQ`R&PRtGOtpT0gJ9^@;Rkz3G%Xehjp zt9bOjUA3zlcPB-TN&uIp;?(LINH?5aTo9+S8{-HhK{<|PMwPggY{d&F)+js$tiAuJ z5Af;Mt;x%l-9Gt=qf_U2Vn_O&*&2XBbRq-9t`F77cg)xcEY#eYb?j*mPxF0WQ_}CJ z{E1H|c;m>#^Qx;B!ZFdLKNMC+_kgZ@znV1SKO6a%)UHybUN<(>&NW0 z)5A8?i~F;FEPC6DQZI(7X^v^K)^uN9DtM~)p6u{L<>%w&&{q_S16jzc>C?E-m85yg z!p51aPO(2?Pb{}>ghVI|&DlCr{=DcGA1-jHUKW33?)n=TBAL{jm``+_`c-g+%9HwM zx~lyBDJ|abER?b8Q69yL6VJ86ZC zbyv9Ohnl1*^rI5(QAI9GC8EOS^~;QT3< zi9N;m!7F?XNK-=~zq<@dWJ`MG!Yp<9hN?X`X} z>%vhBt+AQ<_kLin{gdiekrwOy8Cm7S+DMpA_DIPQ90+jRtea&K|5;7ZkLFZyw$1+( z5m$Lf@P7N37DH;SVnk`cP?$bnk(o;UC|}a0LB)xW`@3}Z*s=z$$TNz%e!vMcr`hR$ z)7YmZTC=jQ;uy%=+IpPO#F_GV{pJ`3A;OKI9u$ri)9IpG!NDs->HS8az zx49bNTSFKaz`KVTi^d|U(Y!@Wpg&34()?LHR|~+&Ja9u^o$<{-n%0*k$D1#3@fF_7 z50lGtONIuXaOA8}6S@0itI*f_ju@GVGbKlV^&;D@Id(m)A~;lrH0_-f zNdxf$vQ!bcG6Z}h00W%`&VaFdVc@F3IS?WU1Jn10fW@?5xL`U++y?_63MPPx`(Yr( zSe9aCTFHcBe>Deh(!Qz9&pwxZJNo(*`Lm zv{5dJaQmbk{?@qz^B#K9HO2gM%^T;3bq7l-0X<0*saGL-SFLK~Rb?naIZlt|Yvs&y z0*8BAMtU=-QYrgPSWT^lANV{eS}gyOw{pJOPBl&NR42D$*yP*cLwsu*ZQhzGrj1X( zmyKuZKSp$A4Ck2cF~^P%E_K2C)d?jT{eX^*bAlf^?I-_74FBZU=7JhdZTV?1)o~HwJqKVsf5V~7f+lCjm>Mj4$XO# z-0AQfuFOTWzz5 zxWnsth0d2fXU1*74@0E2sw;!Pf@{)}qTd?D$QlQS-vR7yj`dF*ivDwGQu@~)rf;nd zq9C6f3=|RJ1p6e|A$ai8UMVdM))r|sfMeu8c2*Yp!SD)H{;adMW=d8s3#=DH?KA6( zSMoe`NcsvS1CPcf4)g9WfCRJZQ&=5@7;OCDEn6Ce#bBW3{z1@M91i|ek^zl{9KeeJ zoJtk)g@7#*oYVs%s}RskQhdgfSjJIU5?}3c1 zI^7hrU7{T6tg9*!G=e?{v4y4ov z+;n<8Jm*5uu?o7(;9z+1Poq@LZOL!9f1SLM|GYU&wLtNhqE> zrCNPfk|mOpe-71JBpe{vjqI8E;yX*xthc(*prO-TJ8GfG$=Xy`5$#ac{lxOf+puP9 zR@ZAe-rr)8?R_#DC&qP4$~EsUGOs}+pPrNNpY5>~H^LRJIXZYekQga>5 z4Nl-^$fKb5aLi5Qjl?o$W&b=ZQaIOE1F#P3y*M-4ne5*@t^Krax>3zIv+=Og*o1z# z*?skaTD}=yMSljZfq1KN8B+V(^i=@5N9_4nQ*BnBZ2o8N-~Y+j z@~%r&;w|arjLm~I)re>d%;_YzA5FSX4lOuc>nplRAxWp4Yp8FGMHC&>^-i_5(@Zj8 zOU@m&H9rSEz3Rq%5T?Gd9+i2h!1&92=izI?>z+r{nGHJh!&UCZV=ZOwxE<`HR#w#g zGHuC>J>ci8=5WRAMoklQ_L_xt$Zv0s+utTC6#2zZcJaQ9TIl^~B*7s$G7`TewYnrc zT(x{%{5|Unam>;q;tCKjQx--o0>mJov>XgVpqHr;29I|@S|PE_mkbSA zp|`Z{QzZnATL!xHC9ZNvvwO_OA7<4SDk8E~$|fr^^|AUUm9gCeK3-71P^iGCl;Tfs zmeMZ4DsVQPA%2slZ*q)Vwlz-UD{bx^tm0(vXK!JD$Dz**WaL9Pjm zP^QgiDH*Er0Vo9IQ-Ogd@*tR^0;7JFPlJHN2mvrd@dem~go80k$H9YWdI!dW{um;7 zSJ@L}!@@yn6;DtL2d7?FAwWPeyZ|_dcnSW4hf#-+kq~f-2&1~7c_H9o5*$obmI6yK zUf>D|P8GpQK|p0S0nirL1MXFagJ^socvk~Xj0+qK}deU$|2YQw24 zs`3ypPa6*Es*QqvI&i9@dLjg5*Ml+QP@uJ@GzBDFl-A*7{u9NbHK1*MIr_QCUhm1x z=|hf$i zPvx*s$&5@XnR7KtDbTOK`em}u0u4<_Y~}?Z>-`M+|_`{cN-&!gM(#JGvZyNLg6=4c;za}RQg#vh;ox0 zYO$%06%v}%aui0JqJwWYpD6Rr9Jy0;Dd?2xc~^ZI!gNNDk@4w}{$eXwPWiVOZX)n?6n+Dr3zYSFHQi;94 zP$h2q{fkT){yAz?;l{MoU~B9jwvX34)S;lNz6{u^5e@1Zz^Rs+;k4agKrdTaa8pYk zJY)z5?X?ZTZ?xZP?Z;r25uAET=Nts|GZvtV=pi7C1U#sNgMn`P@3tBthPG*93=BbD zAvic{K!K3Jx3ss-9wTY+yrB;x5xM<&9zq0L4Paoo(Gx}@ni^pIl<^qCGsr(E5Wu4V z1o=d`yFY-(0tojo0FMU{x&T5CKkr_zeGC9_+i)}q9EBbsC?e1|kPP6q@qz%{Hdqjyx(yc$;I_RCb`K#3`jd~* zYG*sFTa=J3*J$;%m62d_h$lIa#hB)vLoeYir$U3>J;?xJ+kh~Du#Fw& z6GEoZx1w+aK!lwPq8IxXXgEOFf`|ThO%t~5hyV!N7MuhK+vt%1aT_}dAa1)JO%4g7 zbqe+BAyo(j3l7SMxZPP=-jO6lBvB5Dl#>TEHKF7{uON@((8v(-F(?usV!;kO zgbx-^uRZz!0s-jO1&zhfyX0>gJ-L6;aP-3bn?_IE-!$5+aJDBg;c1}P>}4o;dSZ6+ zps}<*|C@%U2V*B63Xj?)i^0)bWd|P;OF;gu4~aw4TLg)u{Fk#x9EM)lJB4vLA}uk0 z(FpXu+eyPCcR7p1qfvhc4~eHeZ~jY`hEF?qENQnwJiVgn=YO+-#1nTbBp^wDM;=MQ z{OjF+%M$SPw%)0aK%ndjHIj(j6;>n>Meod=!o>e4)FdQ*NbKYzVRl6Zg+$T|pLS?% z_@7!sAu*)C>k5U$ll}%qA&K;%wgUu33&JiM3cV|JC=?#K!$5lcQD`K+`F1Kq)6Om$ zmOjM);v>-K+g~&i>hE--FtqbG9|lLD`Tv(N22a{$42DP_?K?p*B-#f2i-tv^{;qTs z7E7P)J7ux7Li(3R`M>bv{+}q|F!WitQy7Qa9T^-EvkREk5xZpZ=>Gt3_!Tg~)w zz7qsbB<>0q0ZCtoJNXEhU9$*9z!7(OOd##9Eh3IUUqd?;5(%hXR+CV>+73lR?>2^n zp+5_D>LcOk>;5kq>3>r~`A_wuk!bp2*};fL;!uApMAL@F-wJ8qOawh`cSm9 zlO}{jqcHSm%}z!vo`eHioP=@o7ne;?P#8dyqra671|FlW7+Pv*AGAFZ8s;7nwzWlQ PLmP*M%E%a48bbdEz2JoA delta 6648 zcmZWrc_38l`)`mmbxAX1cZBR_U+melglv_in#tIb-B`jYgplnNLQ;`rZ%SoraS4^B z(w;3Xl0=B8erMeI-rqIOKj%50dEWQ)d6xHiXRZg4c@t5ahX4z9hDhE2D{*pB?dOQ< z6~FYKi|;by8QUzin3nY-9}kq-6R+ot*(oU6=frdd^u!k_mL0M98m!)Cyp_X)loUC{H?mAyTJKP{*8Ihsnb!9wEe-q-<4KgbRI93>Wpzr zyz7p*7Gcyept0|Jjq{>J%(H~4dtye+u2<*z!e)G@MsM*2E35FYfAoDz>U|K`h$$hDH>yVp- zwQ6^xV70R@h3ALeEfB}k{h|N3=B1@1(>oK&Jo>eE-|8`+%(S=ssX|gNOg4+xJlUpL zQ90iCyl?!^x}UM;p9b%ig$s|m&)?_&bL4Uw$`p0X;KeXrEFk%Dei#(d((3B;wU<`$ zE)$sPb$y=lQ9D}bbYrMk3^~C*>i6L`mh?b%aG7APq0ati(={SsPk2N{m%Fyq^qRS6 zivta2y03>PRKv#t=?cA_^Y8}AUOG+g?=_`0nYGV8(~pcxO!+0TOFJ$s5EDxZf_mnE?+e9jR`NfnP>P>^4CR?9%e*im$dem zumk6>6Rz(*`#-v!9<8EHldXLz9D((;Ueny!K;p2D@Bc--1)Wg% zt(Y zdVf=FW7ADXaMAf!1d|#0`@{TsHXgOO`MmGYC~J5;lsrGHeFNb=B%S-1pRYB3hk?AG=dYc7N(<3^u4gsk&URM` zXj8lkC_tW!87SBu_i}h!92Bh|SJf?g*XH9+zA@=`g#dFNBCg|B4?kb$Qp2XTGFSoxKe}klT#)lMaiWW2RJ_FcWQiKafzYF_?)eA7SZ^(I{^~944fM_vm z@3F{%4+WcSrWZEVYr6{QoOP8;^H1*fkNZZmqY?aiWWjrr`LL=9E zInGDPxy+fKTOWC6L*r+whd;L82&!!gxnS_ZK)?B>uCAU-moga3WM1e_Pai0fUpN{0 z`=SuFyOEscDPjhxzpzkNyhL6owtCj`=fn7{0A_CYy4t#Sz$Mm6XXnPQV$xw96we{u zxlhUDm*B$U7fX>vlZB5!cy8CGlk1-QvOLcvA}=L9G|?=7 z7blrrGrov@arBVWjR^57%D#n;p~BP}x+l6aNK;t~d~oSh1O1*w;&|pMO#H!)`u;Cc ziFVnvg#-NV*3~v$LH_44i^Xx$n+?;PRL>?3X)9(J#{m$SgP9uOkHCQvxKxewi=4cluS;lp@PcfHZocFy zU$)2PL_WO)N8*W7om;D`X+PEsU!O7|eUytpnfC_^4OkjpiIdA#@y*#I#U>>edL+)F88z-9`3t-u)I=3QA5Xfz4@ewy6XAYLxET6n)sN^?OsH z-TS>of)DDbjLQ3BRyMnD6|T#g=#ZAgNYkq}cjeuzGoS4!m$~>rz(4Ck%GLcO1<{7Q zgbH23fsfl;&Ky-z%q*WZbk^IEr=oUd+KopdeDC3Ow>4mAJ zZaYO;qiQ}A2NkCsSO<@2mD3>IlHs97Zw==Y?ZWkePX%=1Tc1f(#>W;L4(ktNBhra- zYQN^C=--^&_!E}mdTrB;nw2KxFP?U;I+cqp$c+)H6aBeO3=CG{6tEYB0w z`Vvfv;`m47e>C_W6-`Oayp(Bshv^ajGwOBv&P_BJFSwyg;lA(Tj1KQb4(lxFofwu-m~I^wUO z%Xk>7ylynDH!QnaIJ;`C8L4sTxTDJ6pU5=FHidQRcWpd`P8oL2C7daJ6<^+OI)r%D zfi`K*CaB@-ET%1GklDe1*}U%ZMYkx0Mvclyh0BAIwkE1?jN{TKF3{|M%d5FPEG@I& zCIMVRwHH~R6f;vQ?;!j67EkjTrq5Z2`WEW86E9g9mqnXPI2dp=jF0by95i|o?HB^beC_di=et7W9|GG>nphQ?1-~xWPIPN7 z4$NJOM_tK!*Oax3(QxSUY0L6T7nX}ma6LL+m_JKs2jeTD_R9z}Mkd8KJeCbx%~b^+Yg1tE0iTe&knNMPk@NBs}(3|hPZ09v!xl~AzIjYpPaLq zVt;uWep>xdubrC8lfJN&a>1w~jp!?MF&<62zwm|924iWy(Ja!NQ;C8&ZDjM+wGQGr zccu;VL<{KBwH{t;UJI0(;n!!2Yq^=3@E%vi@MT(`w%jW!#WcFsDW>;W0d?$BcXL|g zJQ4hAA?5Sgg6V!HCo)CMQSkXy>oxSFl?%d0e3Q^visH2_oTjZ>m zU71P_ErK!fn;x=GYvd&R+^(c2_&TWgG*Jm)8uWuN%dw6)Dcs~1P)1GKBeSjai~XPs za;M|P8fM=JW9C|d+n)H|bZCyK8=~^p)>FD(`_jifRt-qvh59 za@25;8PT;VM0;HeYrV4uZ?~S-zpYko`&Vo|Z&cKK>ib;_&iT9#)n7|)ZcBiOL+eBINlCI*kcURh8+F8D&vupWQe#{uE#Cs(}P`LkYzQT=>W^F!j^dL6S@ z&dew&v*xC8<17_5_1xK;F>(BNC876 z-#Jaf<8fV`uML-Wkn@Gt`uv#RVTn3S3)2N_g7uGzXg(WB-Dvmj%cG~;byDvoK9)9` zn~loOa7&-4s5 z9ek?Xw1v^wcji^{r2cg%q;%}>uQ7u_KDY&(YyLPGw4=NtBQ{Lm=kvmu!dU0MN1o>K z-|FZ2es52yo@j~U+{d_@oSN=|rAq_SMTM6qfBbfxF+X>Cf6M%cRaj6YGxogT{wD2~ zCSCoQa+}=AQ*VCV2)0?P@jyUH2;ugW_g-5+h>-KpoP0h#`Q+#1gSi*>?fb~qY1e{z z_-7wDK)r{&XaR$Lwn@1Lh3;2c_wP%InKSr2KeIH!099YA%e~FZ8pO>qT-G&)Yey^3 zeH9VhuW5_P$Z%GxJ(=e1Gsn-aIqCfjY3dw! zgBL6vZ|r$@jJaq;i&04t-`e$EE?s&^b^I&UHfOC+j=2{mXK~AY%;&pPeJ{s^MVqwX zSah5om|G*_M}N{MpMQs1-#yt7w(!JWPv^z$Jw+Adv0CzsffOL6&Y^Lrk@#CQDc1i{0leGy|>>bOtk7H4QD-iB&HwY9>%`q3G$4CAs^L3|3R+!gL1A zr`wkxQ}Bo3uiuSAsjqCOXMb8>&q?&Ye`CWFjp?bTN%De7>24!1+?C;iAGD3hJ0|<# z#IIJ@Qm?q4?tQ-kqZ^{nPkY(Bfq#D7Jr~>q@?88r8M)>aJ##cc_tv%ay9$#y@Ao3{ zeSapa6B8z=STBO%&1;Axk>~SXvWxK|g1?zk13%u->x8B9-YiH3D0C=H-t~EZTc~jR z5ml#Fa^MInV?u<=oUj=j0kcr(Z%Br}M@VQegLas01<-wwI6Qy_*am?@#j{K{U*Khd zq8bRuPZR~+-jV=SZb3oLVkaPDF%&c{77Pu-?{M)9$U__jiA%&m{1ParTXG00fZx$F zeNclW8ahQ%gc4i&npppQL6M%6Bv|uC5--8hxaxfs6frO4YV-WyU=`4XlVj=+?9$rlg zIq2gP>f`6ZV0~~VuLaEkJQi*-djPWoz~lb0tN=W_?O@Crz!TUO60;M)6aT?%06gg* z%MO5dT=sJU@RWZnX8=$A$D#pvEw+XH4<|t2P|yLwG6e`=4g!Q_4%iAfe3%9hI6NKz zfy2W92+KT348zCEI}{*rxcjHeHi0T4N1c?N_914It37eHL5MPj@G zBAXS2gY=FJ^k(=2L=LYHK;-Zq0*IXW`~V_{^Dsc<@WCrM@%j7sGXN5YF90BM$O70X zCy{{wi4$ZXEtui&$M6gVNH8JD&d)Lwyc`;dLmkKnKFIKgeK-Q(Z=V1UfW%P|=oc0O zkT^nu01`(?FhJ(02m#=xSuRZog9!&m=I99Vi2%qPB_V#a5YQVSbK(vOq#a}cf^fke132skw?RSAn#QUUb!|2GpOf*GDjEI^?^ zvNQ~cqp;hI{Q&*|a@oM+h&Xt6Zj4<_E{x3XLT-#*DQ=9db_E81uv3MF9?`b);_+l? znx+Wiv1|x_{*{8$BD0Ibg%R0pu@a+F*(YcPhR0Ia<9`K)CE{0wfhCft%aXzW5Mzl{ zEja%x7n5-8!^wpa*$vHwk+ir)V@VWtZ*uujssCMTvivg{%RUM#9m%-W82y!-3q6c>Jnb;qXMte~ZuF8E&uDSs>up6J{kbe6Uw#hCpUt4$FSv z|4uTFKxI$Km5Yhk)x<;+do*(Sk>Tr+8-tsH8zaH#T8V*J9Em_(l^PO-%H;_AY5lAE zU(xVXStXiG$w(_RvjHYZbfE7gK0i~j9{d$E`V|HiDJN`8o<~G1cU;d nJITV|6CU}nEcibLJl8`)X~Ch(Qyq`fB2tlxYZXmx&5-{G4g@HQ diff --git a/analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh b/analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh new file mode 100755 index 0000000..cc61420 --- /dev/null +++ b/analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh @@ -0,0 +1,160 @@ +#!/bin/bash +# Experiment matrix that rigorously evaluates PD separation vs Combined on +# the agentic trace (traces/w600_r0.0015_st30.jsonl), using cuda graphs by +# default and capturing step-level KV utilization for the C3 time-series +# figure. +# +# Matrix (default minimal set, ~2 h wall-clock on 8 x H20): +# Configs: combined-ca pdsep-4p4d pdsep-6p2d +# Modes: cudagraph +# Seeds: 1, 2, 3 +# +# Optional extensions: +# --with-rr also run combined-rr (refresh C7 routing-lever data) +# --with-eager also run each config with --enforce-eager (cuda-graph +# ablation; this doubles wall-clock) +# +# Output structure (on the host that runs this): +# outputs/pd_matrix/ +# __seed/ +# config.json, metrics.jsonl, metrics.summary.json +# breakdown.json, stats.json, apc.txt +# proxy.log, replayer.log, gpu_util.csv, gpu_snapshot.csv +# vllm_inst_*.log <-- step-level "KV cache: X%" lines for C3 +# +# Run on dash0: +# cd ~/agentic-kv +# bash analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/../../.." && pwd)" +BENCH="$PROJECT_DIR/scripts/bench.sh" +TRACE_DEFAULT="$PROJECT_DIR/traces/w600_r0.0015_st30.jsonl" + +# Defaults +TRACE="$TRACE_DEFAULT" +REQUESTS=850 +SEEDS=3 +WITH_RR=false +WITH_EAGER=false +DRY_RUN=false +TAG_PREFIX="pd_matrix" + +while [[ $# -gt 0 ]]; do + case "$1" in + --trace) TRACE="$2"; shift 2 ;; + --requests) REQUESTS="$2"; shift 2 ;; + --seeds) SEEDS="$2"; shift 2 ;; + --with-rr) WITH_RR=true; shift ;; + --with-eager) WITH_EAGER=true; shift ;; + --tag-prefix) TAG_PREFIX="$2"; shift 2 ;; + --dry-run) DRY_RUN=true; shift ;; + -h|--help) + sed -n '2,30p' "$0"; exit 0 ;; + *) echo "Unknown: $1"; exit 1 ;; + esac +done + +if [ ! -f "$TRACE" ]; then + echo "[ERROR] trace not found: $TRACE" + exit 1 +fi +if [ ! -x "$BENCH" ]; then + echo "[ERROR] bench.sh not found or not executable: $BENCH" + exit 1 +fi + +# Config table: name | bench.sh args +declare -a CONFIGS +CONFIGS+=("combined-ca|--mode baseline --policy linear") +CONFIGS+=("pdsep-4p4d|--mode pdsep --pd-ratio 4:4 --policy linear") +CONFIGS+=("pdsep-6p2d|--mode pdsep --pd-ratio 6:2 --policy linear") +if [ "$WITH_RR" = "true" ]; then + # round-robin via lmetric without affinity is the cleanest proxy for RR; + # to get pure RR you'd need to add a --policy rr to cache_aware_proxy. + # For now this slot is a placeholder so the matrix script is uniform. + CONFIGS+=("combined-rr|--mode baseline --policy lmetric") +fi + +declare -a MODES +MODES+=("cudagraph|") +if [ "$WITH_EAGER" = "true" ]; then + MODES+=("eager|--eager") +fi + +# Pretty-print matrix +echo "==================================================================" +echo " PD-sep paper section experiment matrix" +echo " trace = $TRACE" +echo " requests = $REQUESTS" +echo " seeds = 1..$SEEDS" +echo " configs = ${#CONFIGS[@]}" +echo " modes = ${#MODES[@]}" +echo " total runs = $(( ${#CONFIGS[@]} * ${#MODES[@]} * SEEDS ))" +echo "==================================================================" + +run_one() { + local config_name="$1"; local config_args="$2" + local mode_name="$3"; local mode_args="$4" + local seed="$5" + + local tag="$TAG_PREFIX/${config_name}_${mode_name}_seed${seed}" + local outdir="$PROJECT_DIR/outputs/$tag" + + if [ -d "$outdir" ] && [ -f "$outdir/metrics.summary.json" ]; then + echo "[skip] $tag (already complete)" + return 0 + fi + rm -rf "$outdir" # clear partial runs + + local cmd="bash $BENCH --tag $tag $config_args $mode_args \ + --trace $TRACE --requests $REQUESTS" + + echo + echo "[run] $tag" + echo " cmd: $cmd" + if [ "$DRY_RUN" = "true" ]; then + return 0 + fi + + # PYTHONHASHSEED is set inside bench.sh for elastic, but not for the + # other modes. We export a different seed per run for reproducibility + # of any RNG inside the proxy/replayer. + PYTHONHASHSEED=$((42 + seed)) eval "$cmd" + local rc=$? + if [ $rc -ne 0 ]; then + echo "[FAIL] $tag exited rc=$rc" + return $rc + fi + echo "[done] $tag" +} + +START_TS=$(date +%s) +N_DONE=0 +N_FAIL=0 +N_TOTAL=$(( ${#CONFIGS[@]} * ${#MODES[@]} * SEEDS )) + +for c in "${CONFIGS[@]}"; do + cfg_name="${c%%|*}"; cfg_args="${c##*|}" + for m in "${MODES[@]}"; do + mode_name="${m%%|*}"; mode_args="${m##*|}" + for s in $(seq 1 $SEEDS); do + if run_one "$cfg_name" "$cfg_args" "$mode_name" "$mode_args" "$s"; then + N_DONE=$((N_DONE + 1)) + else + N_FAIL=$((N_FAIL + 1)) + fi + ELAPSED=$(( $(date +%s) - START_TS )) + echo "[progress] $N_DONE done, $N_FAIL failed, $((N_TOTAL - N_DONE - N_FAIL)) remaining, ${ELAPSED}s elapsed" + done + done +done + +echo +echo "==================================================================" +echo " MATRIX COMPLETE: $N_DONE/$N_TOTAL succeeded, $N_FAIL failed" +echo " elapsed: $(( ($(date +%s) - START_TS) / 60 )) min" +echo " outputs: $PROJECT_DIR/outputs/$TAG_PREFIX/" +echo "==================================================================" diff --git a/analysis/pd_sep_paper_section/scripts/plot_kv_memory_wall.py b/analysis/pd_sep_paper_section/scripts/plot_kv_memory_wall.py index 77d2d82..ed23935 100644 --- a/analysis/pd_sep_paper_section/scripts/plot_kv_memory_wall.py +++ b/analysis/pd_sep_paper_section/scripts/plot_kv_memory_wall.py @@ -83,6 +83,22 @@ def plot(out_path): ax.text(kv, 198, name, fontsize=8, color="#333", rotation=90, ha="right", va="top") + # Empirical anchor: PD-sep 6P+2D was observed at 97% steady-state KV + # utilization in REPORT.md / analysis/pd_separation_analysis.md §3.3. + # The trace's mean input is ~33.6k tokens; the operating point on the + # 6P+2D curve sits at ~46% in the steady-state model. The empirical + # 97% lands on the curve at p90 context (~10 GB KV/req), consistent + # with the model: at steady state the tail dominates because long + # contexts hold KV through their decode lifetime. + emp_kv = kv_mb(101_000) + ax.scatter([emp_kv], [97], marker="*", s=220, color="#c91b1b", + edgecolor="black", linewidth=0.8, zorder=10, + label="empirical: 6P+2D @ 97% (REPORT §3.3)") + ax.annotate("empirical: 6P+2D @ 97%\n(REPORT §3.3, tail-dominated)", + xy=(emp_kv, 97), xytext=(emp_kv * 0.18, 130), + fontsize=8.5, color="#c91b1b", + arrowprops=dict(arrowstyle="-|>", color="#c91b1b", lw=0.8)) + ax.set_xscale("log") ax.set_xlim(50, 3e4) ax.set_ylim(0, 200) diff --git a/scripts/bench.sh b/scripts/bench.sh index 23f349e..57f8df4 100755 --- a/scripts/bench.sh +++ b/scripts/bench.sh @@ -25,7 +25,7 @@ TRACE="${TRACE:-$PROJECT_DIR/traces/w600_r0.0015_st30.jsonl}" # Defaults TAG="" -MODE="baseline" # baseline | elastic +MODE="baseline" # baseline | elastic | pdsep POLICY="linear" # linear | lmetric | unified POLICY_SET=false N_INSTANCES=8 @@ -39,6 +39,8 @@ MAX_BATCHED_TOKENS="" MAX_OFFLOAD_INFLIGHT="" CACHE_GATE_RATIO="" OFFLOAD_MODE="" +PD_RATIO="4:4" # P:D split when MODE=pdsep +EAGER=false # add --enforce-eager back (cuda-graph ablation) # Parse args while [[ $# -gt 0 ]]; do @@ -56,12 +58,18 @@ while [[ $# -gt 0 ]]; do --max-offload-inflight) MAX_OFFLOAD_INFLIGHT="$2"; shift 2 ;; --cache-gate-ratio) CACHE_GATE_RATIO="$2"; shift 2 ;; --offload-mode) OFFLOAD_MODE="$2"; shift 2 ;; + --pd-ratio) PD_RATIO="$2"; shift 2 ;; + --eager) EAGER=true; shift ;; *) echo "Unknown: $1"; exit 1 ;; esac done if [ -z "$TAG" ]; then - echo "Usage: bench.sh --tag NAME --mode {baseline|elastic} [--instances N] [--policy {linear|lmetric|unified}] [--requests N]" + echo "Usage: bench.sh --tag NAME --mode {baseline|elastic|pdsep}" + echo " [--policy {linear|lmetric|unified}] [--instances N]" + echo " [--pd-ratio P:D] (only with --mode pdsep, default 4:4)" + echo " [--eager] (re-enable --enforce-eager for the cuda-graph ablation)" + echo " [--requests N] [--trace PATH]" echo " Trace QPS is controlled by sample_trace.py --sample-ratio, not by bench.sh." exit 1 fi @@ -70,6 +78,15 @@ if [ "$MODE" = "elastic" ] && [ "$POLICY_SET" = "false" ]; then POLICY="unified" fi +if [ "$MODE" = "pdsep" ]; then + N_P_INST=${PD_RATIO%%:*} + N_D_INST=${PD_RATIO##*:} + if [ $((N_P_INST + N_D_INST)) -ne "$N_INSTANCES" ]; then + echo "[ERROR] --pd-ratio $PD_RATIO must sum to --instances $N_INSTANCES" + exit 1 + fi +fi + OUTDIR="$PROJECT_DIR/outputs/$TAG" if [ -d "$OUTDIR" ] && [ -f "$OUTDIR/metrics.jsonl" ]; then echo "[ERROR] Output directory $OUTDIR already exists with data. Use a different --tag." @@ -133,13 +150,23 @@ launch_instances() { if [ -n "$MAX_BATCHED_TOKENS" ]; then vllm_extra_args="--max-num-batched-tokens $MAX_BATCHED_TOKENS" fi + if [ "$EAGER" = "true" ]; then + vllm_extra_args="$vllm_extra_args --enforce-eager" + fi + + # elastic and pdsep both run Mooncake kv_both; difference is only the + # proxy routing. baseline runs plain vLLM (no Mooncake). + local use_mooncake=false + if [ "$MODE" = "elastic" ] || [ "$MODE" = "pdsep" ]; then + use_mooncake=true + fi for i in $(seq 0 $((N_INSTANCES - 1))); do local port=$((BASE_PORT + i)) local master=$((29500 + i)) local logfile="$OUTDIR/vllm_inst_${i}.log" - if [ "$MODE" = "elastic" ]; then + if [ "$use_mooncake" = "true" ]; then PYTHONHASHSEED=42 \ VLLM_MOONCAKE_BOOTSTRAP_PORT=$((8998 + i)) \ MASTER_PORT=$master \ @@ -186,8 +213,8 @@ launch_instances() { echo " inst_$i healthy" done - # Wait for bootstrap (elastic only) - if [ "$MODE" = "elastic" ]; then + # Wait for bootstrap (Mooncake modes only) + if [ "$MODE" = "elastic" ] || [ "$MODE" = "pdsep" ]; then echo "[launch] Waiting for Mooncake bootstrap servers..." for i in $(seq 0 $((N_INSTANCES - 1))); do local bp=$((8998 + i)) @@ -210,10 +237,6 @@ launch_instances() { launch_proxy() { echo "[proxy] Starting (mode=$MODE, policy=$POLICY)..." - local combined_args="" - for i in $(seq 0 $((N_INSTANCES - 1))); do - combined_args="$combined_args http://127.0.0.1:$((BASE_PORT + i))" - done local extra_args="--policy $POLICY" if [ -n "$OVERLOAD_FACTOR_ARG" ]; then @@ -228,20 +251,38 @@ launch_proxy() { if [ -n "$OFFLOAD_MODE" ]; then extra_args="$extra_args --offload-mode $OFFLOAD_MODE" fi - if [ "$MODE" = "elastic" ]; then - local bp_list="" - for i in $(seq 0 $((N_INSTANCES - 1))); do - bp_list="${bp_list:+$bp_list,}$((8998 + i))" + + local proxy_mode_args="" + if [ "$MODE" = "pdsep" ]; then + # First N_P_INST instances are prefill (with their bootstrap ports), + # remaining N_D_INST are decode. + for i in $(seq 0 $((N_P_INST - 1))); do + proxy_mode_args="$proxy_mode_args --prefill http://127.0.0.1:$((BASE_PORT + i)) $((8998 + i))" done - if [ "$NO_OFFLOAD" = "true" ]; then - extra_args="$extra_args --bootstrap-ports $bp_list" - else - extra_args="$extra_args --offload --heavy-threshold $HEAVY_THRESHOLD --bootstrap-ports $bp_list" + for i in $(seq $N_P_INST $((N_INSTANCES - 1))); do + proxy_mode_args="$proxy_mode_args --decode http://127.0.0.1:$((BASE_PORT + i))" + done + else + local combined_args="" + for i in $(seq 0 $((N_INSTANCES - 1))); do + combined_args="$combined_args http://127.0.0.1:$((BASE_PORT + i))" + done + proxy_mode_args="--combined $combined_args" + if [ "$MODE" = "elastic" ]; then + local bp_list="" + for i in $(seq 0 $((N_INSTANCES - 1))); do + bp_list="${bp_list:+$bp_list,}$((8998 + i))" + done + if [ "$NO_OFFLOAD" = "true" ]; then + extra_args="$extra_args --bootstrap-ports $bp_list" + else + extra_args="$extra_args --offload --heavy-threshold $HEAVY_THRESHOLD --bootstrap-ports $bp_list" + fi fi fi $PYTHON "$PROJECT_DIR/scripts/cache_aware_proxy.py" \ - --combined $combined_args \ + $proxy_mode_args \ --port $PROXY_PORT \ $extra_args \ > "$OUTDIR/proxy.log" 2>&1 &