From cd82b8c2a225c727a3e97c8b5e15059766a8b700 Mon Sep 17 00:00:00 2001 From: Gahow Wang Date: Mon, 25 May 2026 16:23:52 +0800 Subject: [PATCH] PD-sep matrix results: C2/C3/C4 figures + empirical mechanism refined Captures 5 runs from the experiment matrix (combined-ca x3 seeds, pdsep-4p4d seed1, pdsep-6p2d seed1) on traces/w600_r0.0015_st30.jsonl with cuda graphs enabled. The headline: combined-ca: TTFT p50 0.91s success 99.5% pdsep-4p4d: TTFT p50 62.8s success 52% (69x worse, half dropped) pdsep-6p2d: TTFT p50 51.1s success 68% (56x worse, third dropped) C2 (fig_c2): headline bars per config with error bars. C3 (fig_c3): per-instance KV utilization time-series. Both PD-sep splits hit the memory wall, but the side differs by P:D ratio -- 4P+4D pins the P-side, 6P+2D pins both sides (D-side back-pressures P-side). C4 (fig_c4): TTFT stacked breakdown. 99% of PD-sep TTFT is P-side prefill compute; D-side wait + first token is <=1.2s. The bottleneck is P-side prefill queueing, not D-side decode wait as the original analytical model assumed. system_analysis.md gains a Layer 5b that reconciles the analytical KV-wall model (which considered D-side only) with the empirical finding that the wall hits whichever side has fewer GPUs, and co-saturates both at extreme splits via D-side back-pressure. plot_pd_matrix.py ingests outputs/pd_matrix/* into all four figures. bench.sh gained AGENTIC_STEP_LOG_DIR hooks for future runs (set during this work but not used by the current matrix's data). Co-Authored-By: Claude Opus 4.7 --- analysis/pd_sep_paper_section/README.md | 8 +- .../figures/fig_c2_pdsep_vs_combined.pdf | Bin 0 -> 19377 bytes .../figures/fig_c3_kv_timeseries.pdf | Bin 0 -> 33501 bytes .../figures/fig_c4_ttft_stacked.pdf | Bin 0 -> 22937 bytes .../scripts/bench_pd_matrix.sh | 22 +- .../scripts/plot_pd_matrix.py | 490 ++++++++++++++++++ .../pd_sep_paper_section/system_analysis.md | 91 ++++ 7 files changed, 606 insertions(+), 5 deletions(-) create mode 100644 analysis/pd_sep_paper_section/figures/fig_c2_pdsep_vs_combined.pdf create mode 100644 analysis/pd_sep_paper_section/figures/fig_c3_kv_timeseries.pdf create mode 100644 analysis/pd_sep_paper_section/figures/fig_c4_ttft_stacked.pdf create mode 100644 analysis/pd_sep_paper_section/scripts/plot_pd_matrix.py diff --git a/analysis/pd_sep_paper_section/README.md b/analysis/pd_sep_paper_section/README.md index 27e21bd..e6418a8 100644 --- a/analysis/pd_sep_paper_section/README.md +++ b/analysis/pd_sep_paper_section/README.md @@ -30,10 +30,10 @@ analysis/pd_sep_paper_section/ |---|---|---| | C1a: agentic input distribution (p50=33.5k, p90=101k, p99=132k); I/O = 142x | `figures/fig_c1a_io_cdf.pdf` | **rendered** | | C1b: 79% intra-session reuse + 0.8% cross-session | `figures/fig_c1b_reuse.pdf` | **rendered** | -| C2: PD-sep vs Combined headline numbers | (not yet) | **needs re-run without --enforce-eager on `traces/w600_r0.0015_st30.jsonl`** | -| C3: decode KV cache memory wall (time-series) | (not yet) | needs step-level vLLM telemetry during PD-sep run | -| C4: TTFT stacked breakdown (prefill / KV pull / decode wait) | (not yet) | needs per-request breakdown.json from PD-sep run | -| C5: cuda-graph ablation (eager vs cudagraph × Combined vs PD-sep) | (not yet) | needs the 2×2 matrix | +| C2: PD-sep vs Combined headline (TTFT 69× worse, success 52%) | `figures/fig_c2_pdsep_vs_combined.pdf` | **rendered** (N=3 combined-ca, N=1 each PD-sep config) | +| C3: KV cache time-series — both PD-sep splits hit the wall | `figures/fig_c3_kv_timeseries.pdf` | **rendered** | +| C4: TTFT decomposition — 99% is P-side prefill compute | `figures/fig_c4_ttft_stacked.pdf` | **rendered** | +| C5: cuda-graph ablation (eager vs cudagraph × Combined vs PD-sep) | (not yet) | needs `--with-eager` re-run | | C6: prefill stays compute-bound at 95% reuse | `figures/fig_c6_roofline.pdf` | **rendered** | | C7: cache-aware routing is a larger lever than PD-sep | `figures/fig_c7_routing_lever.pdf` | **rendered** (legacy data, footer caveat) | | KV-WALL: per-D-instance KV demand vs PD layout (system mechanism) | `figures/fig_kv_memory_wall.pdf` | **rendered** (analytical, audit constants in script) | diff --git a/analysis/pd_sep_paper_section/figures/fig_c2_pdsep_vs_combined.pdf b/analysis/pd_sep_paper_section/figures/fig_c2_pdsep_vs_combined.pdf new file mode 100644 index 0000000000000000000000000000000000000000..1594e7717849b47ca86167db24df30c0b8ffeccd GIT binary patch literal 19377 zcmb_^2Rzl^8-GGJA*76pka_pzTG`n_K21YkRC02nJ-(1IfrxVqINlXEC}(Fpks~0VLe?zNc>Mqu)c$}t)r_w2=yz~(+#7K#evQP z%_=GbtYH0cAh4_GPmfae>}y(bpn?kD?TJ*=0Twkr{1ZnaMNufUxHt@kMu>x8D3mA)4uK(ow!jL2lt5KpAoNdO9e zfL~C2IX%k1HG@z7tx657t3A#E1pU>#vZFH~JP54p3}{3Vi*d8X;$`r{d19?yDE%@X zXS{Mfd6pq;WlmsL@5snU8P20K+*)kJ4Ly7(i7r*q9zL5x7j(KmUaYHSdq+BP{Dri% z2yHkwyJDDex~NvbaCot$lJu7K`o>n&i&hJnmDb&|OUV}I6W%$yRF-a+Uhjrh3_h5* ze7ESbkvRRnakUHLbZvtHTrsciC{lf0ofs}!0G{%mytO>;>+Th<3vSXfWe)KzFuLyH z>hZLZ@gdAUBe|3Las*Jty_yb zz+1ldQ2 zwho6>X0Fr6dq3GK_6#;zZjs7SO%!&~lz>7#$b+V);2Rh7a@@?vyyYN|u{xrcveaN7 z3>0NK9qy^oYQ6U4vep)bt$qQaMX%SncwjF>ju=tBEsY!mn@(dG-zR8d{C~DNlu5A| zGRR*|7cF%6{u3K*Lu=_(V`=X61gZL(bmFSILUZWI1ZjD*vG3qull__N98o1_t`2u5UA;2CefLb$ zX64T6_WHt?51otr+bi4Ov?rrhmyH(m)C)r1o6(8Vmpe&Z_WrtBxA1~-!I`{*QNAO* zzwB`(pA#J7mPXMf>2`m(iNwe2VqN3KVz;&Tie^OFeoJ;2KGo}+<(y#QsC=&%dAaeF z6I?W#N#6r2{V}OcNxjs+KyV?0lw8G;uX*btJfSi z;JI5L-(vMw`Ix?r3>XF+xz2hm;DVn)JlzzpP4HaXrmAjQTPe9#QN}sgV2IMTizD{6 zay?u7mh}zQ4>9o63L4{gH8qt^D*2Tk=W60F@`)_CsSB+d=D+71?h}065Z@@Lo|&9{ zltJ>W#*U|1hfIRetOu3wh7m8HI7)Pep8=gxM`OFt(PLnB;+3zevr{Hj^rm`qVV#61 zQ-V*vIPD2V@fh_x$63_K^Pu%Ff{pGCk}6B{*`&2fz)n^hF1k0c(Adm&bOhUuNM=Wg zM32oe`p4OrezJR)UVbyTu#dl=q3#H@f& zAr1v;3vufq0q|429s)6zqmN(Rt<1AZ+Q?TaRjZV9-HCdUHhqMtLH-3pe|nIbph;b> zzeKT+vUiu!#oj7!IB`h%s(h%$`%o^e=NHoiD>|`npch^2^m?5nTW@qP%6zX+%VXMc zw=vf>en(xA_iC|G@mh&J?^3Q<#@Bv~uylZyTlGu2^Z9g!t{2WwJn@)W+ws~V@d~3^ z3GmtB33xhn{B^5DQ~%=aOYBh>Y=2nrEar{qa>kkroM2LDdp>~SGvs!5>1OF0fvk5w zX{Gg0Kd}_~+fim}xfU!F zv)n9Ib|tBssX7hJQqyOX8WLl+9@I7rCi1@k>`o6TAEt!59`;( z^Sp2!GAZV>u!sH9wezF~355t!O4X-1{My=dXc;;=igQEiU&q%9)#bsfV>#goaRN&| zrv>Vr9i(B<%4u}ISG6L%C(fQ0RoJ+zK) znO#on4*_qCb+8;(pv;DT6Nl|2yZRizoKO8c#xG8oA(Vp_W|Y}Aql{lqk%|Stm#$`pbA2hh zInZ*B=IhNJtCz8HW8=4lPGnWuqz$(?1acdObRK~aujV!%TGbm=r?;Z5}3{Uk+0gWSPCEw+zq)MflX>#WyS(#~Hk-h(vc+L1VITn{_ zWAKz*sr2zsv|`%(O)kmo=<75PyNg1JN+)`63z=p|+-FboO3j`PEYs=v{174R9Qx8J z>B-~o_8ELlwm;7KzC7gaS}MxneBRa9kvvm5Z6_|yq%(ufpO3r>+||$)U)>S4dEpIA zd^xXcx2slyk{jw}Vs+wsFBBX=Rw9@x> z9t`T%vYPgG?sv-1weC0I2re_Ycq80}%(^gB$nbfj_?!^yhsJU@MOH^cJ;^Kmj)QdM z3d#-{-ptkKmQGV|e;JfM1zMD9XP zt;Xb|*{lbKA12tYcA4B$zhdca?{SM;%#h_KvPurKGMmM-3fnv)Su*^uDPq&(xM-Al zq+CegsfNiDI2&gRcxV$xpG>ew+MG=;Eo1G#6asYqJ=12)k#fVsxL z5CwYRUFCWq;EDT>hYgtO>d%7$$B{L{dOp1}x7fW^$FLkHm!dH}$=;CDjHYBAn>@H~ zyE*ph*AY4yMIC=<(SfS0Fg>AgGR>LZtj1AY$PKeq9(Kw2#-W_0pGVWE zXhFA*PCsS9~(Ry8eB5;rqL* z(yu=ae_V`uQFljV?&?I>R4(Oe_auMk?$cc16}LM$*p>N#HT#mAzEf*rC9n7&C_UvU zx-I=+(|vn(`_<0e&hljV;OO?%3b0Y#r|r3g3yjKjlh~$RVv^!8B!1EH3nc$qe(WJ? zJj#ZNBVfPRBzsUCPdpDY|BHrSi2MP^VQ?gX!T$q`BXw0>4?)SB=9qS5^DFsR(ux4J%g$Hq>rJXAg<(-FIT+URoM0;2zuq{&R@pxWD+2N9Hcf{!R2$p$?Q;nbNY z(+;P=KbzReq%lh}>u7d(_{<=#$wfGto$dbs8J3!-XplMXfw^l^Zlkg|srR5|^vfN= zzOy5f!gX5X&($V#?3Yx{d@QtG94(J~9}^>uqve!rCi})cD=aDz=lb=QKzLFfm$*nE zXJod)AZ-Ob-xa%4etnr*E4hf~uX4ld4=%}_b>*`atkY2z%q&H8MQ4AHao{Uj*$v8E zf6%gXNI}z}fq-)Vxw!kigh3(xTW8X-59t-S$Q4%%t9SzgWu<#id3sFd z=mU)03&#{Bjmyg*z+s(vgnXU*>BJI;JXzIgoGaFpD$Q8RDMNA~cVO;{)+=A><7EZU zOT|3LN2MkDpX*qdZiPH?sM+Qp1vf!0I75%2$q5KcV7(>|{cl+{A2-pofv8D_G6kd# zzuA@1P2?VLjiI5>uDXNczikyx>FQxjT_ed>w)tE?HDI2DAzsI8QqdUB`!XcY@_Rp| zJ=M6UwnEIP+_Qywf-P!4V=(-}@(;@uL;W{naHQBdh)!kAA%=pmS;|@EWJ-ZBGg+C4 zq<&Jyexm9Dje$U61$Eox$SST_tc`V82Mkpe*vD;rMD~QUChfo)TRC+(;t-OBm6QNa z|6FGOvJ5e?|Kp ztUVLgsp-*Rb$D6bGIpxrS_)4hfBLf$oAo&*x=GI7%;VGqn8L5F|4#JLk!YKNYH=LgrQ^mtlgf|&B4V$s0V-Wqz-FhN%O?Kv4{noHyx90SQHUsOp7fU(ck2hzpmfD3^oP$F{ z#L^f}SPA-i(yg!G7614uKGxaDdg%6vylTim+z;ua`d&j`5ph3wEIpm9htg(QSVT%# z%di?xm#mAOMW`2IX0Qw6 z`yXQmqpyD|nJ5+oLtYy`%9=&wenj!=5u+1_^qh{h5#R*B(e(!>aL56MjC8!ZD=`#M zzD%g-(YEj7nX7rH8&h9jeZ|lF;X2zmv(km+Mhz2vj#xxO$kh*@+^cSXk#rukvwLHj zlM?wpac+poW;y5b_<*vx34iB8h@C{;ye^uB45e9or$jh^Qq+%jlrQuASNb!}Ls=Zp zk)jl0IHO#w00L9Ib@kb$AJMQk7c|VB^-3;rN?9nLo1Z5;8_A7HVVRqjqxhiq$iXxs z$ybS3Zuv`1-zAFWSFbPh+#@SWeIhAfSGkDO6LnNm?BxK`g(xQ>WgHS0?#+1P4%mO4hA)7sEEHr&wufcogV zvyy^1&jhwfpB z1NbqeI89GdF7l9#%pCBi;u$09!8|<^jpbIF$IYjNOhadhhYvHJ{luWK6q&@x3A+rQ zhh46NNq&+TPjlCpMm+%UfH|YlGoA42Eu^BPKTEk-2^+Ts2y2I) zeN^^F^;l5KTMM7y{*ztUGqiBd<`lyxd0h$AJxVBPDSFr#?@d=7L1?O!3?B>&Vt~twGh)_&9ZT}|n;ltBIv`Y;i?cbf_axkB~>w+m%R_h*arP0)k zC;z5MVPA__X5P6v2q3gAIxQ{(dIAa~fSOR^|E(`LszwK*0}(SCLLw07ATXE?1Od}k zfzAM?N(T38unI<7B}Wp)T9co8c*3*Z%G9e(0(4KO3~k10=6J{>hrylzF9bjh41NGq zgVaXHlWUzys1?B3{W zbhdu2p<^uGc*DV+7)50r7Uw=vI_;WygJstzY0#K#1+{tI#Nz1KBaIOwW_!_Bq!`DH z4s@?;v^ir{jCa{xedDu`Z0ZnPZF?ff%OG#9_XccsUObLJGiv4QD`6k>71~8AR16o> z+@_#^FzQ;)(#4g-Cp_hL%_lcGl$z#wK_@AgM|5qLGv0?-KJdYD+&|6WC7&)Q%Ep#Qt`q;({3C~{FRqC>%-PJQ8V z=TI&Yv0IrcCr%g8(YR6zKQ%-jbe0;LSTW68mQ^0q8YTbP_1)sNN1ze~Jx(M7- z0VqfH5VB;y0FK$?sXz4eGP#Oow>$xwwqvY7kE0{n(Rrr@PJe$2uj zdr;^3t650g`=!)3l2bO;IttGPM&a%UreQ zIMFI`_OoB8V7CPo3m+Fh&EAwx4ke4#`jk!eC;+`P4Gw%bT1iv?RvIQ(d1_WbzROjY zHJw3@-g+*MsW4CHj>)4b-Q~p%80iJK=zJ;++Zsg;sl1Cy$KeIpLyAj@24%4~>`yVa zGELgsuvyJLy+>5eYM_{WX{kG+=t*I8vtoygw&wTK$r>VtU9S49mOm1spI^q23Kk*0 zBz_q>S3N{!9C?x9EBc4Z=G=%gSD1wB$d}0XobXN^w~GBOuXsK*&FJg z;DR`lM%*=O7O(WIYwzG*nLhSz_9V}deXl9^sfo5+xvHZBCUZMRz&}U0@Iv9zrE0t1f#|cv2>5=%+zj+cydZqn!OuediDo;Ch%ci^epyNat6TkY%Es9wmpB@w zK@lqXQRnz?upqL_$5YRGPdny!Pkwc-`1<3}6K8}f0f7hr)dNjDn!2jji2*M=$Mjv6 z%Oauqu0oo^-0o-}Z779I7916;Mqi&099B+;cOBYqZYL{8?9;H zqk1Qs_IIXFrD7&rx0LO$d?7S9ZRID92S$Jv58pd3oQOg(!c3kJf28T9F|`^kI;k5K zER;Axfy*)MC?0rIJv-tfrXcd=O<}<2?+%73R-sST*Ny!wLuFEn(tMWV<^A7kN^%cy zQ0BoTMTG8O>R50M`hKQg@6&eg#wL3O-Qyo-83Z&)0AQmJV5CYh59!-L$ER#*4>qD6E|Q`t=KJ`zjlFc}|TcY6oyDlotFzc%I@e*i@w;^+~$f?WA5B zvlLf&?9R*2(^YHA1>;Pn>&s8`ALr;k?(0GK9L>&TYP@&pIqmo-e>0WFmBhzfZ2h}B zLBiNcPfA|7^FzK%^iy=~<-tP1)|F18*FIul&v(CvGn=ghY}j;*1~QkVJM&))L%^`Ybd^LeG1>)S%I zx+bzedS3WEd=2eYac7=Aa^r!!RA-vIx_H-N+3OG99ryplJ2ne8DvwD2MB&|N3vPat z`EqtnK7rh~>=3PmZF^ZsZejAr4^qKj?Ug*0qn}5KOK`=TK0^n;dzQsvI&CIa_0EN- zs=;V#ne%N1*zD^fi~3SX2Z7!pfOVk<+3-?|?e-uA7HKQRhj}4^lPr&HL`t$y&TN^TMV) zH^&St!ObubKV2frb6MhEcS3C~d>d!Mxx}@?xGN|)=W0zPTD&a#O^@^Q^PA(k3+dT6 z1%;eAF1$1fF{f91B_nzlrZxG1*!1m#q!Q_uD`z--B)$+3mH)0QE*^w45>1J zz)-pn6ymTAW9Tsj(Jb=cJ4>>~p7mLMURJ;ZR{!h}>2cb zz^7Cqh#10{ft9)4k-c55mZHc8c~KS>>W)65YU;om-i+V~`@mF?%h65{vtm_RB&#BO z22ou98#UrXyswyh_iF<|0Qn&fuskQoPx4ViQ5FqhvVp@-liew~Gan`$In-cJ!%v`D z0&oe2Jiux}DNfy!1Vpa5kr~Vr_+v&YK@AIS*hSGiu|@f$6mEy8@f0u!&mFr63F!Uu z*tYhAFWOM@Bm+0Kuk@lRy@F#$;>LhiSn zzdlum(2{J+Gp5c&$u3d{bv@Q!n_*a*p7xmwvU0mfuQGnLCywhld$7-Kv)y+i-yZCG zgh>wGXDw%2dFPut;?r$u6VK#D6g44GIOim4I?5{XIsx82Q?M2Gq`@ks>6*sMw=*My z-*<>P1nj;OP$dDB0!6|98@nsTN-_|0Q7c^1%I4YRA(k1mg9)HE+%RL#dTqnTpleNx z!Ovd?N6rU_*YZx1u9T73V}5)iraTx0~!_DJJyMFH5A)a8gQqB_JyS`~!y_VE7<)n_M~})JsTG76|AH@8fBZ zN?hI5Taqt}REC>5&so6o%|yL>1x>8KkW1Oo|eB1ZCRH&9i=?q>}pEt-q2?gr5DQ3BT-k@tVVURs>?8mYB^pa#_o5M6wp=IMt z?+hmejO$A_eNHuyQFua4=Jjcnt_#$)U1k*?k2B3!^faN|bqzRt!a7uZsX9vW&BZoi zjZ-_efzE?6=a&{1lu$ll;va?rqXKU|K1C#_FJ2;3Z@qP)X?8B{G`~1{q#_5}LZdaS zuJ_ugp?fe(?)4^I%HH>e4sE&)`P$@RQ21nMpMIe2o5*6c;c&CfK7X%vON_2MV3){}^%8+CMqnRl z|K<|(03%phry+&}O5JZBdcwHP|6`FobCi0)?wZQCuiQ_WmGlIWY4HMJ??he`{nrwg zlRIcHhP`*SMezzMffFiuH-;!b2Sm;u+n9WjT&;jeaX??8f(sU(WFJ;XrMn9$qn*)8|HdB-B^q)hu~g=2B?w6n-v z&)0&QI=wDH?`<_vA|Y+_Y)u?wQBwD|uPXFW#HTo@zsvrTcs{ug zyTOBY>ur3m5U!Qv_{DYTaYfZmF7;ubT^)m?WxQ2t17^=$s}QQ-@pkc;6sP+)=87(b znB@9>njRY#2>KumduUhv01YXTenK0oRo_*yFf5sLia4?{c2z!~a*^XXZ_Z7jSE+ip zwM5<@Pq{OH>GF+3(w(nkCZ2ZpvqUT37b$#tAxWGL!l?f=$)hALD3S4qg8f+|1XX@d zpT9ivQ4Bge?LoV{6>)!cNM3%KUNaq*X#TMIONnWOTJViSkETvXuYW&M{KG!NsyiXOtb3{Ol17oJLl9FiAWhYpMAT!?7HF;KxEYk*?UL) zZ*&5j`P?_Cp~wTkbmYx`8bCaSmGodLlA*zSQrg$E!g0G!ymv{dQz5K-z0r#!AjNnq063=-j3~=G;u*FvObmy z%e}kAr1c%_l~9<(ilyx-)t}Vr=?i8`WqIy7HCSKfna#*iE7YDN*%{$^i`8*+dnHfI z+;YMX@`6D35kOxD5%1R?B?3YguN)?u92Y5nor^`~P{+XgV2as56X4ZhjyPF4U%1dU zA(}&PYiLsUHw_rcKGnhy2bc{sb(_k77a@z%q0HjIdytlkqJDmz*j-d!21${k!ZFp} zcKt@%KI;HAWtAJEbS@{Xc&948XPriPwW*eigIejne-655Mmgk&T?q`EVG)*2t{c=YPyW z=0lZocH@xKys%ol;zQO=Ba50g4KF&YaBZtwT_#< z=uwPukNFY#;fTQx?FF;MPny>%;-TM<8#GH^i&ovVP?|7>e%w)nRIz@ga8u$V4T-EO z8gCi23LDs<92M_grO^l$TO=ST0W1fHA7F7JqNncJ38H@GMQNS|-z1xn`haWnd$3y@ zpP-~nYqbV_9~P;0*#uOVJJL}XGawuKwxVPpbWr70k3>X&Y*c-`Na!U+;TdR$R7dbi z$!89ZZ>0mh)nr}ZabG&2ug^oUT+^MbFRhunt(j(wE4{gjFZt=`wHKWln~vI*>GTpI z5xJ~96lghfr{cz&Z}H5I8>ueC<^Gi#lErzIY|h$V7JQbb_6-sogJiv5%g5;TnhqZe zZGb=VdV6}es%-QY6wz1k!sH>Usb)MKeMch(ZdA$RCl<#h4aCiz5TQ8)S z<2+BE^SGNifV>q|2UoRQY*DF7dvwuUyn2UP-|qfJ{z1>2=SAczRrcybL+J9%2gxQ5 zxLb_Pc_D*cTH0DIM({;0G!sRUjj7Q?$75+W6t5U8^>y{r{aJUmzAS&1ys|?i>~-0T zK(`Qpo#F@Bb%Uc&Mt{88Kq8FruQo{ZKz6`V2|_jit3@2SCwq^aSrJ7%fI`567Ko}q z{u

ltHJ);@Lz~y^l^FH7PWVg*qORVrBCsz#Rc31_w@0{HHHR0(&c@z~0KBK)r1K zC+#Pz#l=xAz}7S4111fK)wBTEg$~VW6&pXg@Oro;;aI;oM%kR zV=tR#9f!Q`hv?g6zpW&FK5K&wUVcA~taudhQ8&T7-F-Rvu`l^khnb3hZ z@U|->Mmp3Ws3AC8TGWa7HWMcoYwhQ{qU0M&CNc*8dWLkpZ^BEv`ne{ugg*tV3Wjm% zM5UgH6T!OH1-?Sj+`E^gpMRq=Di?C|YmGA#hh@%rZtwUr*E$5j6Vr|CJ^3R|uXCbW zuI3RzDF{eI07SuojfevnEZP+SpNrd+0iZzFS1aSdD(*)sP0{!=K53@Hbw=t&VhGqe86Y z{Fr+|&49qh#CMUEy+nuQr0E*^)hQ&h{fif-!eMN$^J30i$LcPfPk=r zU?dS;RZn6$@}@zdwGg|zQ*4v;P{$m`5tbK*C5RwOuLQk7$93`OddcR{Oe9?DL~b<`vRu za=oQy9lVSz8ZWGQ>}k)W!9;7kq{TpU%El?yv95+E;0!kqp(<}*rwa6 zL6~<>#dmw_Wjdg8lQ|;BDL&of0}Nh|V9>+0%Dv-KIc;>^X5_J#*HJ;Qy`5DrUAmhDS|e z{&iju>tIW1_uie=BwfJR_z)|f-K-8*Y8aa+M&B!oNLHzWcuD&${kzt2GA_!o)=pJ-%+zFph2A^a_Y(9s3WE6M*3&FN;~$6& zmEP&94(w_^C{ofNEcjTWNx~1mvF^=Qm_F=3WO~{qf%&z2z@ZHGj|3Pb0BoQKc`YM- z^C7UDC0AJaM!74SHztsMS^MsV@a5eK9`zt- zmU~>J#K`P2$e)L71vaE!r+l33(a-NnJ|1+0$WWYZbbjA1uqyEO%cyv49~mhOD_;NOKwB74jp3j=s#?GM=h~>ro*jjt@?=3_Z^};XAP3~ z>jyX+`u8~fdE0gG3M*LN+6((Drm3r|XegwJyg?G-LUN{8@YfqpsSi|~P z9EL~ZQVS(GR`0KFeL2zIf*~{G;1KiXF% z7~rz6A_%Mmhz}A2TF?T4wSgZ-9Y|!2JQ> zUM|?k(H7?bXcIq6{PXhjUI+g^SopUA{lf(I`s;u6Hz19(wY?Xh@dNd|oEHXvgBcA2 z^n`B#|3d_ZKlGqr?XL1ulMqU<@$V>bKN`5X?TC?cwRgq>jDz)YSQkTlMAO=DPa;4d z1ni%er+?3a=Rqh4}k z65tb`j?WJVY-aBX7sKcOzXi(Q>2M$~KH%&49gx5QIVc3L0KmRb!1@-r2o4eh4oahd zv4I2{7sCf630O$DPcr*CjUB90Ix4^#pKWadT0hxfyKzR1?cJMD&@x(o20&2pS`l}W^QxFK{ zUfQn;&?q$McfcFMPaDA7!N0?vt^vEi+r&S@o;~1g;&<4ym0vc2w}*cQJYRdifjaQE z@jKupM?sK49|FAMZGb>PivxXvw+q0D{WJWwiQiAWQh2lD20+zo%Ap%yA12Taz)*vXpVz3@>iOu zo2R`1B^bbfw%!=vLnHqz=5A{Tva!ZE0nPlb0?4}8fTNqMBK{LCrxhh&z~LSUFbhLb zFc=gi1QF+lK==jzp&360HehmMsKHvvlp?}ce_)+(78XN|AuRmxI7yugopuu5y z%l=ncC{lc1Stt?(IOsp~LGh^l-(^u~{6zCF8Uzl9?BfM+ZomloBcC{aM))@kaHsq7 zA@Hv-|I7ymdgo6XAjm$NIDU5hqYb>H{DX#s1IOk6q@jR2SAWvbfW!TR2Kc@GG=S-S z`Na13uNZ8PhP%%8v{;OS_r?sOq)^qRETKbOu wSYYl1f%V+n@YDXDt5tQi1Nd%wld}a0SG8`kpKVy literal 0 HcmV?d00001 diff --git a/analysis/pd_sep_paper_section/figures/fig_c3_kv_timeseries.pdf b/analysis/pd_sep_paper_section/figures/fig_c3_kv_timeseries.pdf new file mode 100644 index 0000000000000000000000000000000000000000..a1af6e2ea274c5fa3cad88f9ab2941f6a3328349 GIT binary patch literal 33501 zcmZU)Wl&vBv^I#lgy0S#xZ6Pj2S2#GySux)2X_eW?yd)Shu{Qv2`-a&?wy&hYHC+? z@9t&KTEBMpvo@uID2S1ji35qUYJ;q@1BsQ4h0NZ_3W<-8j9JCg!IX?y#L&gi#@?Ka zS;5fU)R~O!pMnw@5Qt=IXY!vS_y1u4vbS>~WBa&EeuU8?aavl|0{K}H&!-vA=CaBRuuFPim8VS8MCDA zKM2DAJB$49EJ>#Of52h=|0nPt?*Gs`nf^ofKk%8AOr7mros9qG^B?|y@nua-EDeS2 zJ^n>x`S)=1@Gt?m$=Co~OsoJdF3x{3m_`0&_s@kh8PES<6tc6k|F>fOPyhd{$baJh zr-U-5cIGY?WUT)i7G!DjFKIGnkj=kJh?*MPo0$HG-Py&-)X)~mBYV?VHpz6vakIxC z{F^37=eoW}pbJx$5Mw?pQNokl3KBj1y9o+{6B%oj{*`mX>DG9a-$HF?q@{ux9vhWo z=~i`zYKG_S{-V79$A>_V*W(9i@6*~r@8`mx{7T-Iug`Uw|DZr`XWK;YZk~Yt+X3?1 zT_=3FfUk!|x4!gXub=OmHO=2s>1zCM<{$U3;o;lBw|o1WG=V*4pPQfK%}3HzE7{*j zp02<6_J%cl*C+kDO+N6MZ}G;+nBe&7u@;8RsZ-edv{BogI+3@f|FLqAH6i~AxIP*@ zxMD$+IbLU>bZ6Xr*3+>?yfQtC? zgIkKn{fS;Zzh`&!&;2y?PqM4O?+mkpxVW8y{QALzUpe>-DSb|$Ht{dVV6C3Dq;6PW zXKy!QZ!fD;N5%bJ8Iz5uii?3;JPK_N%YFe zR)Bo1a?J`>S7keR>#9cCr_mLONk_X1?8?tb>3hPvNql|cdU`|PHBtVQdqrPz?{H^s-5rzA|h=Hrq# zg-g5ic#~80DCn;i+OD+v$+Uxf4My@*vv~L&j+BM_O4&N+eE7;=>xpM5@k#(`F2SG9 zT3}bN&)aF1`xSHV`>8x>Z)ex%p}W5C-N)@;Qa|_SzwyA2ml6AW{M=n`p!qW?YSJF3 zDf{@oY5Zz`jX}n>J!gvACxK2KtBs}lku!sD_2}1b6Ulo)W7$dW_!;Z&u+eJedCsR) zFDJwW&+$T!h0v1=Dc$*Xt#nGSv4P{j;em8VU^ff@n6?+m>UEK&&q8QmR?KJZZbCDg z)I(x1t4M5fJ%z8A<25= zM*6V(V{(I~eHSn4Aa}M~aX`wsSY=^&W~DL6tBV(}#EVXU8zM^gbg3WtZbsbO-)Dzg zU^fD&YkdS|SZvJDpq58+6s`Yudi(h4rtCFUT%(|cbw*R4PH-dx22c4%Zi~FW+nJ;F?P4?^D2;n1<;Fqg) zZg^8inm$-znh?y~06XAZyK(zxcK62>)H zX8G-a7+^TwsuKmui5>;i*M(=rX3??)#JBHI7&T`id&UIDN90qm;+rjY z5czEL0Ye^*Qi*z1eumeV64)>SMZ=m}FHL~5F^>J=eaxv}mD3pKpn*7QO>4L(C^)e3 zs*>C$7{EG|`TQ{@dpGm#fstgKDzjmm7%VhC9B9(ME)c@1J0(vINsrqgiLK%v?5KfO ziwDG(4^D9n`I~MjiK#KNr#=m3hTIW*<2?5@YLDaNi>%z834vAJG?*HZR1+7JbGwTgAEHnKe`fi3qvm_=~vwTdNd85?7jABe`N*QGNM9^ z!~_OoJJ)y=jg~U&EhFf+lt#2G#3UV>c*FRt5m7FeK0^APN*Uu{ob3*(Ref>8i-T6t zxB6;$*gjzQdv97p_jpZVA64OIrF3oS2OSu#8x>V^;jw)6FQQhau7DSvTQ_2I3y$o0 zd6#HzJ}&`Q<1-B?T147luU&%?O`Tvtng_dZxO+ry7g@A&1asORRv}I;UinNAvB~N$ zw;JFpiZ*8)=AJPruW9+sd49Wd?PkSI6uyCWb5rF_z(v@nQFGhYl2jz_rM&@dFHx}Jutv3PACp*YLK3 zfGe}+w(y0tYuO2rE$Ro-HPPA?TohAKYuC z?M>~JSrauy=O=Q7F)}~^Q(dBGf~K~Eft+Ja0&Uy~tzodBq`JF$K&F&#LxxZ}pDHv* zt*wdX2M_?Huo3ZTKCum&&@J>YL@I$=1Tti$dcXi0`XmSoZOl zuEgR=9rrOf51PU0XeJHV(0J18Y9nvKMdkUX&>%?>i@`3`Fu2^VskTQ9DDhk zC0ebVdih+Pk@+&we@c?qP-3HJ6$rK8!>DZF(|0gyjq899Wx$ucFbfDLNgC2C=w}V< z^SGS&Qnh9&@=MQdBV{Wq)#MSX?8Ef~e5AjQzY60n?8U;*t)}J8^B`>`^oZ>^wB-rPK z)RE3o9qp@`egu%$sYfuZ9CluN%k>E$;%B|uY!bOPw!O>q9cLF5!YA0}`LU#>T$_dS zhB0|7ti$4|B5SKeudf~*WN$nuDDr4g$fK2&H%H{Fhs)Oo9KwHuj%Ic-AOZFVrZRc| zN*T69YeMd<3M3K4haTb0q6w4252g!TWg%MaoFk6)OI;t48?ooc)vtbN7OA}N@qAEu zT|IuvG}V|ae)u~meP&H9Tkh+#fp;yIsrrh_M9||brFNnH?pNXa9xZ)Fcv60k{{mhn zfH1Orj~fh@Kk86@t-E8QPJ2Ckgg+g|`DgTm-6aVmo>c#jy#q5ZvY zIlxVKHplH1`rHyO?C76@r zkBmDUo9cxPYN(s?B)h4nQZ!rNF&0fZZNQ-v8-w>zfMo#*NE@_+fopDpG5M@%o3mzq z8K@<6H>}!qYS3u5(tZqiYy?y^E}1C4F3I2>TB|DYdN}C4sCfDjt>4IRNQ+P{0i1YL zf}dX}>yrg~bSiy^2=x;!pR5(CXW}!C-&+?c(^8(q<7#dz*nN<>nX~~a#Kpxd&1gDu z%>xzEx>om|E0r}KfFsCS{UgTWlkIe|bNf5z`ancnXac6&V1 z%%|S;*v^B_u8#|rGA3jC=CW8b zu#LIa_77OH()8zWjiXXYQR|1)TJT;ko+k7&a)Ibs%L^4oq6dO0wcfma++0e8NhGYl3-Y8*neJPI22>K+hlGs26Vs zY566~oyrfO`H;LS@|_0>4+1$Xz(-+mOWPoC3JHbhCDZaEUS`8`I^r(=}Tyb zDS2@y=M9UN+tY+jJ0}AE+L|>jas?04c4B{yYV3{50~(lvg%a%cmN;#8e6LAA2D1&CF8G^DjL*btDDC z{U)pjZk+CnIM1Bzto!dN7xH*sar{8(N*^CWgC3Xp4dDg=+BmKjZ6K@Aa))U+)xOGQ zRzH1Se+-g--1{^8f83tcGWUKcx@*_>VmgJ)%@|~cYx3q6*OS$OAxgV}OB&ag=%gQy z>f2qPQ}SEB&*f^}-zoj(U2t@=rLRvgdVN0+KA%U|g|kC?=Zrt@?eA7Kl9nO2eSY%t zNkm&6MpM-S(Xf;?zuvML2n+edD-&q)sv{U^*h9K_Aq-{TijkJU)PesR{~|Zk>6;5F zOE^h-WNmo+cs_DJ<&-CH)+rR9$^#ZJ`aF9$e1s$tGv8hsBF;R%Ya!2Qmh1DB`WY{6 zC8X||qq`HgPp}iW!&{n{K&IyPv-TnEo&&dv78?)3u)+_{N1i*^lJ1VQJJXOnrx9>$ z-u}vy{4_q&>v;m#o&)a8vZnG9V{yvgV@qhxX<0yaQumg5{udbU@Xf2hKX*8o0^#;261o&@vY!g9iJsu5mYg<1*e-=|hbb3nr$CaQ?afANv80&d2Zj{WV@_ z^rOG4Rh-Y^!QUr`Qt!d^uJ(OD_zWSQ9dgMZfcm3IVf7Z&7A$ zH7`dB9lyQVn+u)JY#GsI#v4x`6c3SbfifG{%J{bocr(R}<&KJ7i2m&?tRe9H+X;K# z3t(WToNMD;&YqJiyunePcflS>)*JV6w5~FiB76jE%SbxQ5Gi}n>Vm85<7jO|u$eRF zD+_%_UXNL}SzA{^pY3S7#Uls@FX;XbGS*E8?A#^a6p}tKdN6CMVS3)Ih~6q4PVv?| zUuW#%q%ExSUVye_$ZxS~nVJ&2AYA)G2?mz$#8@d-8Jrcf39#G^qUCGLz@=dV;#5U4=|`ZzGz;<% z80Gc}6A2uEq2ElCU`yM~(e0xP`>v7)2{gaNOAavvV+90yj7m!|iApv-Y=N+04rEw7 zXhozLK>{S^oTkU!jlb%-szlTGzFO_M!+S7l&?D9>qoHSX2E6O=#3k#POwm}PUf*b& z!Z%+jZ%CUp#N+3E8C|8kbvcknQjASa^8qrTMzGR?* zkBBc6#c#vr5vr1Sy+`?`b<^kPy`obr2^en4>LOCcP;7o$d_}<^Tnp9GPU8@()c9aK zz(fBGJ?IcrrJyB#b{)gOvi$oEpp`vOV~SAHl4jH}9%{F){cSPCC04yh@~&t;!)Q`~ zyIjw}U93tu=3UpTNx#tyWgfy^#t<7^FJ+|6HKweid$7o&Lo<0Em9A;R3?BI#WzYYD zL}NAx{X>pB7{@8w$V@=o9qKO085Vy|zJ_%>uNG17Y_+8-NQ!Z#Zp=G?{5E3@qNJzO zy(K3iBej& zw}=!Cp+pg^($%^;P`1WC1|PBGwnu?;N~>g?;Ze*20uRUB;scgQrD3Pk82+*g7M*Y zXv%^#Yb%mmkA22(v)QrligJ;84!B(BS^_qVfTXn4E745S$#7FmQ0}VNmp{@cX)?K{ z$j$L-BYKWXu@avBH{TCnu<)|%t@3V3Sd$cm5|iY8`Os0-h<^~KTcMDI>)?bp5yt6D zyL?TPM1wx4!ZcXvOkgJ`gra9^8?8B1#7Hhu!d^DwDAY`?Ri(gKHsy=vn7tYi4Z{S z`{W}(_PK3u(Kg`@vOT6V9BLK4Drv%XlY$FylCZ^RzoC3@h{N{Yj4jQeK#Sy{2z`%$ z!15h84C(Z#bOL|idu2KG8ROIuc$z)*dgJ&bL~CCc;)dVigy=`EGs-?vC0XSwl%6|f z|JDFm(nbmwsH~e1$-ik9rL3F$$)OGFu!@<%^9&GxB-Ra+Iy2=!{v)VY{uLZXS1jl` zT@#66N~%b5K$bkH1Y#`!O*`!kpnaQ!l$JTopq*I-7#}!F!3AT3RlexFfDYq`Zl5%7 zw|0?JWvAn~%e>9rEID-uC()80|IkTNXgu$k6F`4WvRSAchYT@Vpq!QxZk^{FcvC}HAhl&Zs&{%>PpVn4TLc$gJyGqR7W6;Jmi zI*ebXSmXQMqOhVe>|VOyvx$BQ5h$LBcqZ{PfF?MLh(4UcN-@kzOM!+0P-5|Zy^l4n zfNhRsdcFs3x)UvOaPP%RGx+3)WM*+#1|z`{q>L;e#~SoFsyjT9%YdW z3Bn(Sz%b`tt!LSoM60c~f7(wVn)Y?kYocJ;}$g(y#itUr+gihe!?@FRHJ;rTA?`G%~vI8BmwwL(#_H- zBeCm6SX5}ciumAZdcL0__yPhLuu$+t+MYF03A`f*2@4>|)d$mevZY-jn_p?KYX|Dnw+xL`O(~RSEpnYM?U=II zU?P8~ll4S6Yd|o*U(c_S;8;m6tN-F_nDstO$tUW`ZCls*;z&p{%!KpSIfjUi*KW7R zF|g!nIm%kn#{1N!vQ3GSmeZ-^gLNkIo%XKUuj5vsCU}+IQQOUtoM49x&_sHP=Y~aO z0)^!hp13zVnW2rOTsBA^yliC|m5+9$CczUA{1QcooVZT)$agg*nG9D06&FV|H95|e zZR$13%OTueC4@Dwusld^3o+mXQP1%dZP%u2(=rlAoJ-epP-YmyJJiu|zhu|Mhjf@I zC;}L6AaNV_rh(~oDcq6#uIS={>A{*S(*+Vz%*7wt!csR=rEalbe?zs}S6KrN4-y?? zjnIw&jTiep1~{Zu*7nc2@J4nWShABL;XNPlWvvUz8J&L{7ki-Of3Wejux!Eo1_}~} z%Q8K4!0rFJP!7yzRF2#JdSNvb>)kPI77UYYT*-baYdiLG$N~OCa6J}Q%tmWD=7G`j zKBAnW!4dXO9eP7?F78>yaI$SSp$zN#bGT~sm5c>=r%^lz*M^j`T!IJR7GYjZ|{5aa=Wyk89wL$5ZyGViN^hdy->K zB+32UH^ko5uiz@-Y>@J!G-TqBh?CPaEa&C+UKK|W$lASQ%e;9cQ(Z(rlOaWnWeE+h5LwEaaGXEBuu)$O4nB6xNbv$cis)@a7z_8^g znBCP^`8+F;PpqJBz9{h(KXp+M{xWXAmNJBU2ZuOC9`FWlD#zebDXkd329~_wdlzF;jeZhFXoL>`RBL{cL&;k=UfY;$p_%6^G zPg8$2LU7^p>X&$R${7NU1HCuCn~Dx4VRi)T1i4bt!mE8r3c@_z_;sJ(w(Dsniy2_7 z0t2rMg_badEZIYU5Xt?`&dKT8Kq8W~A_Dq=MLY6PlW=1Q0^x%RwVo`K#Q~)-Tkz23 zJocoJ?`~3)cEjcPf(laicH>V!x40*Iy2^&I%xaXveYG8l0@nI!`fsWu^?gPNk=7NK z2LQMoIU!yb8vAw8=%8nA;c=dv{f`b+8Y7E5pSa0#%xyW;M!|V0QCrRYxHiu;pXPHV z>PswB+|OH_`-zaEx+HA|P`GGpwrnhSo$0w(_AASP6Z?eNJlT@4gk#mboAs?OFUOc4 zMw|b~PttnH3I=&nr9bm2DV41c8Mos<@Hmy@O2F)> zCpQt!R`eqZ=E3kS%ixz#{1s8ludP1>kn^$B0VqsNl; zvHrO0Ak7QLC%5=+V>AHO(lAz0rci%4D?jQjk>+$|crv zVd_M=McrBXy=@aKrR@63X1k!s*>Ws7rMzH`@CfvSgAm7>iE zOV7YVEw=7xEb~j~2`?Geu1G71aJG|@e8VJLzhdr2$(nQmh^Tr1xdupC6~!vg3>_Uy zaXVK|czTLPsgE2=xYl)?uv4+yv?#!R1g&3~zUIpG7ax2(dmw80lL-Y(>X@l)$BhNA zJMAMTux`+#O3j>&=muS|H1NOiS-ZncfAe1@!hJs zo)Oue+wZ(RMR(uT;wZTBV8=N1SW*&r4cL;;`@uV|1T@19Ve_G+tO3+X)KA_{^l`hhoWSeRoZ&B=a~Vj)pUv zvt9;FMEALp6H)`?S~}aP&h(Pk%}rw-ML>o4%Aqc zHxOguWU-C9io00pzgeGk-TSUvjrxBmRRgf zde#)0QO(OxV3RYc->1OI!k(7Y2F;UTED=XjEmN#FqX14Kne#*AdrWm}F92+1^Zfjh z0$pl-RK5&@aSE5?jLP{#J*m2HQXS`x{<CvtXCNFizk6%IGg(^Jh_AN_&D6nm}Y zY=)ukPEos!Yv6)joeLKC+L3x-5|z>%tely3B@{K|O0mpK@UapViNzZ;38#dD{^FmQ zEtgaZ-Ihekl&uv?>5c#?RR&d@@sI{w2K*b%fT2V6Cw=H+7}}kv7zN>(mh|kQ92dw$ zxfe=H610@LBM>b0pD!YD)WW}qkQ=jj$#iRBdcV*Mumn`7H#jwTAnAk3CFv`P$$)C2(ti!qht!ZdG1Rw;;YFsRg7P|pt5PAgE`Az=JX0GP z+v!5t#`txSiXSVYxHNd?;F0(LOh7S&YW@)8ZTmT>7gq|6Kwtf9AD$|=WDlO#T8KxE z5z)qGGbf53k4K){1+zNHX$|XoDK~MT`{5+iduoB>EdgKufc4NcAF!e-;n^Uxv=pE33i!?}l`So#r$iV{ls0n!CLysbdb{pNr<-jegq zScMC};B{)2R(b!g>C_K}YP7LvGT%xk;8q=Xq%*((q&jQ(w0`6rbCOus{TawrcNk#w z3d*Vq*cvA!_epb`;@3?h4yLEaR=&gwK%-Hpo@tiiuFmRYx&DR->L)}0ZYw!NzHm5i z%0xk)7{u*`J+SBVf%@e4eU`k-e(`KE;nnj7=`1G4k^kNLR_BNyBuhRb=vTZr6>=)0 zhrHlXP10ge$KDAq^^})G?Ifa~K#DDJNIs3Hyyq({3*aO{fSsq!TTIR@1~YorJQ%e? zJ2uIzR$W0fzFsgSfet|kr%FxU!tq}5`s!Jzyyj;p{!o)*-&0e)7n1tWcf@C(WNKnl z=|C>k@QiOds?xmGH81os4OY$SCbZ*3aeU6m#jX=SOXmYXUVDUY9cKEs32@mgAmx%X zpS+KslMREO6w-}G0V=DY&poLJ+xfz5Der)iY^i7Ll5DFaQUfsVW#C@i_Z>|+FfdAJ zvdSs=&~$q7R{~RCAw4(tPQK+q^&HM+StY6~4#o>&uHtH91TUejmcXyM_~H1qUBkd5 zs2_(p@@2+F$a4nDYFqH`qv)c>2j=wQ_^|rAPS#mpYKLB6iTY8pIDXPlSO#`~J=H81 zOshep551x%A}FyAGXl{Ke%Qcrv~(Tbug{BO)*$z9m8n+_Fs@qHbpsQr2CNaXw6s;P zhoBWBMt1Tk-960W4?o~pL0I6O9l$Mr5KXs!atWa^~y%Rl$P&pTnSq!;qY6j z#~b0%Up4rqv%c)@0brB^ul>(*cO+r^<_nBKVA z$w@cP_7F?WvS5Cr0d=7+R~hf5EqfY6-dWdVD_&NrCScugv4xxt+JJ9ItK-`_K*k@m zIyFe&XIF4<9^mWU)`Zi-UsP~O1@af&oN5fMO4;I199SrYoHgCdSc06DRmfIQ794ee z7o9BV6xKz@4w&J1$tb!D%5B|@_>=nAhN)f&XDB6C=KV{t$uAUJ|CRyWhl8GbX0&O4 zP~LCu=E}xCsnU^dp1E3wHZEMRm)}|UlJ45u__Y+D4#1t7X9{wbOs%|{+v8j-OdQBTMg?f=U@jlGTqNB6QXNwYe&U1Qz%!~GlFEeKORD|E?hyP|f#9>w zr)!a=V#JCC4`-25=L*S$lwQOm*u#!R5xyT$gpCEJZgvdJF1P{*VOF$fW;4u12SZ&s6MCzk*qdVnp3#lCfj``v`%TQ_*=wds+xFWY z{yA6NF@AuG`_IbjLeB>jAc4R=1CNpc2%G2vt!7d8BA1~&XP-%Cp3ER;p|M>VH>+Z6 zdb5|fU!H=$Feu9f4wg&bF=P&q<^cC^_!8VCJ^+_w^gODHIznZuFqviEh{OgsITv7= zZ_Mj!`(e&F>@Vi*J`z$cTPc%%=%faaSwN$uC}9>-Q=5F-u^h7djYUPr^4_wBrEJ}0 zgMGS)y@OT}w@~zxqi{Orn2jdOB@^?`Em~r*k)%Ji<9tET1M-%0ky z_W9q3s?+BH3jS)whxntces&b2g0JE|k8U<5%GWjOg{x3`)^g! zH(4aJ#ZUj}nfcd9pO;lBZ6cU?EkNiqEw)PWUhSeTFY@#UeML&zQ*)(xuuysG{9#`v zljuc1Tgcp5OkK}Va5pA1Ks(2e^7+#JKZODNW`T0d?XpzGLofrQy>3lX>5dPH=K^> zAnIo~_tth9;#zc)Ni(@gX_cy&t{(yp`D=1ALul=YU^14PD*Ig-hzfg_NYn50?e52W zZmCNuC)A;!e~uE4VCwUqX9_Kku8B@~O4XusEiXCQ_j#UAZEN?eRNvG-T|EV;5e4}y zdj>JT;Fz3*OmU+wXQ|eqnG$1j&Vy_zyQXXT+xpMF+cQ$%zf&VuETPs-?Na=X8TZyH z1)056h3a=JU3XR=wFUp){NYI7XX2kr&zdjqJ~=3T4OdzeZ)EXk?gMkaPDycnuFFcj z_>NvhFMg3j1^KvGo+S1pJ3Ra&bG}Ze&aRQYT1makw8+@V5j{R1YyNA9O#81Pk{R1MXBvq?e4{=PNmAO_kbdO z>a3EaeutbpypL0{m4WcID|~CiHnrxXeOn$K@(#qM%g#Y?YL7?pzn;l^DOUX5B9Ztd zwc7T*9p`G5LVDHnZbQqFaJ$fqU6=nJLvesk{Xzn;wh@h_POc@)=v{_0J4DBm!L`*m?? zW)Sqb#&E`-J#L<{D87%CilzDOAg>#&!V&(KJt4hzY#+m6?i?KZ!7eZEJUVmIl^pvq zEaA+{wR$XY=YkUa7O18-4)V9I;wjX&xzF>K$`txYpa0J;)>Y;Nj_~~`H__~;K*_xl z;6>Y*Sh=Gg^tnOXof-S#K{^HUpS*Cjx#vpWGOml=bxV2K#gU`7e*gdIpyl4SBct_LoH2NAur7jMD9!$Hj&8|!_>V6f18QXEL}yhmo$7PvX+gy5M}fU)`PiD ziKBEcBq_f3TU9D{>tv197`DAwLgXj$)?b~O$@LuTQ#qWLqQ+*>Cq-T`C9gCmPTc=H za)_fneuJJAR8m;#grQ{6iqhV#1}@&qEDovA36r_o zn}lALXq7nS3BHX{*3&lRN=nO70-3rUbeNJCuvA5)oZ)0XU29=;>3QbW#U`3I+tfJL z`$V?u1n53_)^+V(bAAEj=tPvEZQAZ%F5i>7_6zTK_!t1`2h#!hVg z&mXoqtq`%wV#P`dUni8JL|frM-wC6t?fE>CTQg73^R%4~X}pH~f9heU=1Z_D@4FjB z85WA#_;8KL@lD9CR(x@N=o#~uATDXMYerM1l`0$HtK4S+SGA(o0qVu&Mr;_EwX(B=_2ceMEpb?t0hg!&p(Do2A>ox9#NA4kI=~yHjKZ`|R zR2_;`%($sxK9P6*%~$Bg%ka>VMVBp z-?mW55tm)+r(`p~NT_sb^Y z=(vcp#n);SGquI!6}F0er+JO@5p#Z(TU0pJx__N!{{hJP({FrX(Xp|E7s8)!eq63z zPr2iR3KN#{%lx>+Z#uTae!N~i>sWRExQ%xjZR-2mu9oe7x*X^Avi(YUby-tHU{Uw* z$nbEF6t+KL$Q+BS0fNd3@rlFJhhIyyOIR+mlH&g*L3UKbOl^fS&Q2 zW~gnNQ?b~+Qj`4h(AGYduz17b5{l3u{NQ1C{!Bg|!MrPd%D>+v{QAFwXIF*)oG=tCAD$8FUGq1&AQz6XeFH`$frFAhf-or~B|C?718p>#Pk;NLhExq3F-7II3d z*fnkW#$KXLe5=&Ncxc+QwAZ}9uCD1q%Bm22slxPRr5(W;jyX9c=NaIV*4E%U5vKenTEoZK)l|2{C6hhblG^yJz3y(u@-v8{$D~ zQj#xNfbwKc8+`?di%5nkX$0$***;6OdC*-hzvCqnRSJAp4%3SC0pz?~o%xoBA#Wop zPEn&g*Zj!t4?Fv*0-3Jy7B^WB4o~=~g6nPLh!9VjLkvH`&Us^E()n@-@9sQ>$8{i85ks-TIYY+EYt~Z4+v^N zU`q55$6jJ<%^o}#X8)@5~+li}5?=%pMcRsWeFT@Jz?0yLRp$UD+uh$nkSF=l2FYjj2;zkYl^7M{pxsj3TjiJu> zpVp zmZ98nyd@_y@0}k#5&z!ztE+n-qV0vTU(FvFjNrAn^FS2jcPSyvjYh8Qyc!T{8nh^6 zv#}w8@nOsiHiQNY6*p*OgUev0ob`a~rC8$R`_i;0=*Jt?ImX#Px}B3dOFC|oJHu8& zh#e?wTZ3-9X34SBl)6@93UxSAO5%(7*ky|g`Pq{`lyftV-lN)wF!Q^V^rF)!%vTXos7GocwYej#Q<%-2(@L3JpCe1T%%Sqfa~f%t4+<@LZ7vE= zHu4_kntpfaaM0`3dk%BwpOlok^XhdqBAeNZ{*oV&PWrR$+6BJ>>}*;sqt){;0L)A+YG#L>GG=~#^lZQrh@`F9mm$NHEW zr0y%n6*}5Tvv1J_Atd*Rnq>nYKA(&ubAlopA){F(N4DTub=d$M=7OdxDpkooGi_h zV%8}&1s;~WR60bnnDX%;3|@0RrgW|)>px1&v#?gm``|_KK-|IalTA47pY(}dc74U7 zPYjFq8AD#&#v-`~IJL>=v8u`YH)QJl<=V3r>qJNCt_{) z$!HmfR->~QH5*l@V*9@eln%25f}Un|J${T?lE$JZ-ZOU1Tamew@qU^Rhfu6he- z=k;EvQ5~(ESWzLbJ^U>05=wfuxjLOC)6`!!6l_33BluYr99E%_u8y#TW3D?(hqauR ze=KYWxr*6cRzX1#*Oji&)Vpk~YL}ZPiGPDsLJ(aZRIc@UY-H?f7{UZt)>;bDY%W#{ zT>e&N=a;;vGKT)Dc2WGZgB#yEnD)s9i#!if<1z~8RWG~BUY!Lz2I`4Qk_}x&TFrd4 zuThwxN(0!|L}$eQ^#aYYEvOr&5|&k%Rzaf;wb{2iGII+Abf(riDS$2)vHJ|CZUeBRdjQuYJE4}y9JM5%>{$0)d92)uCd;kkFdGWMdzg;gYU3JXX&Rw;51DKeHxJ)JMd6d(ZMZhVxCAKG82n=q89~N96 zmi1HfJgXcAQk0;XWdbONAG%|y+0VGx~@sQT`yHwv9L-T7`jr&2pQu#wes9=E05rl$EbZK z24huf>8a;dSb6$hB~_%Z1|fq1zmZ$pY3$s{#Mxy#;EOpM>TeADnog1rg5?Ep;IjSdJ-c$vteM~z?J`XWv0EjAZt5@*_jN+@7 zig%)^CCp5!&rXa@UDzZl(1|1nIH~$%VXi+F_-CK<57cThZrDEen%U1MgSY@|6wD9v{gcRHLubSpPV|>0PS%8y@7|kWl%3dLXu64qg zdmDR}){e3dxcB?DFE$~M9eDqWtRIw+*#e`dPU~kxq3l9NAAB*cY>ViX-00qi;Cj`! zW?A6{KRMXpx}&04;oisR(}HF*zr^FPi2r*SvdxGc!S4a}7(F}I4)mVm(wl^;0Fqj ze?()#Z2RNw4xPZA<*{_%tc$<(CDid5@bXywU#)!!JeAECzcpM-Sxbd0Bw6l$xwh;} zl6~KI%9br@voG2ANM+BOL@E>|St7Jal0qnJk}dI{=T`4q+~5E8`G5YspLw4#&n#!o zIdjf;=9xPaw9@4`L0P!xeT{X1Q)Ef3L{{97uaP_KxW6qp&01M_mjth4RXuYi{8#y} zXnzyj{qoWk2MdZq-9y`6ddyM2-08}4VE^sBfU?*bw|TB(`aHYP@h{V(mB0CTKY5i^ z9!B4VxfM?xo6h)DxP2SrSoxU;r+g1ZOn>BfYJUiM``E{YsNRkga>t$RP7T5@)@WNr zQW~Pa_#Qt;wdPdv0l$QJ^!;wZIn?_>=`#xF`7G2aIgARV55K@QigOLozrUn&)W4#D zzImc1#X#DqXcDzt-XK4Wi=Fh9`a!WUN@Ele@;TYO$g=Jf{fqv>FazhY*J|&_xGgWr ze!W@!{_RpT>oX(wy>}=26asdc7}xe(xZXb~yyw!k8M1+V{=uvLWe@1MZ?VBX+4V@8 z=kzq?X5YV_si!^2@uLz}ocyT&XsR~VR|i7Po&8maFlDt$CylA{+KJeCxctX=4?f4# zF1-}?NSIB$h}QSjRgpbvT_Rjh zg-gHEICrb+t2Ta98Z{in{{v^M#cns*s43szuE&lst z5#qrbP^uZ+;-56=IFXHqYv8fqM2-JASwkk_0ka}6t znO)_DFUrKd`g&y~pS;kvH2og>)VcbHU@xK$Wyur9MxZ6ZmOsx{`E`;A8Y}+aoz>J) zcYlOrsQb>o+DMjG)_h9qg8cTW^$Gzlwy6g@_%zIirXMMBSk`FwDntx7saVoo3YrOd zm9@0Ha_G<-mHgC0`etxeln%d-j`;3HHSu~_&$;|yqmo7AugoWfJIy5hDjOCbRF5Ve zELD&mdA+w!&9qmk=Zj4BJ8wmwc6gAiI4_NoOPpm`e#1hOuN$`R7QRZbBt*3^bWXi$ zcjj{;$5z*(7Po_jXqb3O%!TjzDyG}#@K4c!$-|*&Q!T%;{kf|z1k)3CvYvP-CMCsT z%Syi|qM9dFwsQ5%MCZ~am-2#y6Y~48%>{!_^(|xm>tv}~=KD$TkHkqV5?jy}T&z4x zBa%va@#qIyUir7zo_|u9X%JPf{ctLzBKLK-69-Pwt|b~imbPckh=fgYYGwm zk_&gu&$-;f`Bl`3`4n})q7U)C5Bg5oEwx)@Gi_+-)R>KmEROtd*~%nnur$FL6c@7; zRLRx@zg9p06#VwQ*^>;P5H`Cf2qhz*$<&CJ$0@Ok&URX%qUIlcb|2N_jE}B!w8(y9 zgMRK&pLw@Y-zM?pWbVqY`kb$2ju90G7*wcu2Fq^igD1VVfBSY#VyZtW!PUs7`||Gm zDr86Es>}{O?{4qN#8o~kuj4k|8KXOQik5PgI~?hmv?*~FW%wK`f3p4Ci-7B2k<2q+ zDF@|c?MZa`&!c&MAB=;;|96*h@fu7`NLr;u*=F->7AwW4$t9on-CM7*7EEH+TIkrN zKpo0ZZ;a|>_Ilr$=bm{k#9OSEL;a;@iUE~hxLPp5lj%#8{sFqtP{Wsm2m`sGu*K5k zcV0%8|KQ^va4ZKre3^mV9Mme&In%a z*14kDk9n_;D|9X05EhtMdmWDP5}bB^Jo=(_C}HhgER{z5b!XnM?=WJzOKoU=?2EmdVXQk zuu;nA6s)SVEQ*s+sjNKcGl+5YncHZoS;@W(Ppv!d=1mSw7#p+qQU{+%St_2a(LHoI zr(|X~;hmT+&d0(2YCujVeMds%b!HPE;dEcQ=PA>=A9lN%1+#rSTqbxq8#~V2K6^6d z_!(bTt%}MK{Z3xf&T`h>7jC%v%fD#jXTHT4EtuxbtFv6Pr&LnRI$2<7KJaaJs^gge zU3F$;$eFS>=MD;dcXFN6&c~S=+{$rA*o!x$)N!+ci$gW{7x5?JzwTTLJ2|*6%yhPw zLiRkF#+t=863l}R2HVIy%oaux?oa3UYPw#qIXhxuy;rxLjaE*e=8k}^9Ulb(wM2+~ zyOe~>+kei<%Vu3G&d|%Z)Yu8e9yfvC7RwQJkj%2*|7VS zxSi{Msa?i2Gx((oCYC#1KRoivh{H*&pW4nP>k*;NJ;s8yGS;{Jnx3&LGKV45r=}?d z?ya9c-xh}+9hXQHyb?XL&@bXgIL9bk@gEt9yPv+_w&}cq?Sei?Q-{emYpVBU8JqDlc1C$Fo@W+IZR3^B=9u zMM?0BB-|tXH@{@GrQ(!$872r}h*|zQK2L6yQc=g5!RwTnLfS{p)nNF$nS)drP$?C| z0_EA)gBqe0zPJxfggk_oD(b?Y25gV?nJmO7vPyDgVGh}d+r>31qqb#z@+p*|*-!ZX z5Ekma$6?Y?O)#otU)*O*oY0knr?2{Rh)lKO%>{h7PaZ{OK6>DD~SaeK|DWqcWZ&$aq9miBtywt8cKG_}Qt zyJIP<`e_J(y>@Xp3QwaM`(}J{k=5lOLbr~~^zu1hmehe=W4l<=oz8O(+OIFutVNja zBf&uufDc2!ZGnTj8t%<?r~gExA`ZRBig^yUJ1efX@J^J!NrJ~M~o}49xv@n{4 zqq8~m56=?Th7o}edT%h^`5=RqzrjB$r102XhclB!p4nzBk-aEi`>M&qL7nM|MKtv> zkC*~_yS?*t)zk`ZYL6&B%aJKfrs$U^#5wV^Hn0yk+3vL->$ypOol{>a?et_zWbxCY zn0lp0vRaxe2hxs+8a{T{VOAk35)WG|N+K4T5z}`mpNzeiL8j>s)v5JoO3UNqyycqo=`6pMKL$-+C|F zT)B^$^0ReJ*HtPXJwnS~jui^X6}hoHvt9-Bz-9;YtHq{P%H2P_>UUtT9p6`NyzkO2 zTcy|wuHG{Ds9qhJX(qgCVtwd3bNN-yO$_EBZ}6Mi@M9BlepUCsde&S$gc;9u+bmNI zDWHtm0&3KZmuI3x(NfNw3cD%?m-Fv3G}zw1Zl&?s?>RkpTcE4XwjhBB|A;*wdiu8y zD&yImhO?}6g=~2*YI$Bo>g|mcIyq+-A9y0=n8-C}e@l(0noT=JPqK;@l&2?idfb^# zUsO$XOJr2;5~Ww@bxm4E2hp6_by3xK*rlLlV8OLwVU_HuD^{IEhe*I+GERR zwlSJo_ZIKbi4GP{c}3@wYxt<7<9*fWD?f2X(Yg0UXTPjC8>U-_^=N!E_O}X?y;z*# zH=U#q@Ig~*Uk5iVA1x&+eCzb1&+b7hho0!p{%BiV;;Pu*xoVz8f`cS5HeoBUD)s?$ zBb=6MU0r&M^TQ}FvJrNt2+G6!OMy!Tl)%W2A(l z?QhWY3{v}$p76E01Onb>myW#bd{1ZX>i(D+-W08~`xMIxSFv9F`wExT z>8Slu<6Dkvl*LXa7O`D@^JTblUZt>)-SpdZPeEs{PG@^7p>;>j753Vjr(ZDj%?6mO z)y|}J^6q`It{o)eFyI9fkT>f-ImtY@o$GqAaInpd<6MpD&pWcT zvlipZu@`;_%juZNt+u}Od+-j`rsl~px-IUWr*w0Mr-sC13b_mShIR$a3cMag7+sId zoTc-vwMW!Hyz*vrOd*-}WH}j=rF~O*XVvn z@^$;fT1EK$bNPT>3JkVd4{w+oov`s~ioY$2VLY$OIsJGAZCC%&7%n(Y$&eUT+LOQH z*UVWmKl0M{dS0$MLUJEVN78Vq9N!tqn=Q#THJBehmOPWZGpy?e501Ipkc*W}%Ps5j ze0dSyr}H^8C;p)DaqeSpj6yA#5BJN8T|*xoxJPOF;a+N~%$u1*+3N^Air%XAP}0+F=T-B!OF~)V7|cB5$WW^4ye29c*dvpc)#?bAEAf&3Ry(P*iIqOw#{|g z*L*rjFqGOu&8B?dVw=HuB}J)}-lL4;QO73i97gUum)h36=cxsu*#1ObjIX}$LC+&y zso!)4miaM zLXQTLiRy;}>&201xe11#$DMlfBP^4{!+v8y)*h#r)%td{Ci3p$3ii8fzCQG7`QG}8 zaH+0aoY(ix44u6A%CE)BHi_MvJo>Fr(b#b@(_T)=cgdKRk;3ocPw!f%*M%IJSw8ft zYh{g+TgY)`Gb^#A;0k^V+NGQz#X`x;pm_Rd4&M?VrEHfYS_r@BVfWfqxAD0#Jfw~_ z_{E#xsPW*48i4`onQ|&8yVXy4cO1(Im2aS7{;+e3?%BJ|E~!3Woblem^yREvj!Xe5 zd7RlhB-!6h<&Jn2PCs2%^!#S^t-*GHa49ONt(}Xu^RwRDWxP~wZEf3He#_gk-gCl} z1~x7x8c*nMU#@eMqkF4IcvC8KhzBOQK%%cCkPHUBg^`WZsdH;aGECyAcOv2E1UiS| zYKgbcUy^!LtoFQ~$G{R@U@qp{cF@FTu0YZELgAS!j)kjC-x7u|a`*Ap>b*)ky^A9) z>eOMb%1n_(PkQ`)vP>3l`Y(&Lv%-NojY`?>UlzC5ny|?2W?N+&&n!(dwxwW<4lD0t zA2J*eGQLx~}ay8Bs*-8NwolU315?@u*S z9^qfJ4|MI4HJkkWSsCvaF7c^5Fgoy3CqKEo9{B8g$L9O7y3w(W1A-ESR~5Oa`;13N zHFVz@-EHa0mVdW|k#;&6r_Ge9O*=n80go7HR0);6)zv6QS)*Qe((dqbmc7t@Dm-kV zNRO+1)^@VP(Bt)L%Xqt^j%z~dx1LW-?QgSLZMi+{bIfkmd>7wdb#|THit{vywN)$$ zNQ6dhVFb%)-;Je0F+8yd+ilz!Fjeft5v@_UKCiaCu&;+hS@$3=BS{G1n<8MM_fGOm z+9Re@;UC@Y@d5{x5y>|M7Q10z&PI*0Ee^a)t5U?KI}^^)V-A+=;d-ut&-6UVTATg0 z3A4iJ=0j;1!Ch$v@6NaGt$};BY~RK3AcHk}AH!kIZR^*);;&jp?~!f#qR5t*#59Uq zH~ul_%~+uqFPP}=aev4%tC)$Bd2do?J)ebfr5Fpd3r=3MtIngxj(Zk#AwCM~y(XcpB=Fc)%)W9=9b+R{eLh`iO?7Vnf~-!;!*>mgI|SVC z2oA`YbJ1l8xnr6Xw_RZn3AZ>AG@pi~H&@k;yF8g^^YZ>@s@QLZx?g9ur(BsV8w#Hw zr%>4KNP;vHU<8fX%9KF6%?(KJdmRjiY#iTP$4wJ0ee=h8#df-+bZ3pBoVgUUw04I@ zK7vPE?MKCkqp2=)?%kagm1}tn6n^X4`a8-6Di3#1t!yhP;hF>3N53KMqLNgv8fZ+AQ?o8TTRuPSKI3esSjuufeW6{Yxl+xMUY9ssmVV6S zi(6Q|^+&!Wq4p$z7#6vOsXb0t<0c=NAuS5sY1Ge%N6BgxTHaG0UF!$OsU~Jq@l!c= z@hXm{9Qv$#x*e9>up(kMR5&-&c5lze!(S=|ydz(xbR{#}M#Ws)^_lZlFy}tnN(EKi z7(bliNaq0?*3hmh>%fLyTR*o$FT?!u)32vZ_@m_$${kGnV_XoV8mtlav`iZH zx~hkahW!NLgoHMoSN86Z{)M=!ErnOk8~TJzogm2GG%(LIkn+2#(oC5-_2`(Dhu?s1 z>h$0#2{XrenW1~Lon4ElBF|E}Jh6a@`zLuK6wfPNzjKWKOXC!KSLdlP){IzIuAVO| zR2LMsv44mdt|P!w!i&Fb^A}jK87ZtzTp5Rb9O*R4Q(n0f`vFUCZDj8f{IP|D+Bqc{ zyB7D*V0D#-`BTMX65=HRX1Aif7<@Ir{YWg5CP^h+E-f)i0a}qJd(`nUm~QmX)#wmm zMzRmpj2B5XO#;xO@LMnp$_WG(uxLc(eJ@y0%{+Lq?8&VME9=EHuT*nCQDFDha%i>( zusO?qs#M9T?LIy(ayUup0Y?&-_BFdmsRL?9)a;VfYPduAGL5S&E@tQ-tkV5Z?b7_P zq0jVXt5U3I>}u4fZThQPpUqQdHA5!m_s)R!!k-^3d>V=!Bo+KK9Fcs0Wr`n_Qm*_b7o5$R7p zwf^9nYErVY3X}Cb>SK7+;WH-i+j*}Z-Hz>$3;R$}+7Z^J*54`_`6MCwPLgQYX(f>n z)FbIf!84^_xVe|hI@+pe9wYirZWms75$eD@+|2pLhP}mxeZ=^NFK@|df4%&sV*b~| z(d)9!-l9~ZXH>catwyd^#Jyim;&55K==S`2z>OnPCHXh@x@vh_?zb{^x+}@uMbox$ z{WY_09R*t$I7G+$!-409D%Bi*c(r5Zt{!nej1PB*xtpt8;7hn#yiYnizpZ0GY$!g9 z+v^~u&F9SPK3;naPF%avfx8ra8>8+xabK-6o z-Gu8`?xmSH`&?qJ&kyZ-d{pb`eIv{SFM*w|*w)nOfeTxPEnPp0m7b1XW&r2f_qpjW zQs>slMZC{=lMo9DWW1G+`xp#f4@`NH6fA{>;jmGaG~$$3_V;wVL!=`}gpWQUZLgRj zy&clc)-~VS(RQ1?2^e&|R%h?$h4=W|5Pn@k(m=AAE zl9p9PwFOS%qug?-_m6QG2y{GRlf2PFQSX@f(qP>BA$NpOj1hfR5x47{8Ea}jSNSk! zAM)K3q@Hcgha1!{Mr|E}r#}wkDjr5o=_Fe;c}}Nwo}}$@UfaXb{@#u~DNyxVB@3gA zL}O&*6-$X`%de~irp8FS%b6k!cFy~+1;j-)c3-I~uzB3VZTju&cKuIhB1i~=1O(j* zNGDpvz$P}u>sT_bQA%ABHIqP)D9l!-onUVT71$TGn3WsX*upb*0{tn>dwnYKvOBAt zOqhPq-QXMQKuORM5bRIH(q1u(5+bY*=pKsEkkVgj*Z-;BSS_@v)y~~Zhe?Ffa zMMt7HByb%D%&fPdumpF2hbMl(&cXxT`)B&V#B#NPvABa@!ZDjd^t*u=;lY`yVIUd-*7ZLm*&zK?;n~a{7SUNm|AzwfW*{=#hg>3H@8LJ z%SgKb<1&omI4A!lTQ&CV!Y+Mdc+b>cb$-&JLmjUT+NBFejI8(4w$p55jF$5;9mz*Bl*=02FJ4Pt1&`v$D2Q(IGt|yI=t91X;C4Q zA>USJ{z!mz=aH8-d`?V;Rdlqs7Y^8H>)m9iR0aq#$|IODclw(!~}OD%J$BG!&C0m0VmzdsWtv^ z-~P?}X7ZrF#@JJi=fkHv+vXnXTQ^_ob(h1x92%^T9R40tF*dfmN@-pas?u1Mn>v^oO!l$P#3!7t`xKOPe&d};4+ zV-?(Ps7GSG?T+jtl>9)hsByYqOJ4_z_d_&_qDG~yPdc}8dy6?Oi+wuJ{YLmci}+>3 zuO`X!UALtM;%loNtz>c=qb6yaZ;%@W{-|vU(s;OR`Fte(pnx4~#9YwleY$w5oE^9H zC%sB8R1I7b_GF)5?NOYbqQ1o^JxHS4BoHCETkSuGHRcvp7a&4B8B3v~3aoJd=s#}J z!Py?>_|cQoBvZ)Pn2dA3jU#pLmv$<>~%x4x~JXXK(eLoklaT}=D5NjLHTbPmhiq$&8~_ZDS759l^M;Fiqs&hso;7T45f0s(5>G~7^I`HUnJ z{{Y%Gr#Q0=TbIj!EtGRc>)Nr1>GcXejXL@O3cHxF?(RJ$nN8u=mxi043FseRVr!NY z5O2+}W-7>k#-*AVU@Ni9oEx0SJXATp!d#c+c((hoLal&v;OR0wm$!u{SRZlWs6PvR z&dRIbarbJm(`#(f0JbX0`wnBn=`OwA^=qZ|PnN0gofKOm(JN8_X$vzynHVW1z;;Og zRh`fA(-3)Stdht zW=sR);DBZ?eWAq8MiMlUfJtc77St1`Ltp|+x08$PYjXTC2!E~I1D$IPoiROz+z-sd zt5bM0=5&*bct7gN24sk3)xPcRIEEcds}}VRyi-$;C{JpPa*Pal;BxT}d8RbEVsB58 zj^a-4?iU4m2X4HjlD%--efxmjt)TQ8-mxgS?Ey{^`OacxRt~o=!F*f;SUG!_J~hy) zSMWG(SI{=WsUr*s(#vL{s30|tBF7Fh9qA2h)}TccK8+3 z)X`Bg6jpLLW@G59XXEY-jcqS)A4O*yFHjh9#O7BVjl;nZP#Gl$Z#yp+Pah9(sUd`6 zbm0g+Ut1sI&67|Y0Ky!#aRc`^Dr~I!XL*I6cQ%TmkZ2TK6bn|`QR3iRFbWO854f`t zYC_Y-$IFFyv7;Cg3F#g9@1K|j3;_|R4epx+S2;qi4O|AP><(^bgl?n!_ZH0GNic+x zi=(3hxJwecJkksf?MU;|z z9@GIUh9khep3ZQD3%JD+ygm*`xWW-`a0F-q5a9_!0hu5G1wf9V8@_Oa9~^NKFc+*r zoq{6*z`lHNdnLli#oosma1TPae_n)16zK0AY5ywBA0j8p=YPl-2+!5V$s2HZOOlrN zCUTyD2F4p|0TP2K8rmqLXyd8&lVM00!uWR-+#U+q^ip@^o9 zKe0!kLtt6`-*0!0P+|8D{I`#lEag#uK^?|=ha_)tiQ0bnmZUIJ7^kbsMW z*)1N71srG`x-L~5hlZmGI2Z_UJV1;%9H;~gPH`;AD**^Vf;jYq-eW)j_qK|I@=%=w z;D7NzZJ{*iN$d{!U=0K<4lZy70Vhr%J_!I| z;Xyhcg$5n`*)A5efdlG{AwcD@a2$9_0A@i?Vn9Rr@I)p+PpEyO&>#i^8USNqP@6bV zhRBrP#6eG@{y;5Zp+Ariiy*S0u0c-_h;o6N`x#(3&_!ZNh`;|v0D(WNK!YAbZy>4w z<-gV7x43>kfylrIfj?@%iUU@HU)ez9LwfKRsZb(OnSeq;rT(b}A_|Fw5#Rl)fPg2! ze+Nhre(C_E2Y&}5uYq1bI`Ky!>I0+`zXMTMe(40H4}S)TF5=&y4oEkC2cljeaX^NE zR!9d(1cC&R38WXmY5f^~>%{LT#1tTR81V_k{|H1`|9lVS`6obn0ZA2l{`#d5{S;)y(ZAYpfTMr4^V8A+D@d%v5l-Ac2ihV!5g;*QHAKGxSW5iE3267v zm@^!7lbH6)MS)%tt8oEZ^Yg`V$j=e8xWa*ah%q-Xe168Jb99`z>z`R4|0 zgrYpigs`&#%YKmml}G>YY+^uj{>X{>pE-d`|F=Ty|ICIIg8;?9IdT7&oWK);0Yo1C z`k^84K>rq21eb0@xbpx2h@P$*-2Qnd=HcZe1VaF% zXzyzWF7f`en5VrX+}6hKIB4c~6(B3I0T&N>kkTLix4JD3)82?uq5{X8FQS?_D$Ol=BztYeI z0&o`_)5Ik=0CoARECz)HuI2AE&<6PEzcEb$vdDjxMMb zeQ2H-g&`9JaxuKC@!``NR5Z>5O7KMT=-{1A2 zFp#O*kOsN+jcF2)b^SXZ8Vz<%ZcM{%(if0zlZ=4W0JPXxmH>IUzuUkdA>Xhe4I{oe zALPO}7*0Ih7whuvg{ zKnvoV`XM2{nFkVtO=AYwFaX*9p~c6`#>Lga3kFTV^;`lRfK7xWbUi#Eb53-<>h6vn e0EH3%01m_3$HvQtXg)D$usQ}45Kz`qf&CxxSevo{ literal 0 HcmV?d00001 diff --git a/analysis/pd_sep_paper_section/figures/fig_c4_ttft_stacked.pdf b/analysis/pd_sep_paper_section/figures/fig_c4_ttft_stacked.pdf new file mode 100644 index 0000000000000000000000000000000000000000..aa5da0f1f21d9f03b090b7ea87bb494307f5b48e GIT binary patch literal 22937 zcmd742{@Hq)G!<}bW9l|Lq|e{GoRxa${d+zA@i6q4oQ2}JWmhjee3_O|G%zZ*KY5#?>(%&_PWu8CQ>pCFRp>1 z;7GW){b88AJRD(s%*P3iP_ZM~d3d|P5&Cv6PJVDSXkZB6yBFr<)9W6av;G8D?il&N&AgRb|fb_cBMk!jzn_u^@8J|pD=`xtDU2pmkS)f`qbCk z!N`dOw*=Fwssm7*0!eU$rY8VFc{NvA%{AeNegg-!aRCtA5cIxI0J;_M5r$5F-u}K0 zU_B80V16AZM>jiV??5mk68ysw@zPi{5|75g(L{_i28$%3uvk1^226%f0m}qq`N4@R zz$$rpd4pdlsG}k5U(`U4*Kn!rz3)lf zafC=vT+XDy!H*;H-94&QghMBjlUO1`ebuuge~`BG5VwuZYX1^18@uo4L6_fq+jrIv?n%qUeBPi70#CRvSHei!D@4UrqIbIg%h- zuvNJ-%XV)t-+rC}*FJACuP{7{Eiy9jyIgMJ=glJm#}raOT#xP=im<)}`_S_E-DV7@ zl-xgPv-%yZ&FwIk7fpA4Q3wpeh?F!7rkN1V#qW8K{w{0 z{YdNiz#R*iVhJ*$DWDe~Z_qQ`zLCG4hU4FgV6$r00iEG{8BKy@P0`@4uC1tcZQYLTAZJ+(z?92IkBSSxjja76r-@JAo(&E985;@Zw z_DneA36Gm+oRtaKLn?GqyE(Q}Ck%wb`U|$k!!8T?E>2W?pNla>p?vN=#GbY^@+I7f zVVA2w&%C1A%{zSM$#T^G1b&|J;+_zrnx8kiwtn|LCOsA8V{L#u8G)PkMaFT`L$>+MGAJO+>38tM@$D=aFM&~}yJg&#)#*fzxRt`T( z_!^YgE?fR-CeJ#`Ac;cbGHvoSxjzSkZmwE(Yn1S9EI1 ztLagZI^0SPTnD8VHw||)e#MAgP2TcR-zs-ptXmf&@{Yshz5HC?t{%rnUYdNeul@BA zzQ_CeER%iw7Iu}{YxwmZ%or6uF(Ub#>U!18BELPi6W%i7&aFU=lB-8_JQgz_eQ4hm zFMMUb}B0>QY~ImzAALZav4bcmK~Dv2~(5 z-O)B}Q(wk^On>&mC(YM)yg3<}nSLpu`Gjl&p-KQPmtkYo)@$Qz<+>BiZkIbB=p+wAswpQD$-#5d{0DNht1b}9-~ z{TUPF$94Ep`WLDjmx`Z=cpqjBkN6dNT&LOBe0;{e`jCHRU;Jqcp2LIp!;@=0H!D^h*g25kU3r5XU4i!7+e}2Ce8}M-^a4I*i$o+g)6ZK)eCsK@M;YBSH zHXr&}q>abBn~mkM=Z+ftKY4hksP$(4J9Oh?iI3yg5wG3`9X6>em8c32)Hq;fbu*0d z!vL#Qvxr~d{tu=Co?FZGy*|$9pOUFdx6a5|=y3URVCF{95QkT1k#`oqt-syOh%%8S z?RjHrlbl4=w5wIM%CXb3_Y(GXd|H|v)qQ#Bv_y~1=srWXgl0&*uS%m;K~EN4kT^r( z@q{($PnLO5r6v5(KXs7EUnlnnSRm{EAoDO4NK>T#4JjTlHj^_*pB(y%9;B zv>csO71~l=A(Fz3G`o@`3xzbe!<{^Awej-FH+=gKyZMrb(*>`Vup2F!4 zmMtwZck$fl5A01(qx&0EtZoW^^HY0%qxZ&cp~nNusS|F=jHft*F7xzB*pJprto?c$6t}k0aY!^CIyAh3ONY7eFUqsxWcT+9qK3fBvVe_)F(iT`~R))4H z>+CY0QuL02*CQ;r<}+&Q2AbyW?|T}8vid^>ZnDR3znv=Jd+R=97cNFt<6$N@mhj@E zzO(goR^^X?jE=<#dbHki;?<=SuARa%6u|sl6I~@b3s3?cDa1n#4<_f)no^)v0N{)QgumttF5~sE;W~n!KN5 zX5EP_K5QQnbF#TetUmdY-rI0%#t%`0+QyA@aMenlK;c6(bbGGbpWKC)NE)SY8&qu? zqIT7{59xRsslQ2~i&GwNWobo_U^Y~AXzDwN-Lzn#aq~7w%OG2z(E0AyB-2Nq&0o)P zh%ag<^NHH!b;WM~_*~=|X(sBfSdiQv-6>JpEZB!FVJs)N^zrn)-WyHOM%l;x_z_Zs zV0appBM#ehn}L<;okIR6i+i1#boX9$?h*Av%5XEbeU$T{v-3IALnmchyYxZ!$hDca z8cagUJNCT90Tg%KIX3o?_AXl~Ii`izX{hHkoMCRNPA39IwO(!&5V?G;k}dpjr0n#* z?>B#)e3)rQ^0+CLGT-nGL$>+Cz<+}y-E!z|iJg~Nnce2Y% zr*_j0dN^u=V9@7muZ+qN?^nh$b$<%`@gQCJ%WgvZ1HEE78}EbI1?zU<1;$WoKJxMD zzx)*q9h}y|hQ|H{y{N^?d2L29sT|J;-DckK?fKr`%~70#->cYg9JxEGG{-7Zc8T78 zljkRt0{0!&I@@wzcCm8mb?>63`!?OCse7M1s+do)cs$kSa?ZQ$!`vCo5Uw{XxG)$L zU0`0*mYyo)(pbE_eJZK=p$GS1krq*0?06X8&U|k)uc?r|?c3r+sebnxDKY$Iu1M3Z z#?0My?`wZ-f7N8#5cL*QvN&~T`bpm1mq#_S4ocVcmdF&=3_R71x{Y_#dZn>wv^(9l zC^kQ6OqA1ezq421&-RmVjt8927%VJ(q0;K?6?=w2;4{+b?7w#UWm8q%X)3Q%)?X;> zDs-e=hdCVj&o3J0Hel}hNGy^%Ui~dM>?}7~;=rl4VgQFvrE%un7|NdP+izdyUn#-Td?O-jw$z*@7|x_9WW0c0}~nV zjzsLSLamlKE2DZ+Zc(n@(p1-ca&N>e`(gHa5y@SmE`(!e%Y@mCh7)*3F|vD`zRq;n zKRn*t!1SzWx58Wdg!4Tyr|KkCjgJcduwZj;uH;gKds zvED-a{2=!0Uv4_4&&52yb0tgEQ`0)~jmAhlHsr3^kb{|AqwuGBy2Nc5&f8_Vnernt z`TPg-BIU9hmoQ!t_OaBZ&KF*0KNm$=uz6;wYEYZ89h|;WoiX>`s`j9X9q(cBQ;LlP zF%db2gE7}i&&4zkr_|wqmWKR%N@d0jC7*9zfL#0oVd7v zcv^gFJB2NRYzphPi0~Kh6{Z4f1jqwJ&vw>%qdL-c;X;U{gM)D9G{3Y8hZfS)-Zjhi zup5J+LyWB=$;l^=YSX3H9LyBJp`adNFv!247(*?HVumJwTrgY|n@Bs{uXcRd)6@Dc zH7)YC=8JeKr;l0ET%rOJ88!TR)|RJ|hpu!TjcDbP+dmvrn|y2Z)`N~_*{hQ+Ftoy? z+BxUm3qe`RoF;H8lgJ2#3kAcMB|Y@^XD+8jRw_k4QgS-+Rr_rB44k7!emFw;wyMC_ z(?Qnn+{c;S^6<`s;;;7HmkGHdXn0&}PTgbYm!zE;XOw~(%EUrZ{V2bB500>uNWwI| zmq+;~u^D9Q_BkU|H!$E9O@AS+hVM}~gC zI%7?{c5wr%w3ln~1AbT}-jx;krhRIQaN4+I5Gv9yT>p%bAKlEEx87vF9^ znMd54LA|wG@5u3AhPFguN81zD(;7JhJHLzKkq-Et7GW&Q!fCkRyPu4QMowO(uq71q zLzK*)cLakk*GC37ajF3JAjnH&LSbYIDjx>-7sQ8)*ShH_#GpEHJ(St_%!wyo`lDh5 zjpt84r}wT4QG!ux+FywmzcrR8ohPzK`xHqvY4OMs!O-Z0UOV=~aJ+leQ2au{l)adD z`uMXRKB~g}v6qhu=h$QRJ!LPtl7B^@?yG|2&kwN+kHimLG~3!Yvh#p&on&{-H}>%p zXLUKERrbj1b~gVstZ3J%GM{jdD82JlVOVhNi8t?-^Dk(9192&5UJnM8zPKC}+E&_6 zL=NT`KjF+z;p5I?vi7t)HIv<|Ms>IMMxM=>kNX&tpvIfxU0YMmPiXWBCssua;Y+3W z={H{9Rvla3tg7s5@uM^QD8v4NXHH&Aa7=S{vr@}oyt`re_@0V!X=2?V%O>tCBi+>7 zj+IW|Y&smdSh}66vB~TEy~TxB*w$7K%XaZp@50AH*f&L)6Cv$2_)qHF3-42hAHQZl zJrljFV>3p?#Nq6ZQymM7a=(7iw^xTEH|`U7|2FUlG_WQ9@zA2652@x%&nGzfg?_c{ z?JX{<3anec9he<}0hUnZ0Q)!Cebs!eTsAi=VfNt}xK_K2bDpD1=Z+It2g>aRZv>c& zSOw08T8`ejY<==p>B5rrNoC z=zqAZ=jec!`t>+pjMg*_>k-IlAqxzs&dK-|pK5u<<`3W_WmDB*v` zO>Ad04Ny$xuPZZ4+Wwpka%LO5`P0TM@d`hkC~}M=Z$K%*<00!FwK_NDd-9T^ZZbb_ zp)Y<*wK`B7_dVN%M{FtJ(XHJB8nd;%ea8~Vyw05&zL1Jj!=|6#<~OY;t|=mu`?1YO z_lG#Fcx%CliT7z0G3c0p%%S#7sok!u(=HhZ#-u{P{N&(KAuo}lTmbn+WY2o zXw#&EzW))o=0Un-nUG34tCs@$xT5Zc{YBOH#ss$!NvPlTfEB|z*&2>ew)1mZDXE!iswnC0QFS_OXXx{>YQ(nut<<>22x zCCKUxAk+tDd0^-!8|h&PH7{VdhK%+o`{E`kVrIKhJc1+ka+NrKu$Pt+JPEiWWZeu*f=Q1;Biog|0%(Ke})AM zg%UKzTEc-bz+^}`0a%*|cr=`V!GebXBQh9rk|9GyfMFnDKq3I*$72X^BCsCFfLsO# z`eFcJP!1@J0GJ$J1^^ET3>p)KLSjGx1s;G2&^Q>h8VG7AhoHcL$)SV?0~5eof+K?V zAjcB`S}-eh1TYu|3xoqmc+eN*07Y0Z19*U!A&@&jov@%Y4nT$loypI@k%kB+#$y50 zpo{~Mkp}~VKsf+lW#1qa0wBn!1(V_daAdTsK?mhz5`xx;1wBzHJb~N=tZ9vWVB{SE z7=^l%6NX4$+do6C6d(YhA&^7>8MJ%|DgfmgB|sEF-Z((AD+sCB)dYUS2+$}XLBK)Jp=Y?&0l?-!z1RMOsDVrf5HYMvD5MMyj0GlMrHr+F zjWX8q71COv42UZJlfc~M-zYQ;qL8&jCJyk7fFhC!0V0O=32Zn39BeQ|8NVgi8fC2I zD+IAZ7*P551Od1993p_V1Q7=yAQ+VYYn1){5Tp)AtzOFl?n%A?UFEd^WaxHxh5eH4 zY~Tl(;euC(`~>3EX!0c{4DejC%>p!8W$0kv$&J7*G5K0|m2Jb(WP!25$h=KRu9%kt*LV z&O!Q>NwP_P()IM{tIz5jl%BYy;Fvw1hkPu(xUUiSGUxsF89`6xrpLANr`R*ZU6l2Xa84q_M%@CirVWb) z!uyYF#o!1gz@T2aiT&Eyw-ipW+FiktDq4=ok>x_MPWEYOF7Iv|r_6q5u?y(-w-X zD*WN|EA^OPnD0SLWfUVcBYZSI2}3>s%E5_q;st;iO~zL4ti&# ztj0;(b&Ys0M%{Hjl-X&Ae(YVFUfp197x!c)`x{Sf)~6EZb7dwN)EPn=`*vH2qrR+P zzU0fy4#mfMnA!DR*`8a0?27xLu+_+~&o3hGhp>&WyIo(}+ilyVig`<&v>(pc6?sT8 zO-Cyo?fB9kbo~>OZT=I@E2X`T8Wdksh*}Y+rG)rowI=gBovMb8024&ghITQ7j#Tm*c&drwc*>~)la#* zYf@jH9ugCoI?q4It#&xMM*ED42zHNkhfMfnVh8GAGy;ZW!uCXr7cqj8eJ_Sl8&m;D9wGW>^Qz< zZs_IVwwv^YsXcPy&NtqX45hucof8sl`&jUPJZHdNxp(o#9+A%)17|V5Vsox7Z~NOu zV}HGj-mG=;x~tHqkutBR(zdlmvTF8@y1pfDQ6{+ScJc&c}&f*|2@0POk@(r4)h7cB>F@ZTJF8ZzsN#HpG#f95aHl(iyvABd)U@B<)G@SA3i} zXZU)%hh-?|mjfkYS2D4aY#r~9Cc2;S=g=!FA2aS2vgj`5*na+oN1#%F`!2Ry%u!;= zenJLIMSR7w(zXdAqo0F^Gn1T81Q}|wA;V6Tw7YguDn*Ai5VF~g?bkkiDy z4_+9(UB7@oa`Dr)#qguAmiMrnd`7+Z^d_xehv+CEZ%u!OjxnnjHE8I-@<(1~py~Ah z$#nNS*N=uxPv>gInlsORU4CZR!O5T~a{G>my@N0{0<}nt7+#FW<%phg@iqG)sTaP# zv2;X}GbH7;bwFq*UyIW|7R-*?6w{vEmIS6YHN1j68+t%wt1R*^o&SLI^T>Lk*A>Es z-RjAYpRs>_B(b@}==Q6rPT}C4B-{0NGGE+V4GHN2^dgw*Ir$d4If!4?On00jG z8J1MHxZK9hasN8qTUY<=JFBjcF&hPhQLwiY{^mLrk2izt?VID&)#+f@6iv?yZPRGz zYf}|UVcf-FKOTG#NSucw`fymR2@;J4z@ZH^P-EcTp~v`D+lCTtlOw5O?HG35-|l;7DfbjCPp#ZosIK(TJpy0YRtCffTUh? z`g#1~`GeM52O71XnsK{G579cfU4B4p_quS1qdeNbG~dX4KQfEy4C!`LBHYh7_j7v; z`t77loY>W<`L9Eg0mM@*?-=pXLR=Gz5<#K(uDrJDbR}&zc<3p z`0+W-m}-bc7VV6g#>9wUl{lUe$SpbBhPM=pR~Uo8HZ+WE&&x>c*yq`i2$OYwxLK0D zdc=`KN2c?xrr`zNMD(=awSb)`OXS;fWG5YUdP;=T1|RDUr7iONRM|Xt)JSM!Kp5T?u()!{pFQaX&jb&9ipy!Up^-KtxFo11kF;_f+Si5(^EAfoA?#c!nZeF|gOvgmq}y zMr$D8F{5C6u!jL%h{(CXb9}EK4za#c!*jX3xol%7ZZDLo=aYdn;*SIy_xrzb%WHk{ z)uZg|k4-%uSPGsS;OMtO%la!K)iKaKPXk261lKo3A?t+Ne3dkniREVTgu`S zl2gqYSy5+pNGdGG3HcFz@}0^ayn8SxzI_llZtOHNSTrnpggA7W{P|fdavSfs+c%9`Jd}?f_WG{w>?C@I z`J$upFi&s<{2ld89?3*Jo&$Zbhh~r&~zbrVom`Mk?Mu4IrpU zeH=K0KqUEIa+&cbX+AHMSzD~pU zKid};1_($>5I^p%(p{(+2Fho@fFPvC&)6Cu}EJ=FZSE-ae+j zgEmRjn#-L2fWxr^z%OB%gK+Ay2WXlh!5woA6sT|5sfW4_R!(V z0;9usVR;HrQqZF?e{~1HdN+5^$R|YObxQ1~$|6XGwY;SLq!Eu>?QXbB zhs`>n`n4+$D9y&4k3n7m*&aD2 zQ%n4tFf2&lh6oW$f%=CpHzl7uG^rMS{^uS=gM*4c+MWd5e~D_>@Zo+-7gOgW-<;;7 zCDTH!c)o6wC+NM%z*~gb^@#NMTl{Mr5w(q1pTC_@PGC4%x{1Zwv8l8;ryzNDN$LAqn^U$)n>2ecD%lfu=@ILo7T*k2NZUPf}-{} z=U{~{IpDp)MAN5`|DNrm5Y?QKiNMG&cN(=9S3L&1xAQcHuoql=FM8)pz#}d{_wX=N zPwSXUi)qPy5yvF%G>suXIrgb6Frs)ky(z#%K^OWfLxfsv>k)8qq%vPbErJY&eJN@E z#QN@6+1byJm4bMbne6o%Z&;Zfu_HBItdhbopWe?q*YXYRQ2WFjE;dQS6c<_iF!x(P zGjGx7u_yM|bF!@vvOC$k;@=c23ZIa@*_v?sHs&YEddG~=Jjb$x#Dtd}m2}aZ;*#Nx z5B(Ph4W`qxE=ufi7d-sj?93sy14DbI^U=C5>S!!p*CiDzJfGhu7$EzR!eS}tS%2fd zhQVXNVG~(yj6~xNka#S$JqP4Um&vf^+oxFMdndEQ&lY6pJC8Qe^01&O%uYeG0#@8V zvJecq3dp$|SSt3j+~BC|Oh@)BtS^kLR+iqT!StYms)8xFQsi=IgiCNL+;eL)oLjX# zEs|GN;4)QQ=g0w?O(H|w?G)glpt%5h;veBr=pwQQBN=G>t{VLiKGU{y;Rt_J;Ib+5 zA)R1xP^OSfSk3*?*R~xsB#wEbH&0qFvu`N~JfM?Y!x+?GS>IppI&$~jTdgON`?`C# zU17Z|!25y%XcSZe>|Y$kDWWQQtEu!hZAtnr>W9fbu$@=$C(&)Hc44M)Q;4G=_tCh& zP{h^Zw0t+ifiuk2P~qSoWAX_HoKV%vc;+5Qd_YRU&oc*vudz!`a2`XRZ2#Epczfz7 z(NvC)mQ!taYP-o~Id!p|(StPi$iwd(oW|}vmZNLt>p4U$bUc!C!Qa?l!biI|>5IXO zrGSHpC-<{B!3IkF_&r8CJezpL)Gw9Ju`EQw^~8+!H_yTc-_wglpG+#=<8#I1>q9Lh z%jTcT9I0bbitm_0TDpxskFn3Zc@r=ZV(Wd3O=ECtTbvM&Kxn`ftL4$BOLfaf&dT-P z;=Rs4KYBFvX+W!ueLR;RRn)L}!GycC#WP;nmkF5Gv1{MY_Eg)Z+zr#7U)uMy_uDTT zL2>7A8+Q;-LE}LEjn69Z24klY0^VS{S;C9LG<$oU(c<_8Z-)U7!-mJ^@UXiaq5aQ8 zBPT=8-4=O4J72ol#o@;k-V4V*w%I?}H1~~bqT$KvX165L2Uon`9>!7*0cVz=#2ntt zt+HG%XS2tAug&!=srY=c`Et+x1@Tg3NPC9>L-(}N@Dg4wyQZf7S8nTY$+J^?W=!nd z4r+C7sk(C4S#is-5%GDk!oD3a*{>AVN3_TgmYTh4_S zPm(KCuqm#@Q;ZmiB0hn~TKIGy367e~;U>&CW=|50={do2OL$+d?X%l(-&R&0ru%6e zQ9GFq=+G@+_q~{59mTh|>4OSqTs+HL+_L%4kABQ$Mi<56yu9m2tuGg{(6I5xr`HM} zGB1~AXLG|T>GU>ONCn@mvi`+aDO|&(_%-suk#xhDtb6W0dA*2nF{1$rD59X4{0*;B zyKomWx2hh$X1n>6=-0gliH$GsF>e*|x+C^N!Af9Dnz$FHNrmnzljPY$M?yX)BN?ss z>&IM~$+3G02WTBT!4e2S~qXx7I>%j6EH1!1Chd(F7YIo%|;k6wM-iT>wNH~014l-E0-d5*N~E|O2nl!|N@8{`g)yP%?NJCo}0TEtkdbnh?UmTJ7Y9ADs=VFuXLIsrxOBfE{1OV1c%4|+JMH63rrt9);{%gOKU$Z-=w|Etdt*3@wZp(b`AEU|KTY!?UJMLi4{ccRYfV}jipVSabmQJZ-$ z+;ZVblwi_*(ko&19g*)eAaNrQd1dFmo9V9kHZ}Z#@5Jeq2ZodG1_|jf%tg~VbV*tr zExdVU6TdboE%JhSEpzK*Atq+)bh;9sr_rhKaz5caR%?3R$2X3u3Hn#@e5ki*SIYh zCY)|waVRy-Sfax4b){Q#%)NS_&+X` z92x;<8uEi3%ETWd(cb+rM5PU^Bc+%^6;e>G6BL`?0BdC3gT`1eO%4g=bcZtGAFJ2b(m7uYkPQOm_HMx`mqw9bVo7oy>*Pl>qxoeL$qzTRT!GRzyo1P> zok%15tk*Ya``_9-h0cw?!Id>e%o-#dYVw&&?mo)!(DfG|cgKhWSA6jP{Bm|?H<^Zr zhO5>x&DNhdh%5~e4p-78nH*e2^Mhg{8~UzRyLrl72$fXmY#-#oR8S}dD^-^9Y@?3jku9TP{;Q{jHgv%y!qI2;th zjYFzKvlOJ=X|8bX5aPZ4p{g)BM(yBU;~+y**7lKeB`uvo!Rne42e+4%)6hklki}#q4`0$M%<5k%!PxTG)G8=iE-IaV&T2NPFnMro4Q{)}`C! zW#erE7X{6^gtJr|(9I7fwriu(O$mU#1={PRFL^+(x<1bVT{=M>>m?P znAovS(eW+xrO%0Hv(pSr1FKXLn7dJ}LZ4W!Ht}vb@J*ic-bE1(fyA~~EX-M_Z3J^J zT*kkoew)3|lZ&2Zw4%wnI)v-MHuv}iszU9_kg4Ue54Y-{J=-BL8{9%+b0{c)=)X|_ zr3^HEX_Of5_U`$7#yS6$L&~0WW6RGNI_T+`qZCONqnQ^n;F4yw<8j5VO|LMh&{wqQ zPVIT-X+EZC4A*qf$7Kb42>L8Ia3pn0SHja9OwJ{&KQ4$5Pd2|s@w!w!bL!tZbEr%q zO{u-a>VXKyHti>N!Y(ZP($LbCehLKmgt~c(U96Ysb_{zAHe5p}yy#MH&t@g|3)j`7 z@xGkj^AHzFXHB9DP&0l~o;JNYGVXmd<|QA;5(N((krvyADB=J71y8fhz^&w?qDI3{ zFV-)9Hxph3YEATTKYnw(yZvLMv2F9EXI_fYXm?-kX!&fhClq4y8@F$VXMm+w2WOPb_3c_C)c zCH^-J6>x{qhce6UclJI&DZR*+GEWI;9qeNFYeb`{Z>zTt%4avQwpuZ;JErWYZ-DPS zbaBb_)4_z#y;X8AF4j~!+bCo=M9wfY-=H!J{#nx+qSd%${dg=zLd1dN+{cjVorZY1 ztgW|5i$A|0lkB2IAv$VH_;rpp4e6ab{4C;ku7|~Vm5;J%q>P%iSHVu0aa!fu3a2WWs>iUE^~SMiEs=a)Xu3;>ik?f z>m4Rov+qR=Ka2u^lvK&TxyVqscpn@t8C2$%V9ScR1L9dH^zsj%n_Dgu*1F3WMD1`P zyswY1D81>d?WH%(k3@{!7de|1MF?$awk&zMj|BF|1=-8+SP540#f+9ueq*~E?|iba zMY%@AHTZamk=yXKBODI|aJ19n)0cBN0 zrF}F^C=ePq+8dN6eYs}%S=V9gL~^B+U+|sVwTROAhDhg#u={SQcc{|ksZ^dlEHF^n zCfL`XXSDmq@aDbetGrlWINSe;v>D~K zB=dyOQJKo{Kbw5^K1o4O^dX50-CvCB=l>ssLI8yw3*u<7-~dnd)W+apL@2NTY&IIWgky-n zMIWR!H+KwZj{(#|&Vk1-TVKXrGm~k9xDS%$6%P7vC&(i{?=zzLl^&JhsbvQquK=mj6rBRBO0 z>}93o2T4S79Te|JHbsHaSE59~EkAiI@HN3zI9B3C0Jk9bfZ{~}MU_uGfYXr2BFB}G70Um_m4Jt9;eqNPtZ8*nb?_pu%=NFVfqt(wRYLt91NqOV z4iPF!pb^5s4vvC^6haC8KQ;qu!oSTh|6?o zR|iGYt*Zmk>FZ$uftJXP?T|oVt?P@2;tJQ-VZeRzx;h*Y;tSBs=NgH4RA6nVi4pOkz_a!3N?-3GAmkj}QQP6j9s8|u&-X-EbGP8;j{;vsY6 z5sDA|Pdfnd2D%5qoEvbDLgI<*@dCm*Arb!H@j!^>`n^PAm9RL6T literal 0 HcmV?d00001 diff --git a/analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh b/analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh index cc61420..d948137 100755 --- a/analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh +++ b/analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh @@ -41,6 +41,7 @@ WITH_RR=false WITH_EAGER=false DRY_RUN=false TAG_PREFIX="pd_matrix" +ONLY="" # comma-separated list of tags to run (subset of the matrix) while [[ $# -gt 0 ]]; do case "$1" in @@ -50,6 +51,7 @@ while [[ $# -gt 0 ]]; do --with-rr) WITH_RR=true; shift ;; --with-eager) WITH_EAGER=true; shift ;; --tag-prefix) TAG_PREFIX="$2"; shift 2 ;; + --only) ONLY="$2"; shift 2 ;; --dry-run) DRY_RUN=true; shift ;; -h|--help) sed -n '2,30p' "$0"; exit 0 ;; @@ -57,6 +59,20 @@ while [[ $# -gt 0 ]]; do esac done +# Build set of allowed tags from --only (if provided). +declare -A ALLOWED +if [ -n "$ONLY" ]; then + IFS=',' read -ra _tags <<< "$ONLY" + for t in "${_tags[@]}"; do + ALLOWED["$(echo "$t" | xargs)"]=1 + done +fi +is_allowed() { + # if ONLY not set, everything is allowed + [ -z "$ONLY" ] && return 0 + [ -n "${ALLOWED[$1]:-}" ] +} + if [ ! -f "$TRACE" ]; then echo "[ERROR] trace not found: $TRACE" exit 1 @@ -141,13 +157,17 @@ for c in "${CONFIGS[@]}"; do for m in "${MODES[@]}"; do mode_name="${m%%|*}"; mode_args="${m##*|}" for s in $(seq 1 $SEEDS); do + tag_name="${cfg_name}_${mode_name}_seed${s}" + if ! is_allowed "$tag_name"; then + continue + fi if run_one "$cfg_name" "$cfg_args" "$mode_name" "$mode_args" "$s"; then N_DONE=$((N_DONE + 1)) else N_FAIL=$((N_FAIL + 1)) fi ELAPSED=$(( $(date +%s) - START_TS )) - echo "[progress] $N_DONE done, $N_FAIL failed, $((N_TOTAL - N_DONE - N_FAIL)) remaining, ${ELAPSED}s elapsed" + echo "[progress] $N_DONE done, $N_FAIL failed, ${ELAPSED}s elapsed" done done done diff --git a/analysis/pd_sep_paper_section/scripts/plot_pd_matrix.py b/analysis/pd_sep_paper_section/scripts/plot_pd_matrix.py new file mode 100644 index 0000000..2390be1 --- /dev/null +++ b/analysis/pd_sep_paper_section/scripts/plot_pd_matrix.py @@ -0,0 +1,490 @@ +"""Render C2/C3/C4/C5 from outputs/pd_matrix/. + +Reads each completed run in outputs/pd_matrix/__seed/ and +produces: + + C2: figures/fig_c2_pdsep_vs_combined.pdf + Bar chart with mean ± stderr (over seeds) for + TTFT p50, TTFT p90, TPOT p90, E2E p50. + Bars per config (combined-ca / pdsep-4p4d / pdsep-6p2d), + grouped by cuda-graph mode if eager runs present. + + C3: figures/fig_c3_kv_timeseries.pdf + Per-instance GPU KV cache usage time-series mined from + vllm_inst_*.log "Engine 000: ... GPU KV cache usage: X%" lines. + One panel per config; D-instances in PD-sep configs highlighted. + Memory-wall threshold (90%) drawn. + + C4: figures/fig_c4_ttft_stacked.pdf + Stacked TTFT bar per config showing per-stage time: + Combined: just TTFT (single segment, no stage decomposition). + PD-sep: proxy_recv -> prefill_sent (queue on P) + prefill_sent -> prefill_done (prefill compute on P) + prefill_done -> decode_sent (proxy hop) + decode_sent -> first_token (KV pull + decode wait on D) + + C5: figures/fig_c5_cudagraph_ablation.pdf (only if eager runs exist) + Cuda-graph on vs off, per config. Captures the "PD-sep would benefit + from D-only graphs" claim quantitatively. + +Usage: + .venv/bin/python analysis/pd_sep_paper_section/scripts/plot_pd_matrix.py +""" +import argparse +import json +import re +import statistics +from dataclasses import dataclass, field +from pathlib import Path + +import matplotlib +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np + +CONFIG_ORDER = ["combined-ca", "combined-rr", "pdsep-4p4d", "pdsep-6p2d"] +CONFIG_COLOR = { + "combined-ca": "#2ca02c", + "combined-rr": "#7f7f7f", + "pdsep-4p4d": "#ff7f0e", + "pdsep-6p2d": "#d62728", +} + +# Engine log line: timestamps interleaved with usage metrics. +# Example: +# (APIServer pid=...) INFO 05-22 18:29:55 [loggers.py:259] Engine 000: +# Avg prompt throughput: ... Running: N reqs, Waiting: M reqs, +# GPU KV cache usage: Z%, Prefix cache hit rate: P% +KV_LOG_RE = re.compile( + r"INFO (\d{2}-\d{2} \d{2}:\d{2}:\d{2}).*Engine 000: " + r".*Running: (\d+) reqs, Waiting: (\d+) reqs, " + r"GPU KV cache usage: ([\d.]+)%" +) + + +@dataclass +class Run: + tag: str + config: str + mode: str # cudagraph | eager + seed: int + summary: dict + breakdown: list = field(default_factory=list) + kv_series: dict = field(default_factory=dict) # inst_idx -> [(t_sec, usage%), ...] + + @property + def is_pdsep(self) -> bool: + return self.config.startswith("pdsep") + + @property + def is_combined(self) -> bool: + return self.config.startswith("combined") + + +def parse_tag(name: str) -> tuple[str, str, int] | None: + """combined-ca_cudagraph_seed1 -> ("combined-ca", "cudagraph", 1)""" + m = re.match(r"(combined-ca|combined-rr|pdsep-4p4d|pdsep-6p2d)_(cudagraph|eager)_seed(\d+)", name) + if not m: + return None + return m.group(1), m.group(2), int(m.group(3)) + + +def mine_kv_series(log_path: Path, start_epoch: float | None = None) -> list[tuple[float, float]]: + """Return (seconds_since_first_log, kv_usage_percent) pairs.""" + out: list[tuple[float, float]] = [] + first_t: float | None = None + for line in log_path.read_text(errors="ignore").splitlines(): + m = KV_LOG_RE.search(line) + if not m: + continue + # "05-22 18:29:55" -> seconds since first log of this file + # We can't recover absolute epoch without a year, but relative is enough. + ts_str = m.group(1) + # parse MM-DD HH:MM:SS into a comparable scalar (mins since 0) + try: + mm, dd = map(int, ts_str.split(" ")[0].split("-")) + hh, mi, ss = map(int, ts_str.split(" ")[1].split(":")) + except ValueError: + continue + t_abs = ((mm - 1) * 31 + (dd - 1)) * 86400 + hh * 3600 + mi * 60 + ss + if first_t is None: + first_t = t_abs + usage = float(m.group(4)) + out.append((t_abs - first_t, usage)) + return out + + +def load_runs(matrix_dir: Path) -> list[Run]: + runs: list[Run] = [] + if not matrix_dir.exists(): + return runs + for run_dir in sorted(matrix_dir.iterdir()): + if not run_dir.is_dir(): + continue + parsed = parse_tag(run_dir.name) + if parsed is None: + continue + config, mode, seed = parsed + summary_p = run_dir / "metrics.summary.json" + if not summary_p.exists(): + continue # in-flight or failed + summary = json.loads(summary_p.read_text()) + breakdown_p = run_dir / "breakdown.json" + breakdown = json.loads(breakdown_p.read_text()) if breakdown_p.exists() else [] + kv_series: dict = {} + for log in sorted(run_dir.glob("vllm_inst_*.log")): + m = re.match(r"vllm_inst_(\d+)\.log", log.name) + if not m: + continue + kv_series[int(m.group(1))] = mine_kv_series(log) + runs.append(Run(tag=run_dir.name, config=config, mode=mode, seed=seed, + summary=summary, breakdown=breakdown, kv_series=kv_series)) + return runs + + +# ---------- C2: headline bars with error bars ---------- + +C2_METRICS = [ + ("TTFT p50 (s)", "ttft_stats_s", "p50"), + ("TTFT p90 (s)", "ttft_stats_s", "p90"), + ("TPOT p90 (s)", "tpot_stats_s", "p90"), + ("E2E p50 (s)", "latency_stats_s", "p50"), +] + + +def aggregate_seeds(runs: list[Run]) -> dict: + """Group by (config, mode); for each metric, return mean and stderr across seeds.""" + grouped: dict[tuple[str, str], list[Run]] = {} + for r in runs: + grouped.setdefault((r.config, r.mode), []).append(r) + out: dict[tuple[str, str], dict] = {} + for key, rs in grouped.items(): + agg: dict = {"n_seeds": len(rs)} + for label, family, percentile in C2_METRICS: + vals = [] + for r in rs: + v = r.summary.get(family, {}).get(percentile) + if v is not None: + vals.append(float(v)) + if not vals: + agg[label] = (float("nan"), 0.0) + elif len(vals) == 1: + agg[label] = (vals[0], 0.0) + else: + agg[label] = (statistics.mean(vals), statistics.stdev(vals) / (len(vals) ** 0.5)) + out[key] = agg + return out + + +def plot_c2(runs: list[Run], out_path: Path): + agg = aggregate_seeds(runs) + if not agg: + print("[C2] no runs available; skipped") + return + modes_present = sorted({k[1] for k in agg}) + configs_present = [c for c in CONFIG_ORDER if any(k[0] == c for k in agg)] + + fig, axes = plt.subplots(1, len(C2_METRICS), figsize=(3.0 * len(C2_METRICS), 3.6)) + if len(C2_METRICS) == 1: + axes = [axes] + + bar_w = 0.8 / max(1, len(modes_present)) + for ax, (label, _, _) in zip(axes, C2_METRICS): + for mi, mode in enumerate(modes_present): + xs, ys, errs, colors = [], [], [], [] + for ci, cfg in enumerate(configs_present): + if (cfg, mode) not in agg: + continue + mean, sem = agg[(cfg, mode)][label] + xs.append(ci + (mi - (len(modes_present) - 1) / 2) * bar_w) + ys.append(mean) + errs.append(sem) + colors.append(CONFIG_COLOR.get(cfg, "#444")) + ax.bar(xs, ys, width=bar_w * 0.9, color=colors, yerr=errs, + capsize=3, edgecolor="black", linewidth=0.5, + label=mode if mi >= 0 else None, + hatch=("" if mode == "cudagraph" else "//")) + ax.set_xticks(range(len(configs_present))) + labels_with_n = [ + f"{c}\n(N={agg[(c, modes_present[0])]['n_seeds']})" + for c in configs_present + ] + ax.set_xticklabels(labels_with_n, fontsize=8.5) + ax.set_title(label, fontsize=10) + ax.grid(True, axis="y", alpha=0.25) + ax.set_ylim(bottom=0) + + handles = [] + for mode in modes_present: + handles.append(plt.Rectangle((0, 0), 1, 1, fc="#aaa", + hatch=("" if mode == "cudagraph" else "//"), + edgecolor="black")) + if len(modes_present) > 1: + fig.legend(handles, modes_present, loc="upper right", fontsize=9, + bbox_to_anchor=(0.99, 0.99)) + + fig.suptitle( + "PD-sep vs Combined headline latency " + "(trace=w600_r0.0015_st30, 850 reqs; per-config N shown under labels)", + fontsize=10, y=1.02, + ) + fig.tight_layout() + fig.savefig(out_path, bbox_inches="tight") + plt.close(fig) + print(f"[C2] wrote {out_path}") + for key, v in sorted(agg.items()): + print(f" {key[0]:13s} {key[1]:10s} n={v['n_seeds']} " + + " ".join(f"{lbl.split(' ')[0].lower()}={v[lbl][0]:.3f}" for lbl, _, _ in C2_METRICS)) + + +# ---------- C3: KV cache utilization time-series ---------- + +def plot_c3(runs: list[Run], out_path: Path): + # Show seed=1 cudagraph runs only, one panel per config, all instances overlaid. + selected = [r for r in runs if r.mode == "cudagraph" and r.seed == 1 and r.kv_series] + if not selected: + print("[C3] no kv timeseries data; skipped") + return + selected.sort(key=lambda r: CONFIG_ORDER.index(r.config) if r.config in CONFIG_ORDER else 99) + + fig, axes = plt.subplots(1, len(selected), figsize=(4.2 * len(selected), 3.6), + sharey=True) + if len(selected) == 1: + axes = [axes] + + for ax, r in zip(axes, selected): + n_p, n_d = pd_split(r.config) + # First pass: P (or combined) lines under D lines so D is on top. + p_label_done = False; d_label_done = False + for inst_idx, series in sorted(r.kv_series.items()): + if not series: + continue + xs = [t for t, _ in series] + ys = [u for _, u in series] + if r.is_combined: + color = "#1f77b4"; lw = 0.9; zorder = 2 + label = "combined GPU" if not p_label_done else None + p_label_done = True + elif inst_idx < n_p: # P-instance in pdsep + color = "#ff7f0e"; lw = 1.4; zorder = 3 + label = f"P (inst 0..{n_p-1})" if not p_label_done else None + p_label_done = True + else: # D-instance + color = "#d62728"; lw = 1.4; zorder = 4 + label = f"D (inst {n_p}..{n_p+n_d-1})" if not d_label_done else None + d_label_done = True + ax.plot(xs, ys, color=color, lw=lw, alpha=0.85, zorder=zorder, label=label) + ax.axhline(90, color="#888", ls=":", lw=1) + ax.text(ax.get_xlim()[1] * 0.98, 92, "memory wall (90%)", + fontsize=8, color="#666", ha="right") + # peak summary in title + peaks = [max(u for _, u in s) if s else 0 for s in r.kv_series.values()] + peak_summary = f"peaks {min(peaks):.0f}..{max(peaks):.0f}%" + ax.set_title(f"{r.config}\n{peak_summary}", fontsize=10) + ax.set_xlabel("time since first engine log (s)") + ax.set_ylim(0, 105) + ax.grid(True, alpha=0.25) + ax.legend(loc="lower right", fontsize=8, framealpha=0.9) + axes[0].set_ylabel("GPU KV cache usage (%)") + fig.suptitle( + "KV cache utilization: PD-sep concentrates pressure on whichever side has fewer GPUs", + fontsize=10, y=1.02, + ) + fig.tight_layout() + fig.savefig(out_path, bbox_inches="tight") + plt.close(fig) + print(f"[C3] wrote {out_path}") + for r in selected: + peaks = [max(u for _, u in s) if s else 0 for s in r.kv_series.values()] + print(f" {r.config:13s} peak KV per inst: {[f'{p:.0f}%' for p in peaks]}") + + +def pd_split(config: str) -> tuple[int, int]: + """Return (N_P, N_D) for the given config name. Combined = (8, 0).""" + if config == "pdsep-4p4d": + return 4, 4 + if config == "pdsep-6p2d": + return 6, 2 + return 8, 0 # combined: all instances do both + + +# ---------- C4: TTFT stacked breakdown ---------- + +def stages_for_record(rec: dict, is_pdsep: bool) -> dict | None: + t0 = rec.get("t_proxy_recv") + t_ft = rec.get("t_first_token") + if t0 is None or t_ft is None: + return None + if not is_pdsep: + return {"TTFT (combined)": t_ft - t0} + t_ps = rec.get("t_prefill_sent") + t_pd = rec.get("t_prefill_done") + t_ds = rec.get("t_decode_sent") + if any(x is None for x in (t_ps, t_pd, t_ds)): + return None + return { + "proxy->P queue": max(0.0, t_ps - t0), + "P prefill compute": max(0.0, t_pd - t_ps), + "P->D handoff": max(0.0, t_ds - t_pd), + "D wait + first tok": max(0.0, t_ft - t_ds), + } + + +def plot_c4(runs: list[Run], out_path: Path): + """Stacked TTFT for each (config, seed=1). Combined gets a single-segment bar + so the comparison against PD-sep is direct, even though Combined has no + stage decomposition.""" + # Pool all cudagraph seeds that have breakdown data, then compute per-stage p50. + by_config: dict[str, list[Run]] = {} + for r in runs: + if r.mode != "cudagraph" or not r.breakdown: + continue + by_config.setdefault(r.config, []).append(r) + if not by_config: + print("[C4] no breakdown data; skipped") + return + + bars = [] + for config in CONFIG_ORDER: + if config not in by_config: + continue + per_stage: dict[str, list[float]] = {} + is_pdsep = config.startswith("pdsep") + for r in by_config[config]: + for rec in r.breakdown: + s = stages_for_record(rec, is_pdsep) + if not s: + continue + for k, v in s.items(): + per_stage.setdefault(k, []).append(v) + p50 = {k: float(np.median(v)) for k, v in per_stage.items() if v} + bars.append((config, p50, len(by_config[config]))) + + fig, ax = plt.subplots(figsize=(7.0, 4.2)) + width = 0.55 + stage_colors = { + "TTFT (combined)": "#2ca02c", + "proxy->P queue": "#1f77b4", + "P prefill compute": "#9467bd", + "P->D handoff": "#8c564b", + "D wait + first tok": "#d62728", + } + # consistent stage order + stage_order = ["TTFT (combined)", "proxy->P queue", "P prefill compute", + "P->D handoff", "D wait + first tok"] + x = list(range(len(bars))) + legend_seen: set[str] = set() + for i, (config, stages, n_seeds) in enumerate(bars): + bottom = 0.0 + for stage in stage_order: + if stage not in stages: + continue + val = stages[stage] + color = stage_colors.get(stage, "#444") + label = stage if stage not in legend_seen else None + legend_seen.add(stage) + ax.bar(i, val, bottom=bottom, width=width, color=color, + edgecolor="white", linewidth=0.5, label=label) + if val > 1.0: + ax.text(i, bottom + val / 2, f"{val:.2f}s", + ha="center", va="center", fontsize=9, color="white", + fontweight="bold") + bottom += val + ax.text(i, bottom + 1.5, f"TTFT p50\n{bottom:.2f}s", + ha="center", va="bottom", fontsize=9, color="#222", + fontweight="bold") + + ax.set_xticks(x) + ax.set_xticklabels([f"{b[0]}\n(N={b[2]})" for b in bars], + rotation=0, ha="center", fontsize=9) + ax.set_ylabel("TTFT p50 (s, stacked)") + ax.set_title( + "TTFT decomposition: PD-sep's TTFT is dominated by P-side prefill queueing", + fontsize=10, + ) + if legend_seen: + ax.legend(loc="upper left", fontsize=8, frameon=False) + ax.grid(True, axis="y", alpha=0.25) + fig.tight_layout() + fig.savefig(out_path, bbox_inches="tight") + plt.close(fig) + print(f"[C4] wrote {out_path}") + for config, stages, n_seeds in bars: + tot = sum(stages.values()) + print(f" {config:13s} (N={n_seeds}) TTFT p50 = {tot:.3f}s " + + " ".join(f"{k}={v:.3f}" for k, v in stages.items())) + + +# ---------- C5: cuda-graph ablation 2x2 ---------- + +def plot_c5(runs: list[Run], out_path: Path): + modes = {r.mode for r in runs} + if "eager" not in modes: + print("[C5] skipped (no --with-eager runs)") + return + agg = aggregate_seeds(runs) + configs_present = [c for c in CONFIG_ORDER if any(k[0] == c for k in agg)] + + metrics = [("TTFT p50 (s)", "ttft_stats_s", "p50"), + ("TPOT p90 (s)", "tpot_stats_s", "p90")] + fig, axes = plt.subplots(1, len(metrics), figsize=(3.5 * len(metrics), 3.6)) + if len(metrics) == 1: + axes = [axes] + + for ax, (label, _, _) in zip(axes, metrics): + xs = np.arange(len(configs_present)) + w = 0.38 + for offset, mode in zip([-w / 2, w / 2], ["eager", "cudagraph"]): + ys, errs = [], [] + for cfg in configs_present: + k = (cfg, mode) + if k in agg: + m, e = agg[k][label] + else: + m, e = float("nan"), 0.0 + ys.append(m); errs.append(e) + ax.bar(xs + offset, ys, w, yerr=errs, capsize=3, + label=mode, edgecolor="black", linewidth=0.5) + ax.set_xticks(xs) + ax.set_xticklabels(configs_present, rotation=20, ha="right", fontsize=8.5) + ax.set_title(label, fontsize=10) + ax.legend(fontsize=8) + ax.grid(True, axis="y", alpha=0.25) + + fig.suptitle("Cuda-graph ablation: PD-sep's D-only graphs are the structural advantage that --enforce-eager suppressed", + fontsize=10, y=1.02) + fig.tight_layout() + fig.savefig(out_path, bbox_inches="tight") + plt.close(fig) + print(f"[C5] wrote {out_path}") + + +# ---------- entrypoint ---------- + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--matrix-dir", default="outputs/pd_matrix") + ap.add_argument("--outdir", default="analysis/pd_sep_paper_section/figures") + args = ap.parse_args() + + matrix_dir = Path(args.matrix_dir) + outdir = Path(args.outdir) + outdir.mkdir(parents=True, exist_ok=True) + + runs = load_runs(matrix_dir) + print(f"loaded {len(runs)} completed runs from {matrix_dir}") + for r in runs: + print(f" - {r.tag}") + + if not runs: + print("no runs yet; nothing to plot.") + return + + plot_c2(runs, outdir / "fig_c2_pdsep_vs_combined.pdf") + plot_c3(runs, outdir / "fig_c3_kv_timeseries.pdf") + plot_c4(runs, outdir / "fig_c4_ttft_stacked.pdf") + plot_c5(runs, outdir / "fig_c5_cudagraph_ablation.pdf") + + +if __name__ == "__main__": + main() diff --git a/analysis/pd_sep_paper_section/system_analysis.md b/analysis/pd_sep_paper_section/system_analysis.md index 1a03059..422814b 100644 --- a/analysis/pd_sep_paper_section/system_analysis.md +++ b/analysis/pd_sep_paper_section/system_analysis.md @@ -160,6 +160,97 @@ The empirical KV occupancy on the 6P+2D run was 97 % (`analysis/pd_separation_analysis.md` §3.3) — the model and the measurement agree to within the resolution of the steady-state assumption. +## Layer 5b: empirical refinement — the bottleneck side depends on the P:D split + +The model above predicts D-side saturation. The 6P+2D run in +`analysis/pd_separation_analysis.md` §3.3 is consistent with it. But the +new 4P+4D run (`outputs/pd_matrix/pdsep-4p4d_cudagraph_seed1/`, captured +during this section's experiment matrix) tells a richer story. + +Empirical numbers (combined-ca vs both PD-sep splits, same trace, all +cudagraph, `figures/fig_c2_pdsep_vs_combined.pdf`): + +| metric | combined-ca (N=3) | pdsep-4p4d (N=1) | pdsep-6p2d (N=1) | +|---|---|---|---| +| success | 99.5 % | **52 %** (444/850) | **68 %** (574/850) | +| TTFT p50 | 0.91 s | **62.8 s** (69×) | **51.1 s** (56×) | +| TTFT p90 | 12.7 s | **491 s** (39×) | **400 s** (31×) | +| TPOT p90 | 0.027 s | 0.013 s (-52 %) | 0.020 s (-26 %) | +| E2E p50 | 2.5 s | **65.1 s** (26×) | **53.4 s** (21×) | +| wall clock | 944 s | 7558 s (8×) | 3693 s (3.9×) | + +The per-stage TTFT decomposition (`figures/fig_c4_ttft_stacked.pdf`) +shows that for both PD-sep splits **>97 % of TTFT is P-side prefill +compute** (65.6 s / 66.2 s in 4p4d; 43.1 s / 44.3 s in 6p2d). D-side +wait + first token is at most 1.2 s in either config. + +The KV-utilization time-series (`figures/fig_c3_kv_timeseries.pdf`) +tells the full story: + +- **combined-ca**: 8 GPUs oscillate 0–98 %, peaks bursty and short +- **pdsep-4p4d**: P-instances (orange) pinned at 85–100 % the entire + 2-hour run; D-instances (red) bounce between 10 % and 50 % — only + P side hits the wall +- **pdsep-6p2d**: **both** sides pinned near 100 % the entire run + (per-instance peaks 99–100 % across all 8). P-side fills because D + back-pressures (D can't free KV slots fast enough → P can't + hand off → P-side KV accumulates). + +This refines Layer 5: PD separation hits a memory wall on whichever +side has fewer GPUs, and at extreme splits it co-saturates both sides +through D-back-pressure. + +### Why P-side fills in 4P+4D + +Two effects combine on P: + +1. **Compute concentration.** Combined spreads prefill across 8 GPUs; + 4P+4D over 4. Per-P-GPU prefill load is 2× the per-Combined-GPU load. + With chunked prefill, multiple in-flight prefills compete for the + scheduler. + +2. **KV residency on P.** Mooncake does block-by-block transfer *after* + the full prefill completes. Until D pulls and acknowledges every + block, the completed-but-not-yet-transferred KV sits in P's pool — + on top of all the partially-prefilled in-flight KV. Many concurrent + 33–132 k contexts overwhelm a single 28 GB pool. + +This is the same memory-wall mechanism Layer 5 described, but on the +*prefill* side. The Layer 5 analytical model in +`figures/fig_kv_memory_wall.pdf` accounted only for *decode-side* KV +demand. The full model is: + +``` +P-side occupancy = (in_flight_prefills × KV_per_req) / (N_P × pool) +D-side occupancy = (concurrent_decode × KV_per_req) / (N_D × pool) +``` + +Whichever side hits the wall first becomes the back-pressure source. +4P+4D's P-side fills first because 4 GPUs are doing 8 GPUs' worth of +prefill. 6P+2D's D-side hits the wall (4× concentration), which then +back-pressures P (no slots to hand off into) until P also fills. In +either case, *some* side blocks — which is why PD separation regresses +across the P:D ratios we tested. + +### Updated falsifiable condition + +The condition for PD separation to *not* regress is now two-sided: + +``` +max( + in_flight_prefills × KV_per_req / (N_P × pool), + concurrent_decode × KV_per_req / (N_D × pool) +) < 1 +``` + +For chatbot workloads (KV/req ≈ 200 MB), this holds easily on either +side. For agentic with KV/req ≈ 3.3 GB on average and 10–13 GB at the +tail, both terms cross 1 well below the chosen N_P or N_D. + +Followups: re-render `fig_kv_memory_wall.pdf` to show both P and D +curves once the 6P+2D run lands, with the empirical P-side and D-side +peaks marked. + ## Layer 6: the DistServe / Splitwise assumption that silently breaks To formalize: the regime in which PD separation pays is bounded by both a