From d71a11109910f8b3cd194724926d701af57c6420 Mon Sep 17 00:00:00 2001 From: Gahow Wang Date: Mon, 25 May 2026 11:24:16 +0800 Subject: [PATCH] Paper section: PD-sep scaffold + drop --enforce-eager from launch scripts Adds analysis/pd_sep_paper_section/ as the home for the "PD separation is net negative under agentic workloads" paper section: plot scripts for C1 (workload chars), C6 (roofline), C7 (routing-vs-PD-sep lever), the C6/C7 PDFs already rendered, and a README mapping candidate claims to required figures plus open re-run items. Removes --enforce-eager from bench.sh and all active launch scripts so cuda graphs are captured -- the prior methodology suppressed one of PD-sep's structural advantages (D-node fixed-shape decode). Legacy scripts under scripts/legacy/ are intentionally untouched as historical records. Co-Authored-By: Claude Opus 4.7 --- analysis/pd_sep_paper_section/README.md | 87 +++++++ .../figures/fig_c6_roofline.pdf | Bin 0 -> 34681 bytes .../figures/fig_c7_routing_lever.pdf | Bin 0 -> 41123 bytes .../scripts/plot_roofline.py | 144 ++++++++++++ .../scripts/plot_routing_lever.py | 123 ++++++++++ .../scripts/plot_workload.py | 217 ++++++++++++++++++ scripts/bench.sh | 4 +- scripts/launch_elastic_p2p.sh | 2 +- scripts/launch_pd_mooncake.sh | 2 - scripts/launch_pd_separated.sh | 2 - scripts/launch_phase1_ps.sh | 4 +- 11 files changed, 576 insertions(+), 9 deletions(-) create mode 100644 analysis/pd_sep_paper_section/README.md create mode 100644 analysis/pd_sep_paper_section/figures/fig_c6_roofline.pdf create mode 100644 analysis/pd_sep_paper_section/figures/fig_c7_routing_lever.pdf create mode 100644 analysis/pd_sep_paper_section/scripts/plot_roofline.py create mode 100644 analysis/pd_sep_paper_section/scripts/plot_routing_lever.py create mode 100644 analysis/pd_sep_paper_section/scripts/plot_workload.py diff --git a/analysis/pd_sep_paper_section/README.md b/analysis/pd_sep_paper_section/README.md new file mode 100644 index 0000000..aeb7e45 --- /dev/null +++ b/analysis/pd_sep_paper_section/README.md @@ -0,0 +1,87 @@ +# Paper section: PD separation under agentic workloads + +This directory collects everything produced for the "PD-sep is net negative +on agentic workloads" paper section. It is one section of a larger paper, +not the whole paper. + +## Layout + +``` +analysis/pd_sep_paper_section/ +├── README.md # this file +├── scripts/ +│ ├── plot_workload.py # C1: input/output CDF + KV reuse decomposition +│ ├── plot_roofline.py # C6: prefill roofline at varying cache reuse +│ └── plot_routing_lever.py # C7: routing vs PD-sep as design levers +└── figures/ + ├── fig_c6_roofline.pdf # rendered locally (analytical, no trace needed) + ├── fig_c7_routing_lever.pdf # rendered locally (from REPORT.md §3.1) + └── (fig_c1a_io_cdf.pdf, # produced on dash0 when trace is available + fig_c1b_reuse.pdf) +``` + +## Candidate claims -> figures (status) + +| Claim | Figure | Status | +|---|---|---| +| C1: 98% prefill share + 91% intra-session KV reuse | `figures/fig_c1a_io_cdf.pdf`, `figures/fig_c1b_reuse.pdf` | **needs trace on dash0** | +| C2: PD-sep vs Combined headline numbers | (not yet) | **needs re-run without --enforce-eager on `traces/w600_r0.0015_st30.jsonl`** | +| C3: decode KV cache memory wall (time-series) | (not yet) | needs step-level vLLM telemetry during PD-sep run | +| C4: TTFT stacked breakdown (prefill / KV pull / decode wait) | (not yet) | needs per-request breakdown.json from PD-sep run | +| C5: cuda-graph ablation (eager vs cudagraph × Combined vs PD-sep) | (not yet) | needs the 2×2 matrix | +| C6: prefill stays compute-bound at 95% reuse | `figures/fig_c6_roofline.pdf` | **rendered** | +| C7: cache-aware routing is a larger lever than PD-sep | `figures/fig_c7_routing_lever.pdf` | **rendered** (legacy data, footer caveat) | + +## In-place edits made for this task + +These edits are in the repo, not in this directory, because they modify +existing launch scripts. `--enforce-eager` was removed so cuda graphs can be +captured — PD-sep's D-node is a particularly clean case for cuda-graph +benefit and the prior methodology suppressed it. + +| File | Lines | Change | +|---|---|---| +| `scripts/bench.sh` | 150, 161 | drop `--enforce-eager` (elastic + baseline modes) | +| `scripts/launch_pd_mooncake.sh` | 47, 64 | drop `--enforce-eager` (P and D instances) | +| `scripts/launch_pd_separated.sh` | 52, 68 | drop `--enforce-eager` (P and D instances) | +| `scripts/launch_phase1_ps.sh` | 32, 43 | drop `--enforce-eager` (C and PS instances) | +| `scripts/launch_elastic_p2p.sh` | 57 | drop `--enforce-eager` (kv_both instances) | + +`scripts/legacy/*.sh` are intentionally left as-is — they record the +configuration of past experiments. + +`REPORT.md` and `analysis/pd_separation_analysis.md` still describe the +old `--enforce-eager` setup. Update them once the new runs land. + +## Reproducing the figures + +From repo root: + +```bash +# C1 (needs sampled trace on dash0) +.venv/bin/python analysis/pd_sep_paper_section/scripts/plot_workload.py \ + --trace traces/w600_r0.0015_st30.jsonl + +# C6 (analytical, runs anywhere with matplotlib) +.venv/bin/python analysis/pd_sep_paper_section/scripts/plot_roofline.py + +# C7 (hardcoded REPORT.md §3.1 numbers; no inputs) +.venv/bin/python analysis/pd_sep_paper_section/scripts/plot_routing_lever.py +``` + +All three default `--outdir` to `analysis/pd_sep_paper_section/figures`. + +## Caveats / open items + +- **C7 uses legacy data**. The footer of `fig_c7_routing_lever.pdf` says so: + PD-sep numbers come from the random-sampled trace + `--enforce-eager`. Re-run + on `traces/w600_r0.0015_st30.jsonl` with cuda-graphs on before paper-grade + citation. The plotting code keeps the source numbers in a single `ROWS` + table (top of `plot_routing_lever.py`) for a one-line swap. +- **C2/C3/C4/C5 figures are not produced** because the experiments have not + been re-run. The 4h matrix proposed in the prior conversation turn + (Combined + RR, Combined + cache-aware, PD-sep 4P+4D, PD-sep 6P+2D, plus + eager-vs-cudagraph ablation, ×3 seeds) is the prerequisite. +- **C6 is analytical**, so it is independent of any re-run. The numbers + match `scripts/compute_roofline.py` (constants are duplicated; if one + changes, the other must change too). diff --git a/analysis/pd_sep_paper_section/figures/fig_c6_roofline.pdf b/analysis/pd_sep_paper_section/figures/fig_c6_roofline.pdf new file mode 100644 index 0000000000000000000000000000000000000000..74078f295191f03d438f37cb86a684187635492d GIT binary patch literal 34681 zcmb@u1yq$w7dI~5-AEi-`oN(}y1TnmLK-BLkdp3{PH92umQDo;DFq}HDU}d}|8r2^ zd*An-|94s6`W9=LInOiC%y0JWnZ0Mv%w|xRmSyMS;6Z1oS^`zJqH}@3p!=5h&_zT* zoZ1guZ9tq-7M>PP_w7KO>K1l39w2TYgCnW$@Y2j%D;`#MJ-NMt;#@z+P3;l`Csbz0r?cibu;`{y9{l1l!jVH(m zs8(7QfMVn83F1_61|UfOj-`Ib3Lule!QuL61E9E}=-q7qbid$pYT9_*_j0!a+5^Q8 z)URS=?O-8!-xsJ64E*s32y*a31bKm&n^yqDEy&FQ=7;cc^8%G|N&(FT%JKjS{sJrE z;&LDO#RdIP6a6pJKwtl%rLv8Sou@sB>v#RK4o*PpL7cKq07;~6tnOReKpWxV>271; zjP9GWY+`bsvI#G6)vTTq$y&Tr0sAiXk@*Rovqrv(RX&aiHU8NPiO#g)gelXO(e4sH1_9own$ zM4O!QDX-9baBJV3x=pcIJG)PD3HlZu>&TDD40a`QqZ=;p28agmUPhKptzg|B(VAzD zslrdv^Yk1&fdBH%?G*3P=feu*?8)=bcgT`u9d(Y^``bSBE4hfIS{*D=aGGIFURui* z4W`#dK)zZBoc+WlGG(--t)9L>FK3gek4KPW$)ciiq5QWqT#Dp zw7n?R)zugD?_DNqwO021rCc5rpfSqHS3Ru_oxZ}^${zIT3*;`vGHCjr%Uxg*H_AciycY-gj0culiLE11kH?8 zyZ77!6Kc$8iq((q3S8EE`h~6#a_wu|5;Yoak$=z?I@FiT{M5~9-!(zh8|#T&q^!y} zt6(D0=2e5|QZmh|V$fte{`;kEXd!pFafHFIm&MceWlpjtCR^6kfI>eBmF}O0ddy94vO?cXp4nslgweq;DP@+nzT`p-)g29C@r9?9VI7d4tLC2xg4 zt_@>n1^xQPn*g62r_xQ-Z1$Puqpo4WtNCp$0@q)aX6lY1;we0Hz|8)DVsBEkr0Dou zaf{-eM8h>g{^^l(7IUn>=rC@$5c8?XlpgPzY_W|=Xip+@YHmwhLvDI=uiMk9lCWE`yu=ZD?n4lN{kWmn0S@jguiig=nT+FRY!Be{U*6KW6KO?2K>XI zSze$+ko)$742?KL(5`da8g>*u9tkDyvpt0~oVx>1F4mn9Y3@&hi{Cnw!C#JeNzlH( zyl$YgmLczd59og^7Lb_$0v%+ZpAW`oT zB2l)Q$Gj{4nH(V4Q(cbyQ+v!`tRm}pZu!WS#jluZz#wlRrcABYdT;eB8>jct2^TOb zuc#(-SB+OX1Do19+DSQU@B7SAbKHMK)+Km8y_y@dLHI$??$Om?!~VmT<-o5S!>4(N zAI+NlG9e(fiP^&n zcsDaa;#+&-II{4QcU|<7e?wV>vJ_;bma&Hgfpr%xqWXqUjs#<5+#i0d7UH5{XEWYf%{tzdviJ z@dmyd%Snplt zuzmE#^9FC1f~`bA+ek}qpDTEne!E#Xzdb|Xg_1&jk-Lr4r4 zxs`0ui}%Zhtcr#~{L@)F-ZkUjF@D#Ys9M<}iJz;b6@$^P=b!?7VrvMI#W)&nG_-3s zt1P}QG(1HuxO}Jb5AsyqmDU5scea?s|H0u`=INp|OZH`!d3zgiNzn>KM2EWg(D1XVsqD~WBla%*1{Uyj%(4yhO5-zl`4gpI9-qJq#3%J z+0FwQ!qSHN=WF8>qgvoCeU8&tOwtLukr<8d_4Arfm`@9OsH&VakyNfWf9i$}oZ{qM zp|kJprwzR{%Js;aRK{p0)an{}#SCvz+)v=E#W?02zW(U2bDopKziCtP^@dHu&fL@C z=|@4D`?@NR_#YiFeQ6Vr$ zRp2}QD35!VBX7mzJmmFlQ_{st9XXMsP-=5BTae2KMK0Y{jWoahC$s&p* z$F#v20B<>|kfR}FDtfnG((arCIf_J7k5{ZRU5zc*@c5+v@L=Mne$|o%58smZlZYQ^ zj;k3Nk)A`lIC&PK@yLBR$VJ=}rG6P5Edk$c`5y+mDaB=kocOXtf7JH)R7%F1T>rT9 zA=Bsnb05+SpY{H|qfbXauw?DmU59xT=_K;JbspqUb@@9>it2x6rahUC1=F!aY;cy1 z)rKC}i9`f-TESb;UDDX%ayjmrnwgr!QoD5h{MhGP_jSi-|558h-o1w_Ch@B)aSK;2 zIrm8?yzd6i$0i)S9bD!W{rU4;Pxu0VC6{k&GB@erVPoChGLo(JylOG(ySoqD-((wP9rHY5-TGr-QTu)Sz1yXF zsELlkxdm*yo(5_~Wj$Rgxf8-U7IwX!8&R_Qi$OFxMk}^qTXg2qF(1jeY?ATb#!aTt z%Lyv`PQ31Ov74HFfwR=Yr9r;7(Ih#lKTZKZSXEUdvz0jJYUIj6^N83Mw?J4bDf`Zo z@f_Zupgg^q*C}PLFKA3O)Jtzs&!5s+Jb0mF95~x9+k(2T7SoL>>n+r;XBab!D$gxh zt(u?8iOwOamr;@)P3e_A#y5J*dL*sI`M94)TU4ZCS?w#TM0#@W8sw;wvo3JNaoT|* z`>-=Q07mB-s9DH6Ix=t5>z2MJ)Cw1uSJ_j-jGG#kuo@GEi|@>$^BF+Ar1i^+W}#E|uml&BfuvJQ|y)Jdr0F@n4dV~m?YYC;4^Dlx7R}V#L^^v=?^M`#57$lA~u-l z<%acEu+h?e^D6NS3U!1P+Z$WWiScN6%hwe2O+wVR3EWt6G-SNQKW}W`nbo^3lS_ig z!1;ti|E?FY17Z@RbVi<^F&Eh#4o0IBgD6EiRO4n=r)R;-a znd1Z^*O2#$6(u?|Vhq-ye4NPN16J?s-_AP6ohetJk_%*r?8=ma(B!Zv$UB`OtnT>p ze!#zM=g79sV#XRDuI!#Rde^%s9x5xTtTT$*cuy`WU6*8xR`M~WjtE+%(g9i^t#TOT zS-!Wd4T@GpC;39ov4OvWGTB?V9C%G~xcPZPWziN8RY-gs&JThkdX}d&t{;SY*p`gP z$`^|>*o^9i2x2>MN%NS><0~YUBjpNz-WeT5kFX|1d!wnGz9{RqIJLxwIqdG4FP3Av zbRZJ#CT}I{v2s`;b`dAF*^)g6-~ZP28Pag03YDnvlR3?2m3C*7x#vIM4P{G-W^Are zPE#Fq?ug2<&SJl36l%Y;AAJXxWD_PKBoNH)9iWBJRV>=|fL1~b!MgRx61Psmw)qj! z@w4#6o&-5!v9ySKJJ0#MPA@BiNHk@FHFUoE?28dt7IY&!NQjsU->s62PYZ7DmKoGR zn#}5-e&R#?(;s!L~1Y(9yk7Q1+ zXU44Lr8|47yS1L_3oX9%lRFbyX@7J@UUFW;GQR%|q{ceg|Lp6^J+Fk`4`eZXIrMdR zr&Pe$1stq(Zy#faQtimmcfLE@eSn23BWO!{a5&5H26L%zRLMmV!j+_w2{C9|^?Q{8 zr&qWnmsCJNZNS)F{*B9pJ|`jUUCP%x26*o>7Eg_8ErWx}D;o}ZxW+cgox3UMn3QD5 zim15k=22f?LPnP1EuOHz={)VmVnZiS4oVzD-W(LCIe&*Ori8UbPSJ0V%%#MD*(8&T z7{*F3(x&o!0-Km|XaM&(Eu7$uH38@2zJi88zvjOD0Ptmv>(vL_E_%;C3{kU~A3p8# z9v$bt`mJR5SFcV20xr`zbHw7s-d5_IkQ7nsRH|nuF!yJKREBnS(`Mo4-AyekidWn| z?BQg#l&lTCd*~WUtiU}%P-~?TR{-DfT;=`OT%q?*bnEbzu5|0$t)6XLljE9u-y26vs$BQ0V=7POc%vNE z%jXWw%<*(+ih|i{lJH0cIj4ttL$5q1h}uoDE?=|UE`0s=J+*`4;-VPtgEQ9KdRUi3 zvzC~z^+Xd^Ng*pcw!86(U&X&XE4X_^J9No9x)utr8<^is`y9Y23E-RuaDD-B_M+hD z{$wz)n^eS*W<_;WibgYe?o6_l-$pH6lc(?y-1>&ZeFwuzz^9NiW@ky*=M70(^wRX2 zZ#}rPDZl1N?`002RaOg7aH}uIh+3^$wEQ{Eg}l zQ{G`g-{veLt4l-0S?JE?b?i16D3aTqOGDqcLVV~AnX5Z17mVm3QQ8ySyxo*61qa*L!-1IbH$cz7|aY#&yAcLa9TeL1iVfQIo zIBD*}FbFFf@|6FgH;{#4xcA4wKUh4u}PC#U1#{*uQQ|cV=fn%sC=eOnIw^8TX zOaV1r@Nk3q|5$pi?HkaCMj(^_JcF8d{>R24$OYIn{>8q*8`Fl>4MvsrFKxy<6$mvSrXle`s!qE{WOBEah>>isjHT>wfkrmAclFL!V*SX{%)cAZHD3IP3UReW}2d% zEpFx~c~sq^!dzTRc()`@Tr>o4-*sjfjlY<;u~XPcJIQxGYAyVPT#*g(7D2hv8K-#j1}ezjx8G^%k$l%b?H@cTJB{K4&YY zm|1Lji15MWONhQ&wa6~6!-}q-64BV^7@{(nvv5_=3j?cOZL|a8ahe&!$T1S3>d3Q= zrPn_OB~1=^F?wRnsVZRveSa#i<>c-oI8roVUiOUSt}TC+5% z%d+HF@eITi{D`$@tk}N_&i&red3j4pMZ4vnG~##{KM z;hX52vT5kd;l|=(k*T9d1fy^bW6EPeY*I?rX;Jkw2{x7%;jg&)>VrmT^^wKNom8;L zzERX**6`foB_TtCf$8t2(%%DvU+@N)_<1#dVX`e{{4Y#4_72##AA<}S2av_nDk_-q zF=#QSF)VsE%p2>Y1ZNNtG3ZfV@}bxoVe295k>8q4p}>Rz3u+?$A5!t%U`m1T^6LF2 zRjDjYSi}S3$ns=47PpFfvOLWF+?%qZQbJSV zS$MraL^vVBUz{)jlF&2`akNSJxszUmfDh(sViWh)E6ibV7<|(O@qM#JZDFh(3xhsT zEB^n`hl?Nl@602bpoHHC!jwA6k(9f274)@ndM9M*ka^Zk37z?a3O1SdU6p?@vHd?#@qjjuqq*D4Wr4>or?cyGB@u{SVBX|Mf#d0jLnP1PwBTZ z@6E{gUpYI!X+k5Wwn~1-phf$!lpnPRS=z)%c{- zIcx9Djd!YU8D!Q;P419=mkwha+(;T(4rxpl>9MUsmUr==a0+R>Cppj-5~;KxftSd;Ji}mP89!NH;deuf z6n)M27f{Zya5UPV3x-_^-#QQqxi|A=qTf4i?(wR}qfiXe`XAOs<)5&RS2MF33{FGo zZQ6|@W5~Qv`#O#TT%yxQB|e=F`IAxYd_}>2Z}kQS9idx1|3gOz&kY7Rucq*uYzfrQ zU0U1lxdtnv4|0Z!Pd%>q;Dw3U>LlUcz@udPM$tUes5xa&1Di4(_;iTG<_)$b7#?UB zQ#bQDum99E_uy#CU+fioQGxiGGLRq#^{* zVZdsuiUEh4Lhbune{IvD;U?Gj=g-6k_?erWf0d$Qt0{JNK`2MUytc~v`wjN zZgjRqBqy7mq$ihmv(n`B8}@QZsTpiqmr0;$SLfVQ+VBK2$JS9z=FXJ0QaQDW{qq=V5rFcdnzn+Pfhk2Y9+iWeT=HvGSZ!e4^QN=u7A z4K4xLMGKwACCi6%MV5X9l>?fNgxxTZz|h+uVE!9SF-b};fZir`A{NGh+x=r9_iOQ; z){K=$i%bkVPbe0MW$vZ5DjR4~CGaFaezbGoTL1J&*lE(%_Pt?Ydeq01ttleQy~2Qn zFrv6WlZH1b)H3Uge_*9VbNFJ2)GM-K=COKq z{^*VWMGW`*d&(wGnw9?4BBs(h+uJB|QM6X+fU!yfZAY=!-Y_!NM+QM+@2GLaA8l`O z<=)V9l!}ZAVP@O9&z_nbE+i2&R6|GQ2Oo>(9VI7E>sT4S$k>FdD6{q7_q#q5#z_LEuP1cqap z6S1%24A+aYv#-eSDHz$a(N= zLr-+FSkpjxuM2Cuou1V&Y!hJ`o{0g!##u6fH@C-az)4jx}Q}#4JkUJTL z5PnyzVtSg-vyC@$;FIDQ;6*|Xb~ zLLBDF3~S$lmhw|=1N<}<@W7z~Rm1jU@O)FrZFVFBSxQtgae6!%b;1g~2SF!mFFQ~8 z+~U8IoQC+5=3sLRdw5}6D9ZSBa3RZC~*ZQd~zyaDrmox7wRN- zFH$=Fj@|pHo}N2X*t;!X?ACN>>`j)-oUO`b6$huguvAo%P|u{%>|XNh5nn!<1QhXe z990@^JQxInp@{Mc{5!oMe2wa0buho)6fX~t4w#!;9n8b6A46H-C zDBe|8_L6V@>``k{rfR|g&G0k z3osvhzU)f@d1x1Z8;;}N+!jb=%8lMXUS#tYe2D!SgD;kbXzP^4FN80&aQDGJJh{8X zmC5ocl}y_<1L!sy@vMgBUe3qIW*y$1R4?w}dq`$Ua1hXp@355OW~o;-6y6lI*-d(*{67&TdPb|(Pb`7t1V`pQn)sm zy|Y$L9rUCcJI!gF2o)r1Jz0<9Ls5x+x4nBp;K0JE*+yyj^r07i>N4pTDSo=$BeGTN zt230#2t!&J6ojEl11C@Zo&7~Mm0bG8p-TLlMgg?mfO$oJ4$I*u(U^U4S)kj z5?Vegnc%yH4L6zA7>Seya2Oe8K8S3Kj%5&gxs<*~^U}nVqh5eg?g)eXDT{RZ__Nmq zr;=%5C<&?u1sJ`7f|u)@K|OPISj`_qxh3l8H<={|Tr|kC@Fnmpwi1cTi`DZCdRH~} zKA&(S-Mb%Cf?-AR4XqJL(pml${3r2S(z_|z)d_KS^aNc*%XXF&=3B3y!_|;!OQ-qo z4n|f?l*hD7zYeN|R<;KqshrX*_dM+A{ zd$qfx7K(pVkvM3>u92;O^@>~UX)Lo}AzK}JR=X^u#{w5`_M~qJf&8a?J1$R|BWDVVu9-yx!5J;a;2sL_C}}wx3R|?fgf*nw ziB3zaoC+7Ca1gf!zb9nK*0lugmPRKTF$_d7v~DiQ4HP?-cnNG^L5r~eAS_QDBqVSZ89pMr^&B%zW=on3HP<|_7DX(x&yiJp zkXy<6)&J`#2x0``*X^klO8&3ox5a;eUVSTd9B7?N{NNoU-_C;_Aya$f5W zC4+& zgZhp`LAIhyk4_Zoux3O^0i0E!0(VRKW-1MVDyA0&wqbE^v!i9-n&E4a*d>>9Sz8;% z$5`>!l1rpPk)Y4;&q>)*`1lC84JHu2U=3p#n$K0-)`$*aO_@dWEYyAVZ0vo*=Bzir z6#LQp^1#C%_PXijVXu|G>-(C8iDgt|dhaDk`h8FlrX8a~FXk3zXMN%S>XS?GkGrFq z2S0~TPAO|~#xIO>U{DZ-f-QIhCY6cp!RrR0B3@}rprNC`Q)5g9w3Jv}?f~;G?7hcG z7%v~lQX{1@niA=wC|WJ4$BTD-aAzN0Lc(!ndh~8eJ11@0$01Qchk2yf$^20Hq=6tzHFsjAWo9(Q`$(o+RAB=6MtK}V&k*xkf(!n}8f>-FDcOBaX?zwf;+ z886fr9~lxH!Y#-pYJKiMgS~LzXDr{kpE6EEF?yvQ%x1Ifj?N%qH0864w~9kq6T%u| zQRm1J`o)HO=ITcTvGIQ3iRBX zor-jJ&M0m{u)*%#kqkzZov&AJr3IR7%fvqU$ttd4Abv6Q*1KngYgpcucoR9U!&Rg| z(^W}e0ABn_#~P{M0mJ+zr(R8D)&ZJVt2JkPZ|=LzEy-k5pXyuKrq+Gcl||)gUv@-7 zj_qXJWn*Td1%zml4BrTbti8#nGF&(2uU~Vft8dX;-J|}H!)bi{nM7&#@vDFK4i5}n zm+L0Rb(w@gHzZ(@w*L&C0UU(>y=w3)&gaY8@Nd%+ex#C^mTJ9q#(HiRo_+C6>=3L+ za%6i0Ke(;h-|B;ywh=HBqbgq)|M2c7d-iSPtz}J7p)scrE&f>2dZjpBfY9^7T(By z;fIdYO3`=wcd5LEj$kwvhMvlMg8>8Kg8*w|aJ@E|n@a5n)mrH%&22wl*{hDfra z4Ohz*AJ&95pLcrs&&P^%rTA=Hgb%=h!$;%RFL!(db%k#u5z5@j7}nXYhp!aYdX?!I zb?>v4%|`RIFmnIx2@}BzYqz2pFKsUtSLMmn?;6Wz-Ucawa@aQL^Hm-cPVYyY`$(B& zv#a;8Poh9-Oj>gNUqA~7C`_?|sgE4Jl=a_fzAmwb`?dE4Ogr$rofb6#HvF8MEGlW|m1r9&pi1%;j5- z9ENjbLMzFT!Hv@M@QD`l^tMpt{jf#?;a zTfA2V?H?B9VYW4;jYSLGp=BkFtf|Rme}bQ|p*=ml$(+4YL?n9F<6U zJt+;ywJp8C{+_U&LA5~Bsx_PDPfDB?^+1ubK8x+d6@#zyRu;Yo#^DL-0c(&(w{*VC z;R#Nw0lqjn;RWG#R%M#LB|KJiSoH$Yn(i{QeskriH+>5Vnmd=lwidR`6XvGw05Z0P zM8llV?gr>rE`jjm7GVOr4bjr?A9N!q(_dN#IZcWg?SA?s!{;3?urn1D9h5yz4=14| zP$|}Iael9Db1U->lYroCZ6Q}Dmg=UG=89g+;AFnU$|*#|&L>VCJ4+q)+cG>TV!2!P zvDk~rZVrT(3T-}Cif1|2%$mgv-oYMO$|ta- zx^G=z^!(*HKTbp76`UEFKMX@UFiHHw0L;yGgApvM-V%!lY{;5~k?VK+eW|b`j#etY z`X+yNO#7NxMw5j%Gl`kgD}}*8Yegs^?G^Te@Q*Imd<-lyoXK?zCsXK$fl-@;C(Ccs z8l-sA?FAoVKvgI?K8IQ#@QhHaquy7e%nwgeGjVaMGnOz`#QpmZOY~dzu!Qaxr7{+qyaZh)PEZ%m0 zLX7=hs`+&vCna~cYAyUT9K0mCEevQd^f7M84Ho<=>ch@J@13`y^Mbp#DcYz|qD7wn zd?Yo3mXvOcy0Y<5hlF%mAW==HAB7PnDHBz zdL9MNg+77UbjKHQTNVC~4T^jZ*5{|0gLg!^du$sz1i_V}6W9r=%>%WcriD}K5u#cX zzDky$f2NvYD2!)a%+P$Q%KniwJ#X7TAnulE|4QuA>%mcy7}*!><*)Ch3gCbVv|gvW zRVD_f5Fkt0$?5T6lnjmdN%Ho_3KnE`^tqZNj5a(jE~(aR$FYIihBtX9w9ca#5_hY2 z^-j$9AIQ%xOs~&O{P&RxURatE@bF*dmZ+K_b{`mX@hAF2P>`4Y%avr>?S#um)!X@$ zhkg&Oe6$UAy(i^EScw)ubLbK@nB>TZ^u6zo?5fZuzdSXU7$}}&KOyy5! z2}N$JScl{eMp8B>4{syvYZ}#iZNq)R)uA_Oz_hBbZgmy4K;-TBBij59wu(xU%hOT% z42MK;`sVD>p3A;>Ae?zW`1<(A|1=MVZp6iV6T4abC>GE>sr{@F48*C)=OSuP@*_O2 z92v@xFf+izObf)Ji7`^j=DQhI9~iXNs>SH(npzbEaVymM5!LLKF(Em{O7By5I&DOn zd+^hhpGC9}aEs?u6k9U2KC${?9>U!82Ob}t~H zddb>XDBK_Eo=xU(`g6aS-^6Um7yS00Y|jfr`{DuLVExRisq~x29LwGIK3wAvoE)}Ept_! zQvHN_UrzzOb603U?kYrEFx;?`Jk_!}$9|B8Kb3^=WFf3aZ2gOZ^-#VgrDDu6JV<*3 zdm%KjA-B3z_(<>~W%6V3q(Sv;{r4!g1PM6G1UC_lV2%p;xqXV9;rv8iaB?qoVn|+; zii#6Nza&^LL@8{dN`z~^;q@&PA~!)d0d+L@eDcu6JfbhN7&j=A`OzHv zfd|f9&)Om6;~+7TeM$(=Wn8b$#RUr9PVE4U?uDVua`W6^##Pa1s|GfkK8uDC3jljf zoewyCefw>$_=>4GDplLiJYSGYPP$e|bAqZywI zku2tg9cc;+rs~^C7QJ)MGMjmpI$!Wp+PlxQ?90qrDSSUOqe#v#rMCIesG#n}B3q5I z8T(W`e|n2T*)uaLM!y|vaGC}a%QOqQ%5^q215|&Tt^~&vg>1UcM~2F)iS)43XdtES z?PQ=v;HJqJYSfEBezXVVbBhSM$W}|gPBbxK@cA>Eg^bzgfPh)hn!T6zUcXJ zRe>@qSGWVuL(xjCGU7*QFSb)JThKb?xXq@RB;7-tM5$h86)Aj2URmBMN&Y2OC|#C* zgGolFHcNx%hen6~-mV41w?opguCW(UJILA>YM+c#4pc&Glem76YPSoA#weVc$}Aaj zeYund*OMKi-IrlRdK^_>vCuhb9zJ%0J|{5v6-zmU|1*q+!qAr?kQ*!j*)^5i`$3qC z9_S|dkW-WmksZ%g-;S%7Ny##@*yi83K88mr1{i>vie_Im#g2)GeWa1#PxB+E=bHN9P^?2s74b6H#**goOK?|Y{{W>q2 zXa2rg#eEg@^Xt)9V*MWMi0lEfQ$c1Md9`ux&yt88PBNUQYy9ezg`XAIQ8=l2m@=9f z+O-H#O`;4R*UaN-w!ssIwLm63KHQnsm2D*U=99l?>~ufwr@rD;n2WLO6S{=F3R;o; z;^8qy^tJdLDt8tHi%(fKp6<7G+{$vtc(bFMAPTmhJLT&$dmoqxG+bh8*}izdH0fSA zQ-Qi)Z>KahC0LW&k!E1;nN84M{CILeRZX>15AvBtkO-~9(onC*fiTk&Z4uv0OG7Km zkL>dNXzx(?;UyfKM}P;6Zh@iN-{dF?1j45UER4bM_@$-c|8-#K*OoQgSTGwMMnW0| zhZKet*xAyl*xc3<+1dca1!{#Z(=)ON5MZEzq1{4&<6r+O=K(?Cf&>VPN#Jx@~$(8soS!0_0C&UeelkBx}lXy++ z=!hBwqWYPfM6@%L-?oLyT@~|!^-!A#Z?}gk+zP#Sib7Qa0jY0^542fwYa)cu&wDty zp+>ngMm;KS&s}AzE-v^cYLGd)nygeZMX92?4c9-?uaIw|aF(B$p(ve{{yDH-@>_U! zl8Bfb*Kp7-UzBqp65|$C3B%YcLZP}r_;%Z@w>sPAy;Kp*F?tvcwY`yFG?GpFT=&w(eNbQ9U*0Ajd2dCO6eL$xkB{Xb z&>h*GYbwxh`jtQsyF1eAX%-u%l|5sbUtCo8RBl6w#lRqy;rFjN+B*ReFggN;4t+Cg z4WAzPmZXUi9g-DjacM}CraxhHXa;d?>|+SprmF#IsLn@MGI58D!e58$&L3mL*g@xo zp(fvCjdQ)H8PrE81svO>+(gj)XS;xz^&iV8F31f;l}x-6HX;pR0Ldwg>mH=ua^v0! z^SJsF^wfpGN;FJ6xFw`Ol*19>DG@ad*~`PGinKTx12JtsO^3ecA#f}5viim;)! z_h>|z_KRK5v&PhS`1~6_Hes*OhAZ*EXoky|x9ifp-OuE%ZQ`{<9MUZ5-6-C08xFbK zS5k&Cc=ocs_Tvy`Je58XU4e8rcmJzx^0bM)I$NWMX%5}-zK?TQXW0k!ku+~F3w(KU zQt*KNIdWu2X4(^UO5G^phZ2YRa2t@{Z25vFjGUl%Z8@ ztb#{oR7Ej4d@Jyqg?F8aP_Wk;`b=Abi4;j3lTP3&l(&O-u4)fobk5CDQ+^48p%wAL zPIiSkyUncN(W9-{o9Em{3^iv$;4atMALPnbMS*t6{72}B9} zm>StwK6k?N63%$faGlt4u3ZmyXR}Ea0mrFb{i>sl6e<%p4ki-LDvEF4`4C7n9IB4Q z<-32n;C%OZp1kU-v^7CIz3x9geAKIL-k&|^BF^`AZM8je{XC|2YwPR+!OYw!5(b0A z&@8!bvKOe5pnxp}7}-v8dP&M3L%O#_x}LS#?{`&8p?C#fY=B%vM5f&6IXtbmW8zq| zDb~9iKvrw71+Iph#a?kqEElrtr+W`BjNyCqa&y7IlpS6WDeT4>G)Bd@PN(kd2GN?t zpXq)zNd7k2B)lBo+GuMgTG$=6i`riYrx*0Ibud_|_sn#9Bb|l8iXh@B_!F%rpKt;0 z3+-L^XHOcIvsqnf@L2SUvyn6y% z7od;f^i?aoGk)1M9`jn4s>e9IF@+}cNHe*d=A)LFUnWOR>(boVJ)W(!Ms|;&=9lf9 z)k)n^wvnMd4jIjGSt4*!bFa%aq)4cyW=gc~)GZ;3J!x{mS+;r+oc@w#D+*+Jhm1{P zyh4V6*m;tvn70cs@~xzQSsv0V8Au}yMuyWBpqSg9J4h8(V|?vN#645VJ4%X1AVVv1 z7FYU~;Ikpm8*coW0lcbv2<^f@!K$}z5x9-QTjwdJG;F-OU;nKES?|XsT%MeE+CS+7 zoGSlIp8s;Wa_vWjQ_{l2=66a(Lql4ZRodpBg|3&Dg^LGNJi2>$O4(bu10QoLTl`J~ z=aJDlq2EZ`cv!hRxO(1q2XR41q9%w_%gfU9+Mx@y4nSQ{wQvU9qDcO4(Z9Zz{N)OT zoeRv(1!Cs`HU+r&fxTfaZa~ohJfJ|UP_giIcewVK!T|g44w=Q4jDp>s+**xK3vE?S^IS&Tr?<8mIZ7FISO2(YE(Y6G}x z0i3r$^#WI@9~UPZThHG~cLzIr;0>?}v%2r>Yysl5f8c6w;{xL0N9Xi#vhc76ar)V~ z-v@EtcLAE~=>w!Z?cHsFv@PHP2Kevb4WvDQmT-F5cmp4D+FX~&>EZx~C-6EhtJlucx;`2vn8iDx0;Qto^{r|Qc2%xp|0Q}7Z=-mPk0T3UA4_y$j+wu#5 zLEL~ug9w7J122FR`W~7e0^C=*jsfNc^8cR#`k(g@ATKmP%lH#`frE!!U?>3ucz^&D zBq#vl2e^+97$3Yq<^0e9m?pUedC`FYFex91mlr4m7_j_2Kwbd=02oL^W9WO}nie16 zB8eaP9$Ka#ApiITem#T6*NuVjqeEjT2xz0Afe&Cf=(oH;!+(_lh+#DuNqmG4u@-6@VQ2Ig}0lbl0C4Knv_={H+8YejpQYe-a8G$_M`i75ea+ znShd@U;U>PC@3%({rcVS0tER4L4N|25q|Liln?$pT+hlqdd#YhL-y z6HtEmdw}A){tc7?<&8f9+HyWHFVKeoRwxg^2!aAYpFsHn5M+N3e|X|g3?&NC@95Vt zH2t@5-Pga~L-YJ6K=}gNtI+tbJEwnFlOzDabUhaWGc!QMYe^>s;)1TeHGydjbbav- zNXdhMp1ppi3<9=&uTyFupp$_VIxuf@|CW*(AfUsq-)e!lujLkWCg=WLhXsiHcO8}h zU4P}X0$m^Y21@xQfPk*L{@xlY#I93-5c@S>{}wGk1Ajfg7A!#Xu7BAA>;g>zLINt1 zfKl-4F;q$c-E;lJ4j4VZQuY8J{Z9RscfYH300A@UbumsLz<_=InKM*AUZ()L_`AAm z@pio{4p8SlATEDpfJ#Y#Nv<=yf`IdM*G2s$bb*fk^$seAf5~28LHO&@Zz+AfRRR?1 z2IBo)zdHypyj&OL0kF!i+`oh^FkfAN3zfgXy!9)6EwzCo9lw5EYXR5h6rfB`fE}UF z(SHkZV1Qlcg39&l)hdw33t*XFPrX6FQKaklK2Z66oq7NQuCrX1aIF^b{i^gYr)_{# z|4SYFeJfHBkm&z*qggbz2df)|iEx$KOGl3kzpJGa`1ofR`_wT(p7q(9;+-ndvc{N^ zxIX%d9thVfKco&iddCMlj#3;JFL}w60Pea$PaDcA94TZ>gEl0&>;{obae$m!7m*s{V$WJjlC`Mf39vuadH2jZ*C4y--fYk z4NUs~M0JxBLD#6RFS#NQ{1>Xf-yK4rPYdJH#-9M+dSi~zL#RJLG=tj{w2h?LLxy`B zYLAWibUPRZsi6J-Kcs?cvH#kk>>7a49*nwpPG}5*AX{7JF+Ei!sXbZMr8V(YKShn2 z!OZqfGs%4!u=vcuObpR#alboQ5YvkdOTO(*E?n0a%S~S#+YZFZNYUF!b|PbyRUr?U z@*q7bomw<5-wCWsDq_a=2E!xe>+l&R9p0e#+u2tn@Kp6bL@Z*tV|xcLA~R+VXLZ^e zw@&RGF6{$jsJ?ovAf*JoRwb|fdL_e_1L4Sl4EI3dh{Df^Rve;U;m!f_`e$3L{sR>m z5bJ#A#3P<}Z%9U_%aRl+;Rg4TSe`pSZ1^<27#DG6v_}0SChzbkO0J!@7YvF)$H)Ir z>}K}#YvyX?f!@b>sSW0908TR3t3&T&aC0I3=RSsDf`zpU;_HMyq`7Q_S@^{8Pvaj( zpvTOrbO?B0pn(q1|3SkI%%K0;f4u4~Sig7XP|X7$Zngjh!o@Bu=5_SY$5cs2+l(f% z>FKT3tK2=S?0IcsHB|Hj zt#}-~?JSXtJqIrpTY>1;*-p8m9qe#1k}WJ9aY2sRuMVfYitnxm+h-Z_>E<<#u08AW zD0l7knNjr(i;(Zm$P!6m+%j*~MWPYt(YK3`sOWvK9jN-jAn&eyoDmG$3!s4glVrf8 z{;wocQ3tjlXrTL4X73d?P(Fxg>6ogxLwmA! z%eKH52kmO|LPYsUoLJ3k0u8T+cLq_oK)8<3*3cEJ=;1PsMUPHTxT7!Wn-49n_&v{u zY&$X-V?~mY4^TGJZOd)uwsU-GFh<3nP+NcjuTAeA*j`hQ8QGv&|Em3&0B~2eJVOGpl}YKl(=y@3&ZEu zOMkpLhbUuXui_VBO4ZreTN<(F8s)SU94K&Y)o;g%KDWDPNCz9f4e9#){+_*+rA;1i z8pYZ07EAvVbtF7&3<@)Qq=^t+f(PPX9~v}3DEL;h)E(2hKPJB5`VoewhC)_S1?P8+ z<|aBt+}+9~{Wzo*FBKg6Ze&>Yg-mGq!@Ki=E9?8z#!Q7Uwi?q}zTIcmX!yK8M7gJB`38%sYty}mYTs^i=--5LJASo^47T4PW_avU9 zRxy`1cUXM1)NGbpa{@g2S~>UQZC}#rA!OSQz=@G24nyF)q=hWG+n8U^e0A}Eq38eb z*6I_oLam)oO5iMxjV@;C4C2vjT-FJl@)#!vO@Im~1pmTK`D7f}DY9tKAG*42gNt zsZgJoz9$j+Z`dnnYNGZ;()GH-%qH;#edWkX$G$vm99y5lb?)F5;yf0b6=k~IlIyI= z`+Sb`hS}&l3>w1F?rwB3BW@UaG2<5WV#W}S)!j^Gs&`lXA&d{rBR)QRC2vpIL-_Qz zDbg!)T?c{uF?hAt4c>R0B2c6mB9O};XsQ`Ltv^zCTRz795@=>>d-)YX52tDW)%ZD* z@fsa743sdmJwS#3*E#aH4$8x?aeW+~>p5WU=lSnDP8h{(5g4U3C^a0v?D-OEIEXL+ zz|ib&;(JIYwha&PJp>&(sPmi2!R~@2!|(N1&q5LLxTu_nSH*w>e?7WDx}d>byiggsaNq{-vY_vSE~cW9+xK!l-60d|Lf zrIBc?Fg6sTk7^IzY4|rWowqbCSBCP19H+yw-gN(8dsiM#RoC{DA(5dFLdZOy(K+TR zGS3Y}$dp+e$y6dkrphcsl0qp{rYFgiS)vSOs3;YMlBvnN_EArr;rZ&lzW4k7dR>?6 z+HPyFwby;$>t6R>>-Sq*P2H(?MhUj#d5^KybE-@AM{t{-yV5g|dwJ4VQr>FQBf433b2|Tp>z769YI15ECOu!# zO>O57d3}l9Mc@*3*KUrs9$#ztB-dFDWvB2|ftq%26^qp=#>Hy1%E-fU8`-nMX=Hjp zCfkAk34K`8xN$4kHTfJw7ayv|5tQ>WFeLx;NqWPZ;IO%HLEC8cQbpc58s4z|UYb4I z>oYgs*9s{W$7Hg+rAjGfQ&iSbFdbT+vo?`SICD@oE#mqZv0v*!^ z)0Y_#g>1G}0ouM3{oZbu=@ZKQdL<~-8J%nM^1o_MTjfi{i&`h_e<*Xwn*MU7uVzjp z;*dIKYWuO#r>BCjL!~E#OE$TjuCA9^V18UlU-FG*2_fxu>PGKgTyC#TX>t%7sYSq$ zCY|)auE*~!V)6qMEz7K0@UqA`obTehH7=9;$~1Y7XQG^A)k^7YrnQY;Y!9CCF=^Rc zm*&WbU}+hDP{1sGTuLfRg<_0`DbsaqhHKYRZ*?>0J?PJol8d4Fqpm5^B`-~Ul{-hy zec@$N&|v1*!1@VBi*|0l@a%r2=C&O|BJn5l>IUap4mT}-xTe1 zTje!N`1upxHWcfbHIY#enKb9$#j|7G+r~v=OtGJ6=wkW4O zY8YGL|F{FjV!5?I6jOG#Rd)!N81T6ut933i80Oyl?CK2lr01jG;s)W;XP=!eT;)r?yDHY1R-|5iBvlVt7AO8wMZT`q2Vk-CG4OiCYpic&99 zXq@2v%InB{?yE-GrbVGzb$t`5dmEL;eR%iHYqQVn;H}OHLz)2i-m zGRy%m9X0mHtZf%x9FNFxHrhCJf$s5F2gKd0&F$TB>ZinS<%vs#%&ux4lz#i*QHF`Z zH8sa-V^+O>$75Hd=CmnN%W{s<47`Y%-TK8k!Y!nJgMzvBeKJ-hlc&J{gaW2y)WOiC zy5)-(+rtfsbH`^kt1GIn*|_hS|Df$FudT|gt9XF(YFINSr|!gj1&ZZ7&7vwyZQ%J1 zY+8AfdT#!a0jnw}x-*`)!kXJEYR}L;w|{gtYz)!UGt5xp!~Rh>`ZTO%%ZT1C^_0|# z{c*3xL&6ac=&lcV9361d|WO;)49UaEg2Rge7hB50^&Pg)IRrI^Q|$7o6S|= z+bm=&CU8N=(mT*2aG=6V;;)qX2TBOZPJV&Y76<{JyG(OSuDvon{qPYo-6NAbp}=D7 zZ@Y&!(<|r3!`ZQ6t51}{@h)EPebzh#Uq9aB>Mlr;V=1h6Yd)zUe>O$s({a8up*5P- zZD$4kx_SJHRZ{O{n?qEKPf=oN*L=XP%Xj^k-YnK;|Mi#RqO5gWw=#u%w`AEBQ8@gC zD)q(%sq@m?FE>Ufp*aJeF*GK0$^^%A1!jClY;16=h(96{F%ByT@f z(q?OLxry#cT6L!NiE*~gKozLX;w7j9p8vE|ij%ooMua}U`XpHsfp zO)q{yCDTBWl>Kc_sfgARsy%sYNkU4&SG(TIS3GPAp`+(46ms^PMOJU_vqVii(0`4^ zm5+1yhFOo##a1}cR}WHH4$ie-()Su?e4R==6>#a}jZjTf)s#4|nXIO#YoD9yACPef zGMOq^$p5WBe9z_)RaA+s0I(A&j6b1}kAfMQMlYOB>BPCgC3=Fp=E6Ewa;Ajg8JV`p zEMmkZ~0HZx7jM-~8_79UpMI^V6G`vknsGa|C;#_L`=t`;6!{=S| zUvoW6*-DSvir8(hxiHdX=`HLWbUCW_J<)MKsDL9@hcZ&R%}U~7 z&0Q~M%^zJe(YDG$@%G*f@~M^2eMe{5yk8(%arhSNbgBzysZ0C0H-#E{s^>eCSrfzc zCH8RYrS$BCzg7!Q+s7gC)Vb??%|WGErU98b@wITP&r{Ukz#&YK)hZeNkyW>qt3GVV!J6zN=Apm_T%O`zNiQdWdYSlDLejL zjwk6KI7G{F_34c?q2ntYheRgQ!-gl4j^rLj-*byC3lADQ;&P+7$~)fvr7d5F?@1YH};(fs%Z_X zs~1LM$xS)ra$m56_FF$iC+Z|PzybJmjnyhwP>bnRhv{|p$IDNb!iy~roQmTQYtXYc z*Eg$|%ulW`uYcEJZF=Ix)7=^3b0Q16`*Z`fZzk3p^ogp=85R!E-_@}bag4_1?3~hh z=etT1^#!SM+fvT>rLCyF)8BK{oZF(zYV_uAmmw;Q8v9BUJN$?>OY9D%8~yrU@=$h4 zj9hbW3&Jv-;(_|NUxoX%>*wE4qH2<6n4?3y+eey#f&*KZ-+_xJssb&Lz?!L#b>0q zdt69*Pv7bh>1&}T(>GJGWmlx9eQ{Br-Q(6u9;fs-A5A5lo`?JQh|>Gtk4npld{M5X zr&m;!_3)(V0Sy8B1_g{{-ZOgc!W}`+3*RQ-8Y`X}EeC#6KKYH(lRtifyv&Q)Q&{mRpYdeQVROY zi!m(aVqXpSIeqZetWA#EU1(l&dgp{xN!J{ez&%WM-M!1g_0kH;8FFk!Wo(JB9`t!6 z^0}l^1?*9g=b;|a^h%l+aMBBtK63Q<%;Bg`_|Y3@U|6H=Y6UBAx#A2&R+o4L?mykd zz=msQ9sD|>H8`ICDE!l@*m$v6EUG(3n4i}iy^B`$thW#>2t*{D|VCz zgYV~V+SfgkUVRbUp6#+Q8Bz7P`}H&>y7Q`Wh};{g0eYS%Uo)2r&r%OhkL+&ly-Qg- z%Y2b7iNDAOo2xXrXeG7B-Cp+nNinIJ878gZU@~hB08B>eBw*A2&z$53DE>O7Avt@K z%SFKk{BP?9sCd9-2lmd%JKJFNU_oCTP4BzrVT?Sgp>q8awFpI{l%9~ri-mid0NcHN zFMS;1oyPX0t3lCRN)A+4HdLr;_VMgD@n~mUjA7<@lk9dy=Rti=;3918tr)xTF{qAq{%Vb9~s#wN& zk4;YF z)D<}~HNEp6|YyEz-@HriQFr7G<+I)eJhosR9MOidWF4|&e&&sKvO zWmL?>98#=Re0~mA?#WjrBDi&ocCX+-RbpeN8hSK#TZDg}q?Qz$=BAsQQrKI+72%k@ z9CW1@QxYVc5)99BnbY5!^IlarOr_ay=TRT-rpXb3q>)=)h5b4T)mQrS9dU>h!$>_= z+_=^O_F`u^u1yxUO@+impknB_mTz#`w=#4xKx4- z4bW>4ObrfTiU8nEfb0ev7m%TT;?eMy`;o~0^bYdKZnB01ezK14M*e|G{|~U;vLNmI zr9VFc4Fv55=+6KBFMxLcN9uu|<-c|Pd%FCuzW%E(;xYWkKJhoA8$UDrh;eEVu?9a` z0dfuWR%Z(MBY@4Yeq|5%4&SfvbRPiV!Iy9VTnOk2&~*TkeOR!ec&tk!G8j1{`Dg3 zbDG@uhd2r`jqv~Q0gR6IEAVmZ@BLl}TCXqA0QEpf>-E(U(E2*XYW=+&{D}kq`Ti8^ z94R>Ld*2~^H8i*39|z1_ATz`_25I6;nWbR3Hs}2yoc)hVOlLfhxpzFEkH7W z@}6IdLT&vhss{fxJ^1_M;W8R(ppcBc9XNymv>~I0_~&BM0LlMLN#sA41j|%ED@6Tc zF_<*4KU5Nbfo~oCKNJSWS}zPPl$C$c2&94U-TbL0NS`D^NV6ouU)NC9^MUmbv&-nY zI6%}nK-qxQivLczA7EzT74_r0UNFche&|>Od~6(I2#NwKpSQag&fVoO-!Y^#940L% z$#(>Y^Y)dOk@5KZf6`t)&SHQo;p64tXAeGJ{_Abt4o-ZB?d;t^J3p!bMez-|czN!I zzWXk^TOI)rKQO?kfx{696ifn!6@kG-#C}oDaUVw~IzZro!{`7l=lc(U?xVpHuM^+j zGKf$J{qT8ymmv{wXwVbN;8?(&C6Pg+nXnuJ4ml;U3?gF?%OHjVfeics@d}7!NQgi| zC_~9YED1sx3;_)mLK$FMf>A*rgJaQ9bRv>LsW_1g12Hs-WO5LDhEN7W$wKiK`uSUb zFccOdr4Y){aME&U6ckR0%E?0X7$O;l#2zrT9ORmWIx1=k&GE|D>NY!)`x(R_%0j_ zX_9^*WJ!GzfrV&Ag!Lie5R->Uh9t2sfX0%xi-JLOFyXx@GS33W7V(${JNghuitt{* z5F{Qma6qagwj~@*YD*v^?JuAa68i>N>LRtdEC%9y5k3nbd=mQ$7?lvkjIbOA0VNbf zGBh+x6UdM-5GhGyz!QjMfHO!k_K`sO#Cib%oy1=dz)y+WMZnMy(~IC)z?UVVgo028 zG85wGgTgb(^T8oHF7dq>D5WE;4*_A1iDgKVn1cZG60v;|2swzrM_3;c1|?WTG6YHd z0|X zJtb@dkTgm3EDJubLQ)P*5+@O|7*aiBfYpib#UM#-1jwo+dcj~Jk|N>rVPT}cfhCRE z2rP!w9&*qs7Eyh2a1tGWC<)Oof3%D9v2$^E^r3^!FfelobX;HIG4=9-GA(=}tmEkf fCCk8G>)EX@&dvvi&uoz3pj0fKkdUU47Tv!9f0`VX literal 0 HcmV?d00001 diff --git a/analysis/pd_sep_paper_section/figures/fig_c7_routing_lever.pdf b/analysis/pd_sep_paper_section/figures/fig_c7_routing_lever.pdf new file mode 100644 index 0000000000000000000000000000000000000000..dc53974c339205e4a961d560a6de4c5ac8e4914e GIT binary patch literal 41123 zcmd432UHZ#(l;zYgcU)8fCODdK=LljE?Gcw&H@r7EjedNN*2i=89|ar5Xm`6&Oty0 zNrD6=szgELomuo=@BQ;W`h4g6&iM|f>7JeG?&_+V?y9c-4TG|{Bs-jg8^TaH2`#9F zz@adxoslI(P!P(ge%BriJ@(Xj>ETLEhggNZQ#t zLlKwT~Qx)xG z=i+D#j0dD2=wAVCVqtjG&K>9x2K@2BQ5+~(`wra$@pij%uDmn>GR@bNfQ^9t{S`U(4LYDbAOBu;yrB$Z=<<%O)=R- zmAWWIdhE;x7t2RC9ZKUWE~QZwgnF4y49#;VsG7^aV(ouGn~=;7*-=8F%?8m(Rz*c> zI^5GK1hzRNlDM4CAIK_C(NFpjyTAH=jV`vLBhxzw)#j;J^53hRmM(j=$K)bc)xIDoM z9n;&n*mRl(A8OI*U)ob662sRWsLAWZ+pVu_YSu)$Ia%Br`TAisE&9laX=!Qo(P4_- zk9Z%p5FIr3vzW9?is)7^cMZa?bzTlV*NVe;RVA*D%Qz5!?PbW+w!h+~YihK~Xonm3nvR%J_Y+}`>m31&M$MeNYYkatgN4f(8FXWE2*!DdD{k))q7zLUO(jRbKjh?y7VQlyw}-M3JtxG z@Z6;7bua4aG>a8v3F&1?k+Y8-pkWZmEr~iSWhC`XPcZG0tH_+`J9a0{&$0nJ#c(5g z??(d0k4L&j2&%4Ktw}3zRi=LRQ0krdhyBPk{M1JyDGQ8yX~|!&-D3)P`|Uua)~Y&S z71t)zazWYc8UNWY=HgB+&$Wxhw7w_3A~2xKe$yy%%SJe{h@9(6p>MUqMTdp~v-!En za8aE(obAu1)NS4aSNovYesn6Le64|G8A+LMVH7fA?pr307N^~qHpfeKu+Ey^h6Iv-Lt2TV?Cf7po6J~aiND{sUB`Z2fHqS)+Z9pi zUYf<;G49FpCfw;V%p*!-Yw3n;-q(B;o)2U*FGgd(_&QJ)!m-*i};Y`u{&q`JVM&6v~ zdrfg8@4^5sO=_3~PrLLQ@!`>{qN9xAGl)gS%A82adyxC_TNYXl$ZRE&c0C;Wk_{$r z@?^4t+wYi=JKPJyRz<6K6c5F#p1_+bzQ2-A)~}pbvRpO1JuCL%p7Xb?t0i*>kA$CH zk!EYrak~Eli+P$6Te_TFx%?wLfwMace|fL`i}#s3FS*R$M2EXHQIyee7Li$M3ycI1 zOm@j0xwkW{*Sytt30j;ljfWyK z&WyuBCJ7($%73IG52BwR7-t^t5f10R6S$yT#fD=ZT%NKT<*Kr<@ti1yiK)k%X@|Z? zk04>Ijr>}$ITL4?=&OzBbrV+tn{ezPuE3*w)2K)9lOM&mXix2O8;I1MhbLO)v>ei} zdS@S`%v>36^}RYJ9Zpp4LSwD_pvZ-W)VLO|9l1gJiSF}^uXN{Bk-CdoUM6wp_lc`Q z6;F)>+dqlxeo{ApYY_Q5}ynOuW}A& z3Dbl`1TL%kMwsf^u3^!o1`ti-=D!fZzThVwHY<5fX4P{Yd+lY|N}Kh#KvU28_s_Mh zBr{aPDwH{HOnY8^{G+@+H>BS5@MU<9V*R5pH~P)5EAFuFu%tiYjNHt^`^?^9c`%!r zlgMm!uFq!re5&k}ofiMR7L&~FoJshy)WNA{(h_V4H=hz6PlnE6U#R#7;T6}f4$X^Y zjuzb5Mg&cir`tq-EZM6vvICM=WWC`K z4)Y9RQX$nZu=EXOW(wR`Wb-%6UeXL29h_9GN#WX6`h z`1Vfx!dL4X9(R;B4m;}kV#}H+KX~m0vVZ+#j=IRU+5Mg0LzwxNX4$J%57dtqA5(u( z_#tmA^33Ua90n|f-Nl750<)!d0|&L5rrYzpcIDUYLYeEzZr0yvTaWBJSdbMhiZ8iR zcX7f0%hmKdk4=$$;R?iCj=OjJ>7zAdTqqd~OjNuYnJLXRg~=0{O7-*^M9JuD(@g8< zpBc0|YLqn71hO6o+jWG=gBIXVyXB`Xh%s_N!-tED3;xSM#8^k*k2+AjfBgnph5uzC zaUr<@v*-`zk%Eex?P)mv#2~YmFH_(h{H3JjywJA?c2-n*%RNK{9lUiY3a&4tt#86d zTM~3jFYP-?yel3ozJ6t7>?m=`BL1uoxkoD15R1`VW7x0{+b@uWg-65_j;)7@PC`NR)e7@sUtLV$XYpJIKPiJExibiaA6z@GBOa zPc5>UrazjHf2?+mcCM%LbjkWi8t3UX9fx&h|H+XZm|WxA`woxXncmCBDY$Jtu@d5) zAe1xhqU`9a%XP}6)?iDd(6CwQv&iQCVJ@5c8}Lz55rXD&)aG?To9;^Q8vixNQmGScN4iPw&3M8hwS&Aul>wg*T^ zs>o<{e;#=0kumB=S4tW~U71MdSkZj84;jHP{ptw?H_y~pWmEm-jPmcU={|rG8&txLt+!`=t7*Yks#jOcLA=F{03Uz7tJodx;S~a?`5?a{V zY?k1&`GRMjvIMjGvz|KzktWjkXa8KFeg0qChCsmoOaQy8oMSr-f8u~#_X%RrE`7;` zUvF#bo=!-;1U3PbVBAHnGP^sW_EC!SGq0851_`-or5Kiba&$>NjSi5D5 zERB34yxa4c8sk%x76bLqehD3yS8rK|ikDNl)9bC`u@)J5Uqi9P&7J8Q7jK=wHdi+C z?RgWTd`hT~To9$Jqszl`PF2LXbx520)b4HR(n@DJl}x%k)4H9wTfk$eUn@YH0RGIFCHsDEj+%-+WH|H=$X~ z5Y|e3VyqRR+|Aofy)t-BGrg>gPB31Ivt3^Wtnm!(0|xNe4eGyr;dgkt+Et5)O;{n# zc{B#6hz=$p_3iF6U>6|VolSzj!y)J!8tGs;b$r(;OHTTT*a8N$CTd|+Rde$VW! zm*ojE`Q1GXeC85`n7wQsBjV0=9P*KXNTRhK}wOjJ4?-#GnmxlCdr1JzmXe zaN;YIcbA8zGmqZnChXtz|K65NzkQvjyG^M;;EtU(_pW|7{q9*meHx4-`=8So7wD_~ zm%eau|AoFtLxGyh)sr`f?h9V<%3)=y2F)>Y?xh^*hpUog$7v_IDyjvuJBc4>FqBm9|GZC7&Q z6|;1T<%RdGTD?F)0*t@G;V`Wrsi zf2W>lz4bD55mC7Jse1W!PTjPdbcQxZ!(btAUhUW$#n4KWiQJ6zp4#Buh*mR?&-kMBju4IdRamvDVZxhdPQD>CnF(AmvR08w3GK?FB9V59&0STgrhcE}}Lb1*x63#Lv;+S4MW$|E+wmO-% zh_HzJ)QZiy@e}$6cZ&Y2Z{ELoDZ}f|y@cV5dpBa?&K z7@21nSX$t#8b=t2IHT=zuui4SlASw&oD*y-1OoPFAp!58tvK)wE&;2Z>K@ZllI5s` z=aG$#ejWB{Sf$KV3_JR32FC?PI+pZWS|xoQpZJN#eQpmrF9=9YM%2ew%vH4YwDYHZ z?0|3yeUu0`9SrlxxJj-7#nK3QAQbj&GL_9*N$SZ_Qb@UI$gn8dYezn)e-%nLAUOHp zW~DgY&OHzP6|04F7C9)>apoD;W?tXNmsCCF)+Md4ZpU3s4-oZu`GUz0J_>hgv?lXU z2;o^iV{2V*-h*CRIXtWP-aG9C zk(^+sA&`HeBFGrIQj;tA;!8z-=hOqdUT=?vMqE^*+|XvL+`K3(l@a6*Y1-pcm*kvg^RHYD)zP?s+M&tRB0_)J$PQd zBQ$ww{-$0kh+-&!@C{% z8x+x=*)G~K9%H2%xWHPrz`@rrPIQXm@%w&k%Dc~(OIs~N_MTH?wY1tEH0|xqaCdf+ z>2xzE+U1RSaDT{uvgF%ci`tZ=&TGaF^n7Zxyc&M32N%JhVI1_`r)PIh;K%nf-8Fu& z6XbM)p~?m9DF4wd5mE--62f;&??wHOZLQsp-EN`TizHEJd@m?fJ>Ju)KkU zREwjkeY1wQyPTy@6qx?W8(;?c$1KMMSk`~$O;m3<4qy~*6x}?>W^lOXVM;PzdU!|k z(PLUXM%X-A&X{Pd^($gli3W?CO@eHU7QBZQT#@^Uy0Qh)2bpG6Oy69GE3S`8uhkC@ z-F-A~8yqkhmWY(#PJT$~w5-G|!@!&UwaZ@dJ2Rx<{4=km&q-wwToJBMCc2-nUpFUS zHcL0FKl5g^_}w|#LW+dsh1mMivy4mFcBRXHtsjMyT^uah$MNEMearE5r|FcD`OT&L zmKkMJhA*LMPGt7g)ziJ2Erl1TU;4^2`NuqI3+!;~@vQ%WHS|FP`A;Une_x}H!zY-S z2E=7x??7=%;a@MoC|3A%GY z*g+iaWb9~R?`-D?h5wA8sJR$9VWS~IKaUUhw;Fm?A&k^ z6wb#3{DXtx4G5+m1w&^?3rrM*0|o;2Er~tAch+Ol&xKypxnR+&F|Wqqir!aE$mF7 z+`Ldu5458ll+(@@7?QIaP;xeRL<410AZ!ESG_`OA%1!{GoK9$0pe7P%+13Jh7!K?~ zaM}Z-0fH)=nieL`<^ZL~A*p}ItS}t>wPOEUOuz9M!;3#K)Y{O@35wu4=G*_sS}3Qe z6K2qSTzpVYup5vA>|6-I2^KT7mquHdnLC4W*ZNfiVp~Ae&B9pJ*323W$beJL8Etb5 zEGQVdV}=E&0XOXTz}GL+TL+4Q!T#S!kpHLcAOQJs0}eit3l8N)@Iq01C~>07rNcn8b^riU)!Ki2)K1p!_I6Gx>m>6J8+aMFMpZ0AXMbyeB-sB!}V! zhzB$aY!eQLA%FrLxB+y)g@k~k0jUOakP0NwIhasD!$2GnisS>H2XYh-z!t>uZ#+O_ z2;da}6AGvcY1@Y-hK&L2xI1F2UQ3vK26#~b{4b+6gQ9PI`z?goi4+J9-fTLh_OhWKs z#`dqKjtd}xU=uJtfHH9SAXNaBzcc~V0F2-OjR$j(UB3u9ZUF@j35DSws0sgSAAAaA zD)-L>Ttj%kR)7Q&2i^zoA%8Xi2nVeF>mR5M7)1b;;lC0b#)|~n0y_Sw8Nc#hn(-?? zR;^>r0JY*j63{p18qNg)HRM;qCprLezY?e%fC53l{NMYozq=}!NfiDwJ`c{TP&g(S{;%m6ARUZrA5W~9 zO*G&FoRNVy2op57hhnx$fM0*j>c9()sRCSs6FHy)$Cuzl52zL9;&{g9`ni07lQ`E; zB97;HuAfwZ(>^d?V(MU41B?msdz>AR@;_`6zP~U@ko_2!5O*kz*%o&b9^jpMJ+SN^ zP3i-@v{Md6epn$aML1VNVnc+4Z;ZA|4B+u_cF(_&*3x53M%?aD4kgX4l(+w=U;3kX zGJ1Q~j;{LS)0NYRDB8q-8U%or{)0l`HT$E{gg_!S0pAdtgfV{cmv0zkN?~Bikq~`R z92VR=)F$2Bg5Py2&mYD{==;vG`em)t`((#zuIk}apIk|3P9WtZzYtuH{0%ep5b*REfmE0tBVb@akmb2PY1@e~fD_wCBI_=H+#?n|+k^`aRx7}GCIaBLFh z@yGb!{o6bG_j=cKrG$IG8{U?}e`Ms+j3eZ%FGb+` zkrtmRQ_d`4=QPvhs3e`o)xM1Hh;%-+(Y;aDDsG0pNnyBTfckvMXKCPY6)zZDJL8^J zCQ(KE`njgFJnti;d^jXO^P5COtceJQ%cO;z-4D8<{tAbo^U8PX0I%=(A+4%r_A_-? zr|)>|X2I_im``j7e}3peae_ona0}tURPaZM2t^7+0G>LoQ@S5LmHCYugF4R{KTiwm z8F0r|%NdA`R}C|U@OvQwY4>N_eejOr)2M3C7KwE{F<@xDqLg8#%~JD)@bg-1&@)9c z!LAm38WU{Qnb)HD%x)Exml>wu-`I*Q6x>SOEvAsJyR0s6twIR+N^!r@4vT1LY>9m- z619|J%&_&>!Dk>GeYa0B%yf=^bkwd?xLB?(M#)V%n~%&Rmr?1)olw;N;Ti6U8&6Ld z`bnM^@Ur}Q=*j|)r>_vM@C8QPZ%W*Z>dqc>J3_Uj$jruRO#mY>xJfC@#}set)jgz~ zO?g8sx5e_q#f~?ZbvT_2OE%!MD2E_QZE`QBUYgcrR}LM#7v@)#uI`RINI7=2>x|q& zr5$MselV-P7{;GIE4qkv2fJ)n^27S;G3ERqT<)S$(Vf>1T))}h)aMQ!tqrqN4&BM9 z`PloU5W&`6ZIfD|J^#u*NFC)FWWMA5_;O^8fT*N+M#2;?$63Q#BmCS3iU*lO$u$o1 zz?p?!Gwu|>>ylKV_d+~yh;r_(-<{&6I=aIaOyYfnEjOfz{ZFO=_TE1x69lkm{*h@& zRlXu!IN{0+v%4==;S{hfeYq)dvgzC>QHYoyIwW*bMa#UNZHN{2Y0r32L34IPO+cG> zPf#OhM{-=UQ*>Q&spFt~E`>T|@ynrjTmSb)q0a*JS`Htw5yei*bTc129+sN4 zhorY&nnOo=v_RQ5#GhVKchE3;V5=CPBa)Fm!X#HM^!Ek)2_iZntO1xEf0j^K3Ajm# z3vN=v_M&$~GRfVKP}jK5-Se2j(V%cG@ITe5<;C2;J@Rxt<~|p#1cw5>=~_&AA*Vt) zoq+uf1|jDr{T1?!6DT?%n1T44P^(akoFfh#UwqdI!ttKv%gBK#@1pxj7G-qUB+szkX-F$ILQFYaykUug^K7Uzq8%}~jzy+pCZMc+$5%cJV+4*$+z zw&^vd8X{;=@8E*69^vWdSEP>Az@?B>u=W|%a(;v#KFa20p4 z{MqvPg#ud3C$m#BT!IMQ<%=hCQI*J<5}sVUi(zxKB=|&#S|W%%LWKHm{yz45&2 zN%vf0=x}5jk7@riZHa0r?dOVVBICm@CK-L%<{e%X>Klh^l>xZv+t#w3gQ6d{^dB|0 zm?^D(+t>cN@xbNLjBr%RG`1@9~x(43Uwl zJ8QWgvb(UQB*F850Z&SOsH9`*{D-re@L(HtxAX4}TsXfZWkR1=_g+(bXW?Z zSCWz-Dg3A=m(StJlN9)*V>suO-P(PQt*G*PzlO>}wQ1UwPt`aVW;vD>$i}}vZy3_p z&{mc=vngo_cQs4={>|l%u)M0-ySM1+2d9!^_(@6`hE5~?b7Q-l zXcza46QpuN%!BK%tjw5h8+uVfi4D4TC4>8Ox8wwcU|z+WP5UJiH}JEy=4PJdXJn-A zn`J!AOBOp?^SGV9-`G>h72jASsHR<{+bqelIaIgK+VF@bwb?A!XhCg*wRBo0tB_uJ zb+9#bP|`Ri{E_1wS*nfR0hLq+9XYH>1d}e|iGs zC7qXBU#;5>rxl7(XX6><;Wu)2X7h}@{JjcM(x-T#IEW00l$c7}t%ik^?^G{e^YLdm zdp~zJlc_Z@d`%@5!qZ2UqNiAA<Xhlj6IgwMJr6i= zf7E{@ID3lV<9I4H<72hZ?*Ox>$q`Br67IqktnrwLEAPCH$*MDato%#+BLbBx=h%%x zGG^oiKTEKgH55yY+|XHL^tjPE_Vp0!4h7}q6X-g@ibwpF=^WGRfZ6-Z$3ABw`CT^O z43R{e`r`%G{JeOwuJ8LS0SMXnUa?4L^Kyt1jRnob7sT8(%d55T&yo-!1X~B2&&bj3 z<{SvzX!V%st5@drb-p!G9MPdQY2o@%gE8%l#sj_*s=bwqsZ?fa&Cy;q7126k;SrV` zd-L>5#{17xOGjb|mZ=`n;J#J$k}SMirk)$-iegb62~8Ql>1f~cF&bI(+{?*5{NB|U z(Vg}mzK7b0y!av%c9}2l`()pPw(ADm=Dh}Ls8!xuF|qz^sWIO0_2<{vUS-ND#QCG7 zucLAbi`?Xp!%X75HR)=+fv|mBF^ZeURt~agVuBD)6?{wsE7_DC8xl{j4#GymDIQVm;50>e) z8~Ox^oaD^>O^^#%)ocN0Mtt|iX-cZRZ`(_P5t7>Sz^axt(cQvg^5pun_2l7ewe_P6 z!LmDyz*MR3fcSPd^9o#H@|?4MkN4VjKVhsQ|0Dsia#bT`JH5hlX z;*Qunf@FnJ%jHKEm0=t>-qFJf*4x<5p6c%>i?XU8?hu@B_Vy!UA+vlv784z19$I)O zHJ4zD?99%Ep~GRuB4Zt+5mefj3l@Z;ZmrS$M2ODoIIkCjDssyd;*;j&i|Wu0g{0<> z-dNn zKOXov`^Qe%TUY!kQeZCeMQ(dzca7H$D8)B@SWXbhN!|?iU--I0k#!`kFyPdLqAy8b z6StCM4f((i8v_YN`bK(j>@9BEXU$9=PgY*4Ep$YRpMKrOJTU#VOTzY{(c4H#jkznR zbM4`%XJY{_&?VJsrYt@>%M>rgWPOvo6z&(-b4Op8Pu*M!g6~xCG^ko6uqG=?oeqy9 zv9MM#?=U$<#B;CiHGluN&HTutlHB@*?bGG++|nnE?Id^QZ(_VcDl(3^R|xAiA>Ic0 zpC8^;%6;ni@%s^V(!l&R-!$j_EPc>Lv=x7Cx1>~ zNfMme)N1>4Q?b^qrY+k2>csl3mX{X?Qp$s^jtKBd*zuWEZAO!ya~r=YCK&fUZ-jdx zH&VJd<~_M(iJq%xE0tR6y5M7Vzqdw{xyI<~4y!JYM0aE06iOf@?NjI~yqo`=$fD`4 z_E?18b)V6eW#SeU<*2Ln&{CR@y6@WhRQg(TTS8z>pKTm(Raz8Z5)@9>o+)Nn2^s3D z#=+APd~F1m*v-?6mZnTjPswtgzqNSBzr(HN95oNa@N31DNvVzfA2a(xjhzRSBUvUV zxPicC+CMxRZrESg%fM?KF%Rg>E?*$W(EaVQ$D8U|@X-7NyQNo2pCq8|@z)=2^;H^P z+__+^Ju~7#bg;9MU9sx4EuAtkt-1EJu+U@dE!{2uk2Py9=&9^Fwn|}T#UmO0gu#8c zcqV2iA8EV<9V$1Ss#7}|b8?A(^^a@LN?*3&nOe75r>3~_Q8uUQ8^3N$&4G-*CawH1 zE%r7#$$6c*B~)O2lOkT{6WKVnm(@f)nGoBR^kruOGMw*dIL$+d`O*LZ?tciE3)s^B<8m&f437akkCj76+9~|v!?cE)nyb5jlRuC>cELX??0Ud6 z=(k>$q@1tD^iGf`UxkhQN38;Uio5m%UsD9g$!FaNIUnUM*+<^juSoEwwt5rBFgt3` zflZqC-4$V;8>KwJnfqf7;SpI>|Cpscmj8Yjjh=eDtGz#S<%4%&chhVfnkf`Z-+un~ zJd8EL<@7siGP55hl*=(oSW(?3f*kScfqL_`3sq%RCiXpfF7&b8P{N6~VY-xSu-9hq z1;q?h)ErV%4Brobc<^Lo>HFXwO_Lqp1;wHh7<7Vb2JEx{kv|CVD5NqBt^wmhslZU! zMq*@v@iX1P$sxCK44ek_B&c|Ef|?`3gL;j$B*Fl8f8a5#AWf&ihg8qnUt zb;^Gd@F5r4>uA3&ynQ8r%(a}DHsoyk!0dU#JMJXPs7BZ#9_ps1u=o!(#r;#vSr(&` z$%+Hp?*8TSIm^3eqI@V%FaiPJ^dD;TH~uPus|q}0ggQ+sx!3_&G{pmt7zI(BH|3y| z2RBUdv#$sDQr&ERG;fbNJu_5v&iMqHoM3_?fEd9ag(4JF<8TU&FX5yiN-!t6UD)~I zCBT$3&mZICofnLVl}s7^l1evU^2uNG@zvJ~!Ug4rGpuVp?_^l`-d#aHHR`w*s6MRh zajPWSR4283FV0L*u|pzIHK^$>Dv3#ufT%UynUOVv{NlI5E^79zs6aLA`Jti@A!7~Q z=dN|lJy3Q}N8b&qEU4uEtY_&|?9A9!;O6pObY#z8lPq3Dc$>*eP;hjSYxLpS%|aWaV#$GLLQ(!%VENyM$dE;(cF! zCgZWBN{5PKmTCIV`i1Jnw=sNTo8i)*+P79{zPz!zKb-|ny5eR$* zBR|f#KHR5Cyvzo7a4Bae3F>{VWO(~w!x@g-`Z`Iax+~57rakAERp_Gd6dSw~1!ZCK zuU{x1`S0@HP$ZXqa*Qe7#FdTsh(Oa#oUdBzmTcL2a5ZX`C9W zwq;4MbH5Lr{buCm5|=%9k8OMGc-=NttzI>iYN+)N@`_scwESblvqlrAyj&=nJ1fwt z(&(ma`sU0Bq|b~1p<>wy3_8gm1_G6T)LUe@v7f*MmeM2gxdLOFS#EdjW5RTAAHpFAAMo{3Qfv9Gf2euDttND|M}GksO)f? zVz;7$Kz2dfZk${zL`8vs=c}v@IfAJ7EeRr1Pv6>q?qRHrY?_@(;B!%xHm#9_f?S3P z<0dX_JvyN}v!R>1`gfTq>lWK*aT-QR&z0+Cc1X(@N{Do3+~aV`rc=E$Q2sVOQfX#z zV}vNFXooP{Uj5+(g`Ou9rSJM6t|``nEp1bMQ+Vz*w3F6VVku&NMOkqQSxNmHa#=+= z>>r6rNtT*~RZwC6WPIZh)}os90ym0u=6<+aPEWz5R(*t!BUf~LK7GbtnygX(+T>_^ zhN(QvGWFn@9h1)wtPitxC!B3Zo?!I?5xPIJogF-t2t6-$kkUf`(wlIVOoF-VhfTGJ zv9NZj0zJeldrM@*w*Z1O_4y$S+0(?+9~++q`J%5?U`O<^EzhLRsSqXY7cH*hE%tu3 z5&S~GdnTLzV(STNaf0FcS6fQJ$+HO1C0uG&;^_Ll( zHoS`QHqthYzHfFFo#p&&C2!h3y@+zqIRBhz>&@)w3$hfuHiTsQ&o{8?jLIfcHAvt27nPF4yZSf_R5~Vr<3@WR3keC z&R*}&^IuDv=L%oo8wmF(+L-d-Sg18ozZLwW`^GJyv7G~AE8M#i|C9>q1hel-P1&mph9 z70rw|b-C^<_vwP#h%siSI6(&G8T>`V+6yory)0TuhD?Z?Z(Gm?2g{`K*H`oO3#_cT zlnd8XsH}TFy&Ti*v>Nk>8fmO5l>=q}_nstdYgPVyg_J;DLPz3#%yIsb^LTpE~ zoliZ#6K0%8NAHTZ2+X-ypMO}uzdz2p~sO+^&Xr#qJL%1Vp&+AuS zmp+N75K7#gFbQi+j<$WK-7ysbY#PN$)>7K7BKQYQ-*pURw=B~Z4^&!t6K=@Ibjz;> zWtE!sO&{R|kLuOUzBmX~^g6;mdS@4M0_#t(1mUQ^VSSVw5n!u>^F-u%*t}C(UOvau z-I+Hg*|tV!=I2EdGyL=D@uIUnpxKQ+Qtwq7)CAgpat;XOcBuGR+fQICv4poE_d;@} z^P8#!u}AXPj0hb+tL94o$>N;#?nKI#sl5k*^u|4WM~bslK9<<;0wR66uWUt_c#Bx# zc}rb+%l4{MWtPR81-qLZDyQj^nW|h?`fXM8$Nr01+w`3uSh1&h8BZAQ2{tAIICtt%;THwOijy&sz5;Qr=N2^wuV4dQ)nzC~wxuGxB;d zG_EW3F1ORTe5kLgZRlO)t%7&}daf*MX7)IgmukY1R+n*L4tKqr)25ioEZ^WGv)Fmf zj{@cN6si3v89Tq_MmE`!_hHKfKS3ZTnB7S5xZocTC&Vh0PQvg5!2LY?xV<>9G?|7T zBgZ3~8}_B2UrOUAKkcqC2}zz*l1_a@$Jn&R^@G~&9!C|!Q-(_{iyUdp@hs`RD7*+(gNKbWN!`zHtlzOt0H!gGrE{`y?1>lkcG!cACiY zAgH&s+)&P{ILbx~9r2LDqteoI%wA=!#`dmd-wh|lIO-VNXD;oMxm+B!H^WMw3$W8o z7z=I?eBn`eIp|%Br#>A_s=GnkKy=;eoXE{gQc-7~#Pr*szc|-m0zEwNy<Xgjqy1SI$KEGKw`s$^Xo!+ES z-GAo<4LiYuK>Q8j3>hN>fV%J{c3mJG=ey-E4UDKflGy(-QNM9Ll!E)wEOB2d%K9nr z#-|r+7^eSFzZ{BO^)7R+%D2%*U+8tT^%O!rWl(ZtFHmtl?08FNHp%wTjwgCnTcMM3 z#C-|D=5f?MyBt-+5=w>k8TKMn*epd}j7KF^=*@1MOkdNy$ZENIK&SZskMfw}wBw--x2CMdp1j2%{PA_d@5{IqCHId#D*0>r}nB zD`4mEa=oTWSS+s7CMv%t-xSM0NoQ_SY(;WK{~aT z-!8tR^jh!A%^+63y`^GDV~45INEbui_teE)g@Xdn3*oFTZ^A$DJ6gx3jzJi*gJ{>3pS1PTQlsTxA^ zs18+dtHR)1nlPSH5KhX4f}cV-)fo8k=>~NeXz=O6w)yPQe7YBPElzQV5KW#y!U;w{ zaCYvGlNAEA-|n}Pivb`fx@{P3I!b@ zMdBJyZ<*i@WW<@R&ok&Qu>`#j-okBL1N3wM^dP$A-AeQ8NApac6z&ynp59m{USeeb z96fN__8(h62;gAn9~ldHE8shN9M$RmzP2e(j)vD9p)|@%&3(i2{FXR4$Bn+KCRoQ3 z#d7K;OWoG+Ntlj8T|lIcU0~Jy8ns)h(^bT)IKH@D-lVCeKr!{vLPjCaQ>EKEZ`xSH zmTXhmhKn$EoK#tHOCtQ81m12}%vcC#+lw;Nu80<<4k1{;GfQS&pQgxWPPwI%8Kcjd-g9X1 z3q6%WTB-83@_Mz`t+_WO_?ISCVAjjHB6_dzYT8}~V=2tXi(Im{?|UJ8m+KWYH_?!a zJ%|J$ji*1tv%?$cx?ofxx)PCSY%y&V@U5cz231UQ?9@@88q)_F@sIAOA2gp0^IJdH zD2Oh+RN3|Hl;G*z&m9MwIbAg|8xQ&+o1)Pqn`;6Wmsqy6qfWEp$zC|k|I+WCslmAa z(KkEc6*KQ6tkiE0bWRZ33GM_JaOD2aLR02*Y=aT5Y(u;Z%y2Ko>|HYBVn!X=;qXtl z;E9JoUoQBDOuD~{&W$Sh(q4iJqiOOVsN1<@dRel8Cxyppj1WffhJ1M<& z@ePvBR}l?A;9Y6?EiyvBW-X8+t@5<|uLXLko=DV+WQ}gUwp8MEo2wdetbrr?1M9*K zp$aC?M77JvD=48@vfdReU#$13g+Ck+4+Y7d0Gk00nEyki|H{=v0*;>YP=)#sk_)8) zL*ka6>zBWvN?7xLRTrP^T6*hgH3CE5Y^U=qPM9;FPAbHaC&=amvmT)6kD@_qc2HWa7Ru(D(A+dj%r2YqpqKMDtzpE zCeV^@o<{_>U~z~(h1%% za8COl0)zhz_**CvPSge-;nFX;EP`!q{XH!;lhy0Uj<{0>(q=AN_r4WsP@p~hD*L5X zr&PD)eZ9~0v3R%0%&P)!J#k^q;5GkLNPpf&XSD9}R;}V(v|M{kzR>9Y(+3OdL#&cj z7FdyKeB*UDQ~Z087f&bgGAn5ZD-Lgsn_N;3y_Vo$H7K@uXf~WMAWVhDp>P7@PqKc1 z!`pw0NKo`thPo9E33ad$~rdKY0K8Dz(TdlNpUhT3l>eC>Z z17oNhx5reDYxhbwzp5=F-i4`ST<=rgO-H*gnOW*y&x^z#ev9TN6B>9fuEZib6ZS># zi`X6eRT8Rl&(4g7$BmbAst{;j*4p!*KD={u+sL{rpk-dbG4ClnnY}8WW24{B*_s^F zb$$Blz(P5B?9))a!LIe!uB$N@e+UNk2HR$R#;cI`8G;y>E8pOmeSu!+Dz?eq{%U*6gT!7K~JdNrAf8i?iY zdqK^(Q8OY#q#II6IYw#B*QN(4#&(c#tzNIS&iRBy)Ht)4XhkTr*_Kx$-<-1W<#MlT z-o$$Y9$n}oxQqD;%71=c065U~Zw31|_z3zNModafQ%n5T?-(&9BWnu>7Xb70_b@R8 zIP(BU^G}9}{ZC!|2T}|O*s+6HRbV#&1lFK<|My5SAbbK)^1nlh0jlX_Z)gmBF7SWA ziE%nv8#0zNbd1+XlfF199UM<-)DM>LcVa6%0soJK%>C?|l^12Je&PT+HhK))6MLx6^@ zp`11VN&>(}fY38e2Pmf_Ks`VR&=8c<70T%beEtt0>@L7W9teO~Gj_HBDESYJ8t^$1 zP+I@SsA0tSw-~kmNtzg#8i2pSK-&HdQ#(Pf|KGsW5I~Usw_*LV7XE(#Qv*zPZUE!R z4JbASMT0;=_(0GXaO8{&SOyTlAwMwvf*>J)=MgA46q5sc6CecXe@KvD_YlCdU^<2o z9V69{fV?pnClug|YchcO(cL0}}w<;o?I=fCPZA03aC%Fhc=gG5|Kj z1HynnAR3s1_d$#o09EDz>Vs|a@dCIb-s9iE95WaMfO-Q41QG%c3QQ=#QUmKEfq@^l z0Xz%zizx#yE*F4D022~;SHPSPIL(X#F;f8E>G&BC`US^e{SbU$J#GNf1md7TXcI6N zFoA$F5U=%)7e&qnQkY8NFw7?A<_(1`9DUf_n z5B}9Y_!RgfMwx(?Fns-~9AN9fH3W0-X9Ik|9P}%Jig2t0pdS2Z!tff<3s5J1OBj6s zb>dgT=*mx>0QKSb1k#1M2HF61<5vQ81qB1Jf*>*y)PWNch=|1K1pvrH{g!^|#IGFC z1%N(o2qp*1za@;UkMD!e{40Ta0ZJ9j{{>G2?*DsflmPVbcmsR0r=E~x`L36>z7hEM=hiK$@(E+v?fF_as#BK=wt04OEqcK~?&vmapD z`Pq*t2u8-Vgjr0ue|BXC2=%y3p>&v-5zqt%GYx1TriKF)0JvgGjsQ20OHKfg z@@MJiLIZ#;G0%WdJ77;AC;fPuH*(xP{*aaz!Y{|!r=FSMHg)K--`jTvGH$t|N4H}U%3OnKIu|T zvd4QLL`&@auzA2vIVtohWYgNf4`3H*9r}k>{s*2Ct2rYAX=mb78^dE9}dvH*1 z5j$*Os7%3$QMnc!ES}ICXSQ(1>jt46tyOV*c51R*|s<&V$q~UHBoA zQ_QrHl)s#DcvKOj!ij@3Hg;{JS@PPAwy5>%nqEE2+`XR#wCCFgWoPFb9ze2LS|Ao@ z1|4r92-9eiuoDnpv`ud$Q3y~dE3`Sd&U5>VKCsZ0>iq;85&TF{bA!skI9I#KNP1;T zwW0Ct*L0R1sW($>Rf^-IvQwF6t1`N-`bstycc@&*c}_1?M@1F za9s8kUiSN!Lo(C{iC&N}`j}pG?@3eK$l|Ww5gplX@D!D@Wi(-_QkG;%E9CA7%h(S$ zXUy9;@=e=rX#8_(rJ&BU;Voy8TB=xb5vOgjwOgpx?q*z>a;{)|~84cor+yFxr@7Mk8J$ zH?Al-Jne0RvFK}y1mroJcfMbrM+&zfr?Wm&uU@)&rsTS_E&BG^BrQR!RDr&%z9k>U z2{$3C=TAoqc^t>z3i0=jD(l}q2pBLgKV*8#SqIm@97xWGf5KpYpR;}r7PKz^6u3Yj zd4P}G6_=kPeH!#Hi05yg|B>&z&n6~k5+71>B?@h17}UmvD!JctRSQpq+FF6AZ|nF| zNl40zr%#{>eCvMMjz4MQ0w(Z35`|QjD*~ctZqAE9ulGdKw-7~y>4ZVOm-1e4EUbR zgriu3ayc_U!#%rXom%E9CWU0%k)D^My-{WK52$4E_2WGiH@(0{yZG{4-B2sTk zojSU|Sw8Z~Z}Of_Z<4b=xp6P2xTf=JTyST{qwrmGW2FFg-34cAMYW5Op>?KuPkIfx zM(pa7YhJ1wM!#Oo+^4F~*eo;+exZSY2k<13QX8{T)Xu!=a=_N=o*Ei_g*WuT~B7t=S2e!$sYCwnsgRHenaI~ zH%v|#GC0%!*O0jYd*E+|44z;<9`e0e!N;=wnM*$=y4I7dqw-&VC3B!9GuwQCpH2I{ z(9l$wtUfqU?6rXaa9rC=MDaZ}hcmN%*1G^Fug9$^kmTy%PU{Nk!p95pSC(OnGS(ku zx=;h=jvrDd^VO&2tr|M!QXJjtdaT~Xw=@Lz?DKw5=PO{bibjznNm$V|OI}(n9?U8* z@6~3RYjVJ@j~~*YqsXT-PWom+RK_#A6YubdIH8(V;6_{wqL}* z+P5zc4fL3m@??~@cPW5shrwM`+nX(e*S59pSAa1(_Z(z z*S@do5-B=B{+ygT(L0zYSFdn^Uk70>Fn#)=Y}l>F$_SHbgADF+jMLx`qmn(eajGH# zy^Yn}{6pR+mUofMZFi^*)ly~Tqi}uHJ<{$IEqJBhSKR3kH^2JUG^)y@b>U)kC^a8k zUDS=`qCzUe?Wr{G*4li$w+*vCF1|vWHo&a;vXiLz#$fX~{!9)H)+71+TTvW3a|88v z%D$Lo@+NeCPc`Du3ssuaCn_Eaf1CT-$`|^E!^==3>oJyb(l#3F zZ*xaxk7MoNwy*Ex{38=2ABF7m7mmmF*en%utLz%PG_aJMCs@MDzpNB1TRgCH^;ZA) z#jndN)S-JX@RArEC_o{Y&VQ4x08%+Y>THJuIkR2ho}a*af>qaAU!O0IECPd)xb$Kh zdkUG$X61{UkL=upJIPW*TodrJK~xj;-%uVxNE8SgG%t_Ou5Y`3#}Z;}=tJRCF9!D=;`cCNQo)y!C+{?lbU z>9SEh>~?C5cJ8e_k<30(Z^oWIH;;Z}+F#X?!qp`qN+aoUB;8g}+&cIB#ph?2Ls9Qf zXqh?U^3HP0nkye39i=`PyxTf{`^cao&6q~*ag(6f(<&5-lQZ{P&eBZ2d3U1eE_F`A zGZ{gUQRnSfc(|HovZsgBU!6SAzEC90KdbgC5bY)~dA#9cSJObm%DXU1&1;3n zdA`2CK3QKc`fVr6s4E#D#J@S%BKmKP7|9@1}{b5v0hbdP-b=~&$}x|4B^Y~C76 zR9xwOdo&phd*5Uqnkta(?U?5_>G2?e%K}@+zxsY{EA6N3#IJ2H)l+U@cjni|2Y3(M zCk}#Tq$D6KL88ER9H2G&$8m#gl0Q%otF%1^nS1mUH5ap{w?uSZ7BQG3`+iZ=BzNV| zUE7d(Uv?yaX6lYwmo6NREqh!ujCM|7j$3td{P4GHY-cZh((JE`(0sP5o`H3h#AG5B zZGxATKUXSJ8^Gd08pO>svFY5QBXaE-I0)9n&{oAKY!dL1;srVE;54(+cyKI?8+A@> z6m_l)B{MDEo9Lob?rd6TTZgth`d&upsf@YO=~CSO8>zWdJF)LXaS|T3Hn)6IljvF_ zf(n_8JwQgh!i$(m+=rcxNBvpmH1Y**q==6)wM?ImIeE?#rgNuw$e@kKq^$t9^J*%9=*lfU3t=?7Sa3zv%YUrV(rfP;MAEA0D#zhypctVkky2P~a{6J~-JjV6f*X;`dr~6lh8PE5S%U|52xnj1NMCXwRJO9eXNq|rt ze2A+2&bnN)se7_Ypg!fwx78k83(FP-{?c-OD{EeIG2{X^=Ui2`}J{O{F(J$>E^JjL-TXO%NPv`XgHSCbr*-z@ju%MB+{2JRq*64bp&2oAV zG?C{cJ~^feq*m>{b;(iwM38d)UituiVLxz~=eSwlZ71t&RgK0Mj~KMI zqqZz6)7Y1aPqM9CYzOZI-?tv(F<>ImVI)FV6xfgW<7|`A!A60Y;OUtFdBouzZxkNg zzFmrWwRo{2D)CUnG*9yj7-yn)A}?=|UE>=Q8@SzjeXaUO-|lUN<<7t7+8q-@XBimj z(w#r(d^3Fes#k2g5%m;i;i|FuwpX=U-ACB$Mc+_aJ0w5BHamx!!HUB?3vTNh9Ymzk zU-l@ii-EfvWXv{)qdtyGMGD*unfmrd#0z_Y@iQGJjE8k(LCD7+b17~7%oO=fH^o)6 zz6CCo%29s!E*iFOJ*&y&;mgM!dUVA@hoE7ftK>@lcMTv=Yb&*#fFrAv?6Xg|y6P3(Q4^CoeD z!=>!lYnwx{jUHUB3(5DMUBa?f%nt@*sI|jhkB-m79E8Mhm7FFwE_gD>_U#zi!5nXY zk-gt$b&-1Iipg#gGmb>yjRxKN=N<&yr1c7G+@#N^ueYDU9qy|FwC_r|9HFXUWCP-S zOT>d=O*3ZKfaWsu!lSZT=R}X1B%TSX3<=|xFMS(7$x~|PAzCcOd2ogfbwfxwyDhgf zb>To<0CmKn=~TMf^VpR^G2h2MMGWO1;i5G&t+ACowBBS&l@qW@qT$GOq2{ zoBTYFqB?OZG?UJnW0vL~)d459C*+?LHYtzC7!*W=+w;L5vG&X94ZEzi zFw*VF6DaN6Jrg|4jjl&AKeN!pTBODg9(@1C*psP{^FjGD4?phMZWpb4pD#4c>8yCX zF=_Lxa@&%oFs!XOUCCIeTKwb*O8!Qepii1e(axkQRrF)aZA{(sbxjmIf4Eg`6Tf|O z_dTQC*Y8^?hh24amwQOrtuI!@i_PQ8mp3-S2Bt_VEwS;4EM`m0Ont^MfH zG$#^cO5)ApZzwNH5T$FJ!1@SrNgP5F19EaC^dNe78ovZU@BWwr@pMPF2(?Ivg`y3* z#*tTFKbvbCU0UO|LgvT*O?D6iu3j}e30@trW%{2zr`mezS1Eua7-3yf;4zP`x~-I` zG_u;$!We+1q`4Cn9hYE4w-tGrTSRUllE+>0$F2)$y_Lp3QO&(TTwu4p-U2RL)arm) z(607WrfIp68t7>-=5S&UVjbhHEwl-tO=Xf`2BzyF(-fgMp!ppHWOOG&7i3^yKW)3-K3@>89~W zGklWU`o832x0j^SzM1#g=f5l+H;lIk=+vAu@;(+IpOBO2H5qll=YzJ)?p7|^43x}1 z;rnNwd~)_%+W!i3W}|Y#ZvF z#R%i%Wh%PTGpgKFv3tx}ji@!O-|Ajdc=*9>U-Mh4tu6u=Uw0TJ$Gtf15UGAx@NEY3 znN>#nJrUVwI<0Ta(^-G#l_jnv*nlS<=)ZbLQvu5fAW(YqNZ-Xz?L&OT*L%58=za9M z*2}i{G!~BDPxZ5GT8bX<^YZn?=$zlJl%KUM?#8z}YeAic$}9F-<4Mi@u<^)jmRqmC z3>MF-X7#d~%uRM?wx#K{wKQRywx!-=t+;!(i?Mgw$5gFiDyEHxyOO#NFq3;l)tEsIWyUYf8)b*(O4VKbdNKX272>dTftQvH~5Q}%7xJ*p=5*OM5* z}w<+$t*{Dj`)gQtNdd3RNO)tG3 zze6^wxv+iQ9RAQnwm#8CQ>uYn;p)SI9X`|iuRe+$DGW-Qrtz$>5v!`b`TFC?f#@x# z3pO#D+td~0rDw-YjLG_cvsZCb4ebh%lIDprd5-lTc%H&#GH5DUJmADz{NPCcB=?78 zG1G6Kw`-1HenM(^eh<&r{pDZn1FA$cx&k^~rKwzUeuOVA0MmWl`g!F};Oq-UpB)G2 zt#oROOpmx)demJj+lOYjc#wUvVF_hj_0k9~FiJrm8JyRdvE)_Ho;y4A(yA~$%~VW! zH*;&$V4ec+IqAEN(WRy6A0Fo1<2+NaRUx4fXG=2C+)0H+9QT*5YrT4(l2We;37_OT z@%qSRGbW8U@}jprO0sev5-KcgjqDbAIMD+VtsXlnvWRW--b08adBVL0|DpcnLKR6(| zU5)-p3t0)h?>+ux{~&wc1h~_-dN`YMabhsLGG{VbHM3;bQ@sDeW zTq~B@7qNw^mnt3?e6VP#@PJL}5588{C(x;kD4!o_qVlgzp=HFlf92r-AF`oe?wKUhA}1RthjA>01kC4 zd1ssp?e3e8V>fSlV9!7z84RRP5HDCLC^zAcFP)K^Zj^8+z0;r9cX>!QS_7Zo$neYt z;}xI%C7PH6*%NfM^=9xB|(y`N6Lz!JvEGl~wnh%c_ljq6kKS?|peB!gU?NIp( zna%aPo|$2DY+TbrJqh*$2#`$2l zj)4Bb`U!aNG_^q3`Pe*Rmm7}XIyDiDls^u@5{5z)KGXX(wCT?dF^>-pdX4y5oI1m# z*1N4Kl4l2}zt;`Z)q(EChpVoEGVS--3pu6+PA7DGH6F8yVs$4Ac`KMba#GZ!hh6$z zG`evp>wDm{2Nv;_m$ar9_jk81tx#|Y+AWb9!{6hM|HjKd8vEy$|3Q>ELLY&VINUUh zlEjdgxYda8y7R*6`0zmrcsg0X(w-w#SrQ=$5^O^JdEg^tKwd9Ij*-f{z)KUiVFoIvIg!y;9_75KQzf#KIM>SPIX>YwehH;i)9-{R0HOv@19)mmopxz2rG-Wp`?G$L<>SJH$$?fN+mHtMmu_U4g^j{^L=~X(_ zBk0E^sJ0{E7x>!-;cAg(7q82_&QW_&#eLWum1!#K*(_viIg_d6c{S_YO}ngR#<_^W z1g>753jOZ5vpd-0g3oAh7AJ|!yU<~(HzhH<(|wuWGA-<@SEHQb{AGS?g)y_jPL^es z(WJaMBP()-kbr_-)&avlL8J1#1uwn_)HH5L<57J^m8*hfHRsqxdLvDeKf4*zt~#G5 z-)R{jHC_^;{QgW0g%;n6jjv<7{L%4GpHwhjfl^}~z9GKX+xW;7^`-LU%Pqg3sQfsR zxK}_5+kGbuS*Uzu#UXN>Qp_ zdD>cIG1*42ni50%EnAkj(IiLj8}^cvcmR}0;) z-7*i^C13YNi6t_M@uS45(T^E-hCBM#1R|YJRS%da=P*(*aYQ9m@tPSGi!w7gNGNKz z*O=_{tt>NN*>#V%f?nZU@DtZ0TzKlElP;OSkP1QczVJ7mN+tB?XJ`w` z3wyrF>&4W*d&IDf-??0%PtKH+CQ;BCU8l79CcQ|YnXBJy9D>gDpl^^Sszo%@xB+R}RHG%s6+n-6>v4rziJX)iu~wz+bG=(h(+YdJ!6bGRNips(+incK8oOq zh_44SX+Lvy@uyuAev^Q^acJN0j`&-nXU~OilB<6g_O`R}>Gn|7`}?vxPsB=XMZoks zV_owi{bFF7mFy255vR*+YVkQBQ5%L$O?+7AVnOk=&FC!KnLPQ8ZLPB7Zbli|0O-x!OWE zi|j6K8~>Y)L7e5$&7%}kxTD3Mqhu4?4)d9n&|4Q5RJh1lQcBeOuzs9FSO-TPymeUf zGZ__Z;rZglNoSCyOlDD!z$eIT%pyf@;IRK0Ttn%Pe+nfeu03Udklj;B{&bWb?RRB$ zuBKe^SUt&qn~FXG(Jau*b}2GcNy}n9!TJNgflh%uA8%QOI(A!*t|X<-aV>gua)##n z*s*F`*)nfzK^N7Hh^9VcCvj!=Hrc@R+vBWjV%*QLqS2f1BmcUbr_l_6<%?5#KAC8+@<6{GzP8u+J{)VyFhq$P!v*U2(} zsPb{;O=Qv8$P`LGO1pL*rH?WDKjF@{&_+L65;;1MH8a)xaM!TLmtubRpqDZ2(M(ps zp|^K@V!!XtzI#jYfrAnwd~kBDw!N0H%k3o=zK?pWyqxyG4DiZ`FO2)_jZ&}^%rDX` zex_UZIF--o(WP@Y`_Ts2K$EyW(K<)(>|QhZd}k^JQpkyp-9FV;VP@=Cz_+yGsThIoDwO4h_2<1^uHCX zt?h_5%Je(vuNm+xUl#eQq;nHBt1H%3N|(khW#^?g-@)7YH`QsYY^7XF+0iw99bT2jaF2lmD>o3lAYSqRj(GZR3iY`Os3SAZtb)o254fa1^{6VVqBm=^~Se{fOu*@JugVOz>zkr&l|9t+_ z`;V7abEvxyrj3z{b5yWtxA?Femme!uO|9rSIVz$NrTmyJic|Ntb&$+nH7zykD78|q zOT0-&C1weU20|sc5BD7EYajKRylheqa|v4x9@}iNtn8aCs!mk2iYlYUEOIz7SmT^Ayez%@Nm*E{LcoVRd94BMYHymPgPul& zlt=9gIIAo&gnT0V#D6O93m4a7ert0Hb%R*%>8--wx-Q%D4A!&1wq$LzWF0ap^5n@q z>#d(rm&5mJFl1G}-hChCzH_P_zQ=}c-3foc7{%r=pWyVO(5FaCCO4yq!%@fGe9tix z`v=lo?bOZR3STkdD#=*_9-yDOf7tuNQ1u?07e?)bspF~eA4VD-0nVnXS9v3D+e)wn|(NDMPE=41ML z`bj?QE8k}(zsLaM5)t=v?j*(qPv}qZ3H@(!V9;m`kS!}hkTYjx1(NuLF4F2J!b=ws z2V~3WtbuHqV}^yD)n*%GI1*j~?O!A$!6-;oheQyD26g=7G=XHx)+Axe6*8xFo_)>D z4XFm724D|awGbAAf?~$cHLJV$-(=Pe9p#?qFFx?@+C)+-u6~}agr$AoorCR6W&3C; z^XQ1VOD7hnxiZmk-4TU`N-GqO!k_PzyMybN zV7EQN7c;7E_6rnbq|VA2CqC_u*39hS)IKyyR^L*uo?@oHn39;Fp&S+e`F!z#*}$48 zS^0y=X5Vp4uu~eLJ?p@^#Mqm`}y?Pw^RF;0V@;6tjU*9z>*ITF`-BJAblfb-v}|bPGMW3j)nM%06;dNXBINv= zH-Tfx_M;&m7KHw?!r*EAU^A$V(YT6j$(_lPTMEWmwr!uN(V%vvtM!0&;7s+{Oa ziW{ulJnW@k%5vhuRgC2gd~k+ao8Mo_Z+Ob{HbrFGUtK7WM>iy4XXHLx=Q7_n7>2ud zWA!t`i$c;7*S_6zWaT=Rc67IA)c#9Pgv8zsR&X|Dc2~Yj3#q=CL58Fu)l($4xX>U6 z`7h=y77_{t40q?@zRqu^dclkC@*{?vUKR>h4_u6liGkPCD**=g$2}JB#lB{i96D_q z@Pwv0k9h(YsFq!2$n$b45p}0b!k)4TmsvZMG346pzrQZ?HeKUlY4M%mCeCYIMy$N4 z$~CC^C!;&#o=q0n0l7Yhnrq&dlZCtYH5yUjcJ)b3T%Aunv+wTapofWZS7|v7gV`=9 zen~kPcK+KA10#6n1c$myXs~Xiub1CzHG1q>5idRNS@d2cZ1nXmmA8t1{80(m`U;QP z{)LX>l$BSZl8>Z?#*Ug8TR)fIH4)Pwf*SKNn)UeF-FS&kd@5~uQNOX(J74Zs_w;acPWX}vFz|XK1PdJMZz7C+fV7iBaZ6>p z@a$!~+x^z@!dHe?d$zPtZ)ONl@Gu!j3B3guIZ`zonRmRdAC2_yr@C@MxW~z8NWlQE zZmlbk>h;BEmg|*k0!?dlcM-i^{?_GC!MCIJACT5fY`7E694;Q9#uJcxJ>warB6Db=Y8b|i<$I>sHIKl< zVcE!W_kB*s+O?%lc8nY4&kV(InYoGzY!}Ptn4QB=Y}a$hpHWb&`Qf~E&i!t5zqjVd zGqx9lXWN=*Y7H#vulG1BU|tUNR|O4z54|%ovbaof%;IPesVPn(N<;q5=RNHRbw(w8 z-eB$a?91qy5!pw%700I@6(~`A`YjK^U1ViDT=_&jY(_)Fg|BlA)|9}l2Rfxzo6F^& zAQk)4_8G-{HTJeLyVs(SKqcdg={ z-7&een&9y*^+jYye1B9l`f1iKn!gx|7vi^uU77Lwv>S(!N!@neVB9VDYDwR9VHeif z=pfNbGyGVIgB$UOiT+5O^!IbgH=J)`A zGTpGqfJFZXk^E}Uo|Ctr(2^7_!@h@o4YJ5KWuDk(NU9YiqD+vI|Ht)d9si2~$;o!A z1OQ+gS_lAaoFr&uw26I>-A7%Xgg_2aYkl_=G6j`}EQ_7F>k#1B2xby`1W3H355Z#z zz)F9tlLXjp@kP)ppVj%mH1MdGnuqq)qk450>;ArIbKHQ!uryP6mY;y)T%tL=cZ|$X zWk658byfRej*5PJ)4i^+$zG>uws$V)HzjjUkmxnM&JaNj|1N!6 z56cKPYscq7&OcwMn72;a-nNxTZK0ipT#roy@5S&W&fuc6d4~1neG)~JE8h0Bo)902 zySLBXx4g7UtRSi;*e>YOV~2!tvLsnDrJl}gJ*Dkj9bK9FdyC#u%3m#W-r8q<-!HzD zXCxSIwU=E)u`Ndh#^%&6kRkC1?3BHWpXh7S&uoa>;kTL0P>Q2xv}Zb2PG?W22P>*8 zPvYqg8kov%*~Rdzm$1(!;?GgcT@6h6Cn&09ejpBQ+5~fbN^WyYBc^2N$*JPmlFdh! zR>*E0G>9YaAOHpXwO>XW!FoVjPj3T)6&l}v1P(#I-ERViYskdk#P}g00oKJx_VkV0Vi=b9bwKLFV%P6u522se z3LOWB_}0+=^k+yQ^@=|q;zwA&^+|BJes4!`xZW%$aJYVFXK;vb4(+;-FJk>xry#}E zwd+9uStR~z9jK2BI9$KaFAs|ZFlKOT0F#BB?wY4v`&0s*u`WG~1W-79{nnwiK>2k2 zB)n?nKi+F?n;4u3lrg~u2>ukHCj3(tb4|w*`BNAZLf1kx5BS@!aaSOJ%42#!K!ddl ztg%xde=1`_BA3XY%9x(uaJ@Ucz~Op#K{sFh@52=;`c$K;t1HYqS4XHK1AcvwPRV6kSVuAZRV7>0krNTCdMYNFDo>ClGeV z_m!=)4Rj6?PwesYY$Z2aOGv_dF9=0l98Y;TI#|KI(4t6$C{_f1+{44gU0O`c>7U<3 zPr2C((ux7SZ5vN(+l{BW*x13XEUiz1`~18FD2l%UAQw@FMsKgOGzuh@App|>cvL{< ziZDV-0D%w?{H@=;-E8e>A(jOcEf7s!{{sP}1i(1k!T+&APYCn}cV4%l0l@=!DmU6B zrJ#vQXoJ?ZjW#Lp<%HMGP#Lpk6IA@T(R8wzq@1U86UgusSZ{U)@D6U`0avxwRYJP||^vGE>2RGzSHz`qc+ z7q}G2)oeT$>|GPJ1C4>=2!e8uhB~1Q3tlq`Y~lzguGo04xCHS%#3hOEA%TSGC^nue zfrgZ#32cy7JE2XAxL+`U@sqF|1`YX%jn~6Kj3@*)N$_M6*)T+F4p0Y-*uaemZ6N3;oEvB)n#e~1 z{wAV$13bM%eT4?jjHu3Fq7u~wqzx1HJ&=+l@`Gq#BphEL+(CR_=)ojx8z7q^YMVH* zF9S~kacl-c8lw7%gGGn%9^erqDklY=xL@Tw+$0cjRLzlx43?f(GGqW!S| literal 0 HcmV?d00001 diff --git a/analysis/pd_sep_paper_section/scripts/plot_roofline.py b/analysis/pd_sep_paper_section/scripts/plot_roofline.py new file mode 100644 index 0000000..204a4c1 --- /dev/null +++ b/analysis/pd_sep_paper_section/scripts/plot_roofline.py @@ -0,0 +1,144 @@ +"""C6: roofline plot for Qwen3-Coder-30B-A3B on H20. + +Reproduces the analytical roofline used in scripts/compute_roofline.py and +plots it as a single PDF: AI vs achievable throughput, with annotated +operating points for prefill at reuse {0, 70, 90, 95}% and decode. + +The constants must stay in lockstep with compute_roofline.py. If you change +one, change the other. +""" +import argparse +from pathlib import Path + +import matplotlib +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np + +# ---- model constants (mirror scripts/compute_roofline.py) ---- +L, D, H_KV, D_HEAD, D_FFN = 48, 2048, 4, 128, 6144 +K_EXPERTS = 8 +BYTES = 2 # bf16 + +# ---- H20 ---- +PEAK_FLOPS = 148e12 +HBM_BW = 4.0e12 +RIDGE = PEAK_FLOPS / HBM_BW # ~37 + + +def attn_prefill_flops(seq_len, new_tokens): + d_kv = H_KV * D_HEAD + qkv = new_tokens * (D * D * 2 + D * d_kv * 2 * 2) + attn = new_tokens * seq_len * D * 2 * 2 + out = new_tokens * D * D * 2 + return (qkv + attn + out) * L + + +def attn_prefill_bytes(seq_len, new_tokens, cached_tokens): + d_kv = H_KV * D_HEAD + weight = D * (D + 2 * d_kv + D) * BYTES * L + cached_kv = cached_tokens * 2 * d_kv * BYTES * L + act = new_tokens * D * BYTES * 2 * L + new_kv = new_tokens * 2 * d_kv * BYTES * L + return weight + cached_kv + act + new_kv + + +def ffn_flops(n): + return 3 * n * D * D_FFN * 2 * K_EXPERTS * L + + +def ffn_bytes(n): + weight = K_EXPERTS * 3 * D * D_FFN * BYTES * L + act = n * D * BYTES * 2 * L + return weight + act + + +def point(seq_len, reuse): + cached = int(seq_len * reuse) + new = max(1, seq_len - cached) + f = attn_prefill_flops(seq_len, new) + ffn_flops(new) + b = attn_prefill_bytes(seq_len, new, cached) + ffn_bytes(new) + return f, b, new + + +def decode_point(seq_len): + f = attn_prefill_flops(seq_len, 1) + ffn_flops(1) + b = attn_prefill_bytes(seq_len, 1, seq_len) + ffn_bytes(1) + return f, b + + +def plot(out_path, seq_len=64000): + fig, ax = plt.subplots(figsize=(6.5, 4.2)) + + ai_grid = np.logspace(-1, 5, 400) + achievable = np.minimum(ai_grid * HBM_BW, PEAK_FLOPS) / 1e12 + ax.plot(ai_grid, achievable, color="#222", lw=1.5, label="H20 roofline") + + ax.axvline(RIDGE, color="#888", ls=":", lw=1) + ax.text(RIDGE, 420, f"ridge = {RIDGE:.0f}", color="#666", + fontsize=8, ha="center", va="top", + bbox=dict(boxstyle="round,pad=0.2", fc="white", ec="none", alpha=0.85)) + + ax.axhline(PEAK_FLOPS / 1e12, color="#aaa", ls="--", lw=0.6) + ax.text(2, PEAK_FLOPS / 1e12 * 1.08, "compute ceiling (148 TFLOPS bf16)", + fontsize=8, color="#666", ha="left") + + # operating points: use a legend (not annotations with leader lines, since + # all 4 prefill points sit on the compute ceiling and would overlap). + reuses = [0.0, 0.7, 0.9, 0.95] + colors = ["#d62728", "#ff7f0e", "#2ca02c", "#1f77b4"] + for reuse, color in zip(reuses, colors): + f, b, new = point(seq_len, reuse) + ai = f / b + thpt = min(ai * HBM_BW, PEAK_FLOPS) / 1e12 + ax.scatter([ai], [thpt], color=color, s=80, zorder=5, + edgecolor="white", linewidth=1.2, + label=f"prefill reuse={int(reuse*100):>2}% " + f"(new={new:>6,} tok, AI={ai:>6,.0f})") + + f, b = decode_point(seq_len) + ai_dec = f / b + thpt_dec = min(ai_dec * HBM_BW, PEAK_FLOPS) / 1e12 + ax.scatter([ai_dec], [thpt_dec], color="#8c564b", s=80, marker="D", + zorder=5, edgecolor="white", linewidth=1.2, + label=f"decode (per-token, seqlen={seq_len:,}, AI={ai_dec:.1f})") + + ax.legend(loc="lower right", fontsize=8.5, framealpha=0.95, + prop={"family": "monospace", "size": 8}) + + ax.set_xscale("log") + ax.set_yscale("log") + ax.set_xlim(0.5, 1e5) + ax.set_ylim(0.5, 500) + ax.set_xlabel("Arithmetic intensity (FLOP/byte)") + ax.set_ylabel("Achievable throughput (TFLOPS)") + ax.set_title( + f"Prefill stays compute-bound even at 95% reuse " + f"(Qwen3-Coder-30B-A3B, H20, seqlen={seq_len:,})", + fontsize=10, + ) + ax.grid(True, which="both", alpha=0.25) + fig.tight_layout() + fig.savefig(out_path, bbox_inches="tight") + plt.close(fig) + print(f"[C6] wrote {out_path}") + for reuse in reuses: + f, b, new = point(seq_len, reuse) + print(f" reuse={int(reuse*100):>3}% new={new:>6,} AI={f/b:>8.1f} " + f"bound={'COMPUTE' if f/b > RIDGE else 'MEMORY'}") + f, b = decode_point(seq_len) + print(f" decode AI={f/b:>8.1f} bound={'COMPUTE' if f/b > RIDGE else 'MEMORY'}") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--seq-len", type=int, default=64000) + ap.add_argument("--outdir", default="analysis/pd_sep_paper_section/figures") + args = ap.parse_args() + out = Path(args.outdir) + out.mkdir(parents=True, exist_ok=True) + plot(out / "fig_c6_roofline.pdf", seq_len=args.seq_len) + + +if __name__ == "__main__": + main() diff --git a/analysis/pd_sep_paper_section/scripts/plot_routing_lever.py b/analysis/pd_sep_paper_section/scripts/plot_routing_lever.py new file mode 100644 index 0000000..d243f25 --- /dev/null +++ b/analysis/pd_sep_paper_section/scripts/plot_routing_lever.py @@ -0,0 +1,123 @@ +"""C7: routing lever vs PD-separation lever. + +Side-by-side comparison of the magnitude of two design changes on the same +agentic workload: + (A) Round-robin -> cache-aware routing, both Combined-mode + (B) Combined -> PD-separated, both cache-aware + +For each, plot delta TTFT p50 / TPOT p90 / APC. Green = improvement, red = +regression. Numbers come from REPORT.md §3.1 (PD-separation_analysis.md §3.1). + +CAVEAT shown on the figure: these numbers are from the legacy +trace methodology (random sampling, 1 req/GPU). They are not yet reproduced +on the trace-driven 850-req sampling at production concurrency, and the +PD-sep runs were captured with --enforce-eager. The current plot is meant +to show the qualitative gap between the two levers; a re-run is required +for paper-grade quantitative claims. +""" +import argparse +from pathlib import Path + +import matplotlib +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np + +# (label, RR baseline, cache-aware baseline, PD-sep w/ cache-aware, +# unit, format, "improve_when_smaller") +ROWS = [ + ("TTFT p50 (s)", 1.836, 0.731, 1.261, "s", "{:.2f}", True), + ("TPOT p90 (s)", 0.086, 0.073, 0.074, "s", "{:.3f}", True), + ("APC (%)", 20.8, 44.7, 40.2, "pp", "{:.1f}", False), +] + + +def pct_delta(before, after, improve_when_smaller): + """Return signed % change framed so positive = improvement. + + For APC (pp): return absolute pp delta because relative % is misleading. + """ + diff = after - before + if improve_when_smaller: + improvement = -(diff / before) * 100 + return improvement, f"{improvement:+.0f}%" + pp = diff + return pp, f"{pp:+.1f}pp" + + +def plot(out_path): + fig, axes = plt.subplots(1, 3, figsize=(10, 3.5)) + + bar_colors = lambda val: "#2ca02c" if val >= 0 else "#d62728" + + for ax, (metric, rr, ca, pdsep, unit, fmt, smaller_better) in zip(axes, ROWS): + # lever A: RR -> cache-aware (both combined) + a_val, a_txt = pct_delta(rr, ca, smaller_better) + # lever B: combined -> PD-sep (both cache-aware) + b_val, b_txt = pct_delta(ca, pdsep, smaller_better) + + bars = ax.bar( + ["RR → cache-aware\n(within Combined)", + "Combined → PD-Sep\n(both cache-aware)"], + [a_val, b_val], + color=[bar_colors(a_val), bar_colors(b_val)], + edgecolor="black", linewidth=0.6, width=0.55, + ) + + ymax = max(abs(a_val), abs(b_val)) + ax.set_ylim(-ymax * 1.35, ymax * 1.35) + ax.axhline(0, color="black", lw=0.6) + + for bar, val, txt in zip(bars, [a_val, b_val], [a_txt, b_txt]): + yoff = ymax * 0.06 if val >= 0 else -ymax * 0.06 + ax.text(bar.get_x() + bar.get_width() / 2, + val + yoff, + txt, + ha="center", va="bottom" if val >= 0 else "top", + fontsize=10, fontweight="bold") + + ax.set_title(metric, fontsize=10) + if smaller_better: + ax.set_ylabel("Δ (positive = improvement)") + else: + ax.set_ylabel("Δ percentage points") + ax.grid(True, axis="y", alpha=0.25) + ax.tick_params(axis="x", labelsize=8.5) + u = "" if unit == "pp" else unit + ax.set_xlabel( + f"RR={fmt.format(rr)}{u} · CA={fmt.format(ca)}{u} · PD-Sep={fmt.format(pdsep)}{u}", + fontsize=8, color="#555", labelpad=8, + ) + + fig.suptitle( + "Cache-aware routing is a larger lever than PD separation on agentic workload", + fontsize=11, y=1.02, + ) + fig.tight_layout(rect=(0, 0.10, 1, 0.96)) + footer = ( + "Source: REPORT.md §3.1 / analysis/pd_separation_analysis.md §3.1. " + "Legacy random-sampling methodology + --enforce-eager. " + "Re-run on trace-driven w600_r0.0015_st30 with cuda-graph required before paper-grade citation." + ) + fig.text(0.5, 0.01, footer, ha="center", fontsize=7.5, color="#666", + style="italic", wrap=True) + fig.savefig(out_path, bbox_inches="tight") + plt.close(fig) + print(f"[C7] wrote {out_path}") + for metric, rr, ca, pdsep, unit, fmt, smaller in ROWS: + a, a_txt = pct_delta(rr, ca, smaller) + b, b_txt = pct_delta(ca, pdsep, smaller) + print(f" {metric:14s} RR→CA: {a_txt:>7s} Combined→PD-Sep: {b_txt:>7s}") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--outdir", default="analysis/pd_sep_paper_section/figures") + args = ap.parse_args() + out = Path(args.outdir) + out.mkdir(parents=True, exist_ok=True) + plot(out / "fig_c7_routing_lever.pdf") + + +if __name__ == "__main__": + main() diff --git a/analysis/pd_sep_paper_section/scripts/plot_workload.py b/analysis/pd_sep_paper_section/scripts/plot_workload.py new file mode 100644 index 0000000..e5fc37e --- /dev/null +++ b/analysis/pd_sep_paper_section/scripts/plot_workload.py @@ -0,0 +1,217 @@ +"""C1: workload characterization figures. + +Generates two figures from the sampled trace: + fig_c1a_io_cdf.pdf -- input / output token CDF (two panels) + fig_c1b_reuse.pdf -- KV-block reuse decomposition + +Run on dash0 where the trace lives and matplotlib is installed. + +Usage: + .venv/bin/python scripts/plot_workload.py \ + --trace traces/w600_r0.0015_st30.jsonl \ + --outdir analysis/figures +""" +import argparse +import json +import sys +from collections import Counter +from pathlib import Path + +import matplotlib +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np + +BLOCK_SIZE = 512 + + +def load_trace(path): + rows = [json.loads(l) for l in open(path)] + rows.sort(key=lambda r: float(r["timestamp"])) + return rows + + +def percentile_markers(arr, qs=(0.5, 0.9, 0.99)): + arr = np.asarray(arr) + return {q: float(np.quantile(arr, q)) for q in qs} + + +def plot_io_cdf(rows, out_path): + inputs = np.array([r["input_length"] for r in rows if r["input_length"] > 0]) + outputs = np.array([r["output_length"] for r in rows if r["output_length"] > 0]) + + fig, axes = plt.subplots(1, 2, figsize=(8.5, 3.2)) + + for ax, data, label, log in [ + (axes[0], inputs, "input tokens (log scale)", True), + (axes[1], outputs, "output tokens", False), + ]: + sorted_d = np.sort(data) + cdf = np.arange(1, len(sorted_d) + 1) / len(sorted_d) + ax.plot(sorted_d, cdf, color="#1f77b4", lw=1.6) + if log: + ax.set_xscale("log") + ax.set_xlabel(label) + ax.set_ylabel("CDF") + ax.set_ylim(0, 1.02) + ax.grid(True, alpha=0.3) + + pcts = percentile_markers(data) + for q, v in pcts.items(): + ax.axvline(v, color="#888", ls=":", lw=0.8) + ax.annotate( + f"p{int(q*100)}={int(v):,}", + xy=(v, q), + xytext=(4, -8), + textcoords="offset points", + fontsize=8, + color="#444", + ) + + io_ratio = inputs.sum() / max(outputs.sum(), 1) + fig.suptitle( + f"Agentic workload I/O: aggregate ratio = {io_ratio:.1f}x " + f"(N={len(rows)} requests, sampled from GLM-5.1)", + fontsize=10, + ) + fig.tight_layout(rect=(0, 0, 1, 0.94)) + fig.savefig(out_path, bbox_inches="tight") + plt.close(fig) + print(f"[C1a] wrote {out_path}") + print(f" input p50={int(np.quantile(inputs, 0.5)):,} " + f"p90={int(np.quantile(inputs, 0.9)):,} " + f"p99={int(np.quantile(inputs, 0.99)):,}") + print(f" output p50={int(np.quantile(outputs, 0.5)):,} " + f"p90={int(np.quantile(outputs, 0.9)):,} " + f"p99={int(np.quantile(outputs, 0.99)):,}") + print(f" aggregate I/O ratio = {io_ratio:.2f}x") + + +def reuse_decomposition(rows): + """Classify every cacheable block as intra-session / cross-session / unique. + + Walk requests in timestamp order. For each block (hash_id) in the request: + - if first time seen globally -> 'unique-or-future-reuse' (resolved later) + - if already seen earlier within the same session -> 'intra-session' + - if already seen in a different session -> 'cross-session' + After the pass, blocks classified as 'unique-or-future-reuse' that have + a global refcount of 1 are 'unique'; those with refcount > 1 stay where + they were first seen (counted under whichever later request reused them). + + Token counts use BLOCK_SIZE = 512. + """ + # Session id resolution mirrors analyze_cache_hit.py. + chat_to_session = {} + block_first_session = {} # hid -> session_id of first emitter + block_seen_in_session = {} # hid -> set of session_ids that have seen it + block_global_count = Counter() + + intra = 0 + cross = 0 + first_time = 0 # token-count of blocks the first time they appear + + for r in rows: + cid = int(r["chat_id"]) + pid = int(r["parent_chat_id"]) + sid = r.get("session_id", + str(cid) if pid < 0 else chat_to_session.get(pid, str(pid))) + sid = str(sid) + chat_to_session[cid] = sid + + for hid in r.get("hash_ids", []): + block_global_count[hid] += 1 + if hid not in block_first_session: + block_first_session[hid] = sid + block_seen_in_session[hid] = {sid} + first_time += BLOCK_SIZE + else: + if sid in block_seen_in_session[hid]: + intra += BLOCK_SIZE + else: + cross += BLOCK_SIZE + block_seen_in_session[hid].add(sid) + + # Of the first-time tokens, those whose block was never reused are 'unique'. + unique_tokens = 0 + reused_first = 0 + for hid, count in block_global_count.items(): + if count == 1: + unique_tokens += BLOCK_SIZE + else: + reused_first += BLOCK_SIZE # first emission of a reused block + + # Total tokens (block-rounded) = intra + cross + first_time + # first_time decomposes into: unique_tokens + reused_first + # For the reuse story we attribute first_time to 'unique vs the + # first-emit-of-a-shared-block'. Convention used in the figure: + # intra-session reuse = subsequent hits within the same session + # cross-session reuse = subsequent hits across sessions + # first emission (will-reuse) = block emitted once, reused later + # unique (never-reuse) = block emitted exactly once, never hit again + return { + "intra_session_reuse_tokens": intra, + "cross_session_reuse_tokens": cross, + "first_emission_will_reuse_tokens": reused_first, + "unique_no_reuse_tokens": unique_tokens, + } + + +def plot_reuse(rows, out_path): + d = reuse_decomposition(rows) + total = sum(d.values()) + parts = [ + ("intra-session reuse", d["intra_session_reuse_tokens"], "#2ca02c"), + ("cross-session reuse", d["cross_session_reuse_tokens"], "#1f77b4"), + ("first emission (reused later)", d["first_emission_will_reuse_tokens"], "#ff7f0e"), + ("unique (never reused)", d["unique_no_reuse_tokens"], "#d62728"), + ] + + fig, ax = plt.subplots(figsize=(8.5, 1.9)) + left = 0 + for label, val, color in parts: + frac = val / total + ax.barh(0, frac, left=left, color=color, edgecolor="white", height=0.6, label=label) + if frac > 0.025: + ax.text(left + frac / 2, 0, + f"{label}\n{frac*100:.1f}%", + ha="center", va="center", fontsize=8.5, color="white") + left += frac + + ax.set_xlim(0, 1) + ax.set_yticks([]) + ax.set_xlabel("share of total cacheable tokens (block-aligned, 512 tok blocks)") + ax.set_title("Where do prefix cache hits come from? " + f"(N={len(rows)} requests, sampled trace)") + ax.legend(loc="upper center", bbox_to_anchor=(0.5, -0.45), ncol=4, fontsize=8, frameon=False) + for spine in ("top", "right", "left"): + ax.spines[spine].set_visible(False) + fig.tight_layout() + fig.savefig(out_path, bbox_inches="tight") + plt.close(fig) + print(f"[C1b] wrote {out_path}") + for label, val, _ in parts: + print(f" {label:40s} {val/total*100:5.1f}% ({val:>12,} tokens)") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--trace", default="traces/w600_r0.0015_st30.jsonl") + ap.add_argument("--outdir", default="analysis/pd_sep_paper_section/figures") + args = ap.parse_args() + + trace = Path(args.trace) + outdir = Path(args.outdir) + outdir.mkdir(parents=True, exist_ok=True) + + if not trace.exists(): + sys.exit(f"trace not found: {trace}") + + rows = load_trace(trace) + print(f"loaded {len(rows)} requests from {trace}") + + plot_io_cdf(rows, outdir / "fig_c1a_io_cdf.pdf") + plot_reuse(rows, outdir / "fig_c1b_reuse.pdf") + + +if __name__ == "__main__": + main() diff --git a/scripts/bench.sh b/scripts/bench.sh index 77c6013..23f349e 100755 --- a/scripts/bench.sh +++ b/scripts/bench.sh @@ -147,7 +147,7 @@ launch_instances() { $VLLM serve "$MODEL" \ --host 0.0.0.0 --port $port \ --tensor-parallel-size 1 \ - --trust-remote-code --enable-prefix-caching --enforce-eager \ + --trust-remote-code --enable-prefix-caching \ --dtype auto --gpu-memory-utilization 0.9 --max-model-len 200000 \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \ $vllm_extra_args \ @@ -158,7 +158,7 @@ launch_instances() { $VLLM serve "$MODEL" \ --host 0.0.0.0 --port $port \ --tensor-parallel-size 1 \ - --trust-remote-code --enable-prefix-caching --enforce-eager \ + --trust-remote-code --enable-prefix-caching \ --dtype auto --gpu-memory-utilization 0.9 --max-model-len 200000 \ $vllm_extra_args \ > "$logfile" 2>&1 & diff --git a/scripts/launch_elastic_p2p.sh b/scripts/launch_elastic_p2p.sh index f5e63b3..6a2e8e8 100755 --- a/scripts/launch_elastic_p2p.sh +++ b/scripts/launch_elastic_p2p.sh @@ -54,7 +54,7 @@ for i in $(seq 0 $((N_INSTANCES - 1))); do $VLLM serve "$MODEL" \ --host 0.0.0.0 --port $port \ --tensor-parallel-size 1 \ - --trust-remote-code --enable-prefix-caching --enforce-eager \ + --trust-remote-code --enable-prefix-caching \ --dtype auto --gpu-memory-utilization 0.9 --max-model-len 200000 \ --kv-transfer-config \ '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \ diff --git a/scripts/launch_pd_mooncake.sh b/scripts/launch_pd_mooncake.sh index ea3dd59..1fc90f9 100755 --- a/scripts/launch_pd_mooncake.sh +++ b/scripts/launch_pd_mooncake.sh @@ -44,7 +44,6 @@ $VLLM serve "$MODEL_PATH" \ --tensor-parallel-size 4 \ --trust-remote-code \ --enable-prefix-caching \ - --enforce-eager \ --dtype auto \ --gpu-memory-utilization 0.9 \ --kv-transfer-config \ @@ -61,7 +60,6 @@ $VLLM serve "$MODEL_PATH" \ --tensor-parallel-size 4 \ --trust-remote-code \ --enable-prefix-caching \ - --enforce-eager \ --dtype auto \ --gpu-memory-utilization 0.8 \ --kv-transfer-config \ diff --git a/scripts/launch_pd_separated.sh b/scripts/launch_pd_separated.sh index df9366d..feac254 100644 --- a/scripts/launch_pd_separated.sh +++ b/scripts/launch_pd_separated.sh @@ -49,7 +49,6 @@ CUDA_VISIBLE_DEVICES=0,1,2,3 $VLLM serve "$MODEL_PATH" \ --tensor-parallel-size 4 \ --trust-remote-code \ --enable-prefix-caching \ - --enforce-eager \ --dtype auto \ --gpu-memory-utilization 0.9 \ --kv-transfer-config \ @@ -65,7 +64,6 @@ CUDA_VISIBLE_DEVICES=4,5,6,7 $VLLM serve "$MODEL_PATH" \ --tensor-parallel-size 4 \ --trust-remote-code \ --enable-prefix-caching \ - --enforce-eager \ --dtype auto \ --gpu-memory-utilization 0.8 \ --kv-transfer-config \ diff --git a/scripts/launch_phase1_ps.sh b/scripts/launch_phase1_ps.sh index 8ee9389..ae9c556 100755 --- a/scripts/launch_phase1_ps.sh +++ b/scripts/launch_phase1_ps.sh @@ -29,7 +29,7 @@ for i in $(seq 0 6); do echo "Starting C instance $i on GPU $i, port $((8000+i)), bootstrap $((8998+i))" VLLM_MOONCAKE_BOOTSTRAP_PORT=$((8998+i)) MASTER_PORT=$((29500+i)) CUDA_VISIBLE_DEVICES=$i \ .venv/bin/vllm serve "$MODEL" --host 0.0.0.0 --port $((8000+i)) --tensor-parallel-size 1 \ - --trust-remote-code --enable-prefix-caching --enforce-eager \ + --trust-remote-code --enable-prefix-caching \ --dtype auto --gpu-memory-utilization 0.9 --max-model-len 200000 \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \ > "$OUTDIR/vllm_c_$i.log" 2>&1 & @@ -40,7 +40,7 @@ done echo "=== Launching PS instance on GPU 7, port 8007, bootstrap 9005 ===" VLLM_MOONCAKE_BOOTSTRAP_PORT=9005 MASTER_PORT=29507 CUDA_VISIBLE_DEVICES=7 \ .venv/bin/vllm serve "$MODEL" --host 0.0.0.0 --port 8007 --tensor-parallel-size 1 \ - --trust-remote-code --enable-prefix-caching --enforce-eager \ + --trust-remote-code --enable-prefix-caching \ --dtype auto --gpu-memory-utilization 0.9 --max-model-len 200000 \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \ > "$OUTDIR/vllm_ps_0.log" 2>&1 &