Gahow Wang
72790ae6c1
Instrumentation patches (microbench/patches/):
- pd_profile.py: shared event emitter (VLLM_PD_PROFILE_LOG env var)
- apply_patches.py: idempotent patch installer for mooncake_connector.py
and scheduler.py, marks insertions with # PD_PROFILE_PATCH
- analyze_events.py: joins per-process JSONL event logs by transfer_id
into per-request phase durations
Seven events captured per request:
D_get_num_matched → P_zmq_received → P_prefill_done →
P_rdma_start → P_rdma_end → D_recv_complete → D_request_promoted
Driver fix (microbench/lifecycle/driver.py):
seed_prefix_cache now sends via the proxy URL so P and D both cache
the seeded prefix with matching block hashes. Previously seeding D
directly produced different block hashes than the proxy-routed
measurement requests, making incremental transfer impossible.
Real breakdown (fig_breakdown_real.png, server_breakdown.csv, n=93):
prefill_compute 620 ms median (95% of overhead)
rdma_transfer 42 ms median (~71 Gbps effective)
other overhead 10 ms median (dispatch + params + signal + promote)
Mooncake transfer is NOT the bottleneck. Even with bulk RDMA the
transfer cost is <10% of prefill cost for Qwen3-30B-A3B on H20.
36 KiB
36 KiB
| 1 | D_promote_ms | build_params_ms | completion_sig_ms | d_to_p_dispatch_ms | delta_to_pull | events_seen | full_overhead_ms | n_events | num_local_cached | num_prompt_tokens_P | num_send_blocks | prefill_compute_ms | prompt_tokens | rdma_bandwidth_gbps | rdma_bytes | rdma_num_ops | rdma_transfer_ms | remote_total | t_D_get_num_matched | t_D_recv_complete | t_D_request_promoted | t_P_prefill_done | t_P_rdma_end | t_P_rdma_start | t_P_zmq_received | transfer_id |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 1.572544 | 128.810689 | 0.429406 | 157.586296 | 11 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 180.077196 | 7 | 0 | 11 | 1 | -128.19752 | 11 | 0.6330776134029652 | 1572864 | 96 | 19.875781 | 11 | 343433675745977 | 343433854250629 | 343433855823173 | 343433705134753 | 343433853821223 | 343433833945442 | 343433833332273 | xfer-1f4a9d22-d840-45c1-9643-e93b1456a9e2 |
| 3 | 2.191308 | 0.903295 | 0.506658 | 1.640978 | 978 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 77.893983 | 7 | 0 | 978 | 62 | 64.728703 | 978 | 98.46478694228644 | 97517568 | 96 | 7.923041 | 978 | 343528336446320 | 343528412148995 | 343528414340303 | 343528402816001 | 343528411642337 | 343528403719296 | 343528338087298 | xfer-464cc62e-38aa-45ee-affc-01c3763aeece |
| 4 | 2.093855 | 0.913908 | 0.396394 | 2.31242 | 994 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 80.174358 | 7 | 0 | 994 | 63 | 67.058527 | 994 | 107.13559177722512 | 99090432 | 96 | 7.399254 | 994 | 343529586703278 | 343529664783781 | 343529666877636 | 343529656074225 | 343529664387387 | 343529656988133 | 343529589015698 | xfer-f1f3be4e-2eff-429d-811f-c3638c668fad |
| 5 | 1.938306 | 1.154961 | 0.442667 | 2.36916 | 967 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 131.651798 | 7 | 0 | 967 | 61 | 118.149768 | 967 | 101.03515838490675 | 95944704 | 96 | 7.596936 | 967 | 343530839741058 | 343530969454550 | 343530971392856 | 343530960259986 | 343530969011883 | 343530961414947 | 343530842110218 | xfer-fcc5b5ec-1deb-4656-bb5e-23ef9cd0ac6b |
| 6 | 1.824815 | 1.547935 | 0.503381 | 1.881943 | 3799 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 307.631595 | 7 | 0 | 3799 | 238 | 273.494352 | 3799 | 105.52574869264141 | 374341632 | 96 | 28.379169 | 3799 | 343534160883920 | 343534466690700 | 343534468515515 | 343534436260215 | 343534466187319 | 343534437808150 | 343534162765863 | xfer-29788e04-6a5f-4a8e-8978-398a6daaa92b |
| 7 | 2.073684 | 1.602135 | 0.438767 | 1.752684 | 3818 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 261.321543 | 7 | 0 | 3818 | 239 | 227.201904 | 3818 | 106.44473629804281 | 375914496 | 96 | 28.252369 | 3818 | 343535656022234 | 343535915270093 | 343535917343777 | 343535884976822 | 343535914831326 | 343535886578957 | 343535657774918 | xfer-db2d7198-6d26-477d-ac10-871ec5ea5148 |
| 8 | 2.324107 | 1.562191 | 0.509916 | 1.838758 | 3802 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 259.489254 | 7 | 0 | 3802 | 238 | 226.118274 | 3802 | 110.36011840798396 | 374341632 | 96 | 27.136008 | 3802 | 343537102377332 | 343537359542479 | 343537361866586 | 343537330334364 | 343537359032563 | 343537331896555 | 343537104216090 | xfer-f19a4010-772a-43b2-90de-6c715413b021 |
| 9 | 2.148607 | 1.898994 | 0.719288 | 2.363097 | 15447 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1490.131939 | 7 | 0 | 15447 | 966 | 1368.83197 | 15447 | 106.46487520279301 | 1519386624 | 96 | 114.169983 | 15447 | 343540566337482 | 343542054320814 | 343542056469421 | 343541937532549 | 343542053601526 | 343541939431543 | 343540568700579 | xfer-ec05665e-ac9b-4832-becc-dca0db14483c |
| 10 | 1.8382 | 1.614592 | 0.637042 | 2.790011 | 15476 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1489.352525 | 7 | 0 | 15476 | 968 | 1369.766518 | 15476 | 108.07092176557302 | 1522532352 | 96 | 112.706162 | 15476 | 343543280564385 | 343544768078710 | 343544769916910 | 343544653120914 | 343544767441668 | 343544654735506 | 343543283354396 | xfer-3f66e703-8daf-4ddc-ba56-4109b68da614 |
| 11 | 1.651487 | 1.756148 | 0.616415 | 2.184321 | 15445 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1485.680928 | 7 | 0 | 15445 | 966 | 1365.595974 | 15445 | 106.73917913395768 | 1519386624 | 96 | 113.876583 | 15445 | 343545994724766 | 343547478754207 | 343547480405694 | 343547362505061 | 343547478137792 | 343547364261209 | 343545996909087 | xfer-7ae5e4c6-b064-4c57-8ac4-32393a61e587 |
| 12 | 4.486239 | 2.118211 | 0.759601 | 2.145119 | 4292 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 303.914144 | 7 | 0 | 4292 | 269 | 263.51647 | 4292 | 109.58132928677931 | 423100416 | 96 | 30.888504 | 4292 | 343550691814399 | 343550991242304 | 343550995728543 | 343550957475988 | 343550990482703 | 343550959594199 | 343550693959518 | xfer-92dd2241-80e7-4702-85ef-7912f7295b88 |
| 13 | 1.663587 | 1.230762 | 0.400916 | 2.194673 | 993 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 107.269862 | 7 | 4272 | 5265 | 330 | 94.334755 | 5265 | 106.47487733320762 | 99090432 | 96 | 7.445169 | 5265 | 343553031975478 | 343553137581753 | 343553139245340 | 343553128504906 | 343553137180837 | 343553129735668 | 343553034170151 | xfer-c72bb028-34b1-4360-8090-75271a6865b9 |
| 14 | 2.671227 | 1.545351 | 0.613908 | 2.21666 | 966 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 336.952311 | 7 | 4288 | 5254 | 329 | 322.664443 | 5254 | 106.00567622952518 | 95944704 | 96 | 7.240722 | 5254 | 343554329552947 | 343554663834031 | 343554666505258 | 343554654434050 | 343554663220123 | 343554655979401 | 343554331769607 | xfer-1c34ca70-07de-4cbc-830b-545e04a367ec |
| 15 | 1.692747 | 1.320752 | 0.393379 | 1.902381 | 973 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 104.54372 | 7 | 4288 | 5261 | 329 | 91.983782 | 5261 | 105.86010386061776 | 95944704 | 96 | 7.250679 | 5261 | 343555856846157 | 343555959697130 | 343555961389877 | 343555950732320 | 343555959303751 | 343555952053072 | 343555858748538 | xfer-0e598b68-c64a-48ae-a05f-4fce97806bda |
| 16 | 1.965025 | 1.466853 | 0.52114 | 2.64554 | 3797 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 584.692369 | 7 | 4288 | 8085 | 506 | 550.780198 | 8085 | 109.64250888375697 | 374341632 | 96 | 27.313613 | 8085 | 343559156048349 | 343559738775693 | 343559740740718 | 343559709474087 | 343559738254553 | 343559710940940 | 343559158693889 | xfer-9eb86222-d272-42f9-a52f-e88c05919600 |
| 17 | 1.923668 | 1.717304 | 0.47073 | 2.98248 | 3803 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 353.426929 | 7 | 4288 | 8091 | 506 | 319.055549 | 8091 | 109.78888139463592 | 374341632 | 96 | 27.277198 | 8091 | 343560936583031 | 343561288086292 | 343561290009960 | 343561258621060 | 343561287615562 | 343561260338364 | 343560939565511 | xfer-decfe301-4967-4e45-9ecc-e415b15b2ac7 |
| 18 | 2.89904 | 1.858012 | 0.797347 | 2.92545 | 3804 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 582.360418 | 7 | 4288 | 8092 | 506 | 546.83155 | 8092 | 110.71503391675684 | 374341632 | 96 | 27.049019 | 8092 | 343562488895366 | 343563068356744 | 343563071255784 | 343563038652366 | 343563067559397 | 343563040510378 | 343562491820816 | xfer-066515a0-7a07-482d-b30f-f34a5da32cb7 |
| 19 | 3.116643 | 1.557379 | 0.724347 | 2.912104 | 15438 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1858.749822 | 7 | 4288 | 19726 | 1233 | 1739.911126 | 19726 | 109.85890979175518 | 1517813760 | 96 | 110.528223 | 19726 | 343566288890759 | 343568144523938 | 343568147640581 | 343568031713989 | 343568143799591 | 343568033271368 | 343566291802863 | xfer-ae972fb5-5a36-4f29-8340-7db5b6dbb9a4 |
| 20 | 3.217173 | 2.021348 | 0.720628 | 2.820507 | 15446 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 2165.022861 | 7 | 4288 | 19734 | 1234 | 2045.068935 | 19734 | 109.33368837951441 | 1519386624 | 96 | 111.17427 | 19734 | 343569386870590 | 343571548676278 | 343571551893451 | 343571434760032 | 343571547955650 | 343571436781380 | 343569389691097 | xfer-b184470f-70cc-42b5-9da4-8d3d708c1057 |
| 21 | 3.291156 | 1.326134 | 1.574012 | 2.990072 | 15403 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1855.046099 | 7 | 4288 | 19691 | 1231 | 1735.169482 | 19691 | 109.46580835456497 | 1514668032 | 96 | 110.695243 | 19691 | 343572790365501 | 343574642120444 | 343574645411600 | 343574528525055 | 343574640546432 | 343574529851189 | 343572793355573 | xfer-3bcf9d3c-ca14-4c77-a6ed-fcbab5dc6eba |
| 22 | 2.516234 | 1.866632 | 0.728605 | 2.676584 | 13115 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1771.14403 | 7 | 4272 | 17387 | 1087 | 1668.710164 | 17387 | 109.01684639798798 | 1289748480 | 96 | 94.645811 | 17387 | 343577884306931 | 343579652934727 | 343579655450961 | 343579555693679 | 343579652206122 | 343579557560311 | 343577886983515 | xfer-024c8a47-9da6-41d2-80ba-e481953320a4 |
| 23 | 3.022996 | 2.496474 | 0.437547 | 3.427936 | 966 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 198.121311 | 7 | 17376 | 18342 | 1147 | 181.654195 | 18342 | 108.37898421710993 | 95944704 | 96 | 7.082163 | 18342 | 343581716140593 | 343581911238908 | 343581914261904 | 343581901222724 | 343581910801361 | 343581903719198 | 343581719568529 | xfer-4300630c-f889-4514-b34c-fef12a8925c2 |
| 24 | 2.374258 | 1.515796 | 0.537708 | 2.937961 | 966 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1866.282319 | 7 | 17392 | 18358 | 1148 | 1851.699694 | 18358 | 106.35555699661711 | 95944704 | 96 | 7.216902 | 18358 | 343583151547938 | 343585015455999 | 343585017830257 | 343585006185593 | 343585014918291 | 343585007701389 | 343583154485899 | xfer-4f994371-8c35-4c72-984d-ac4018a001b3 |
| 25 | 2.958061 | 2.137306 | 0.433389 | 2.643298 | 958 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 197.54015 | 7 | 17392 | 18350 | 1147 | 182.146298 | 18350 | 104.54110181425735 | 94371840 | 96 | 7.221798 | 18350 | 343586258882017 | 343586453464106 | 343586456422167 | 343586443671613 | 343586453030717 | 343586445808919 | 343586261525315 | xfer-cbef70fc-5e11-44fb-9f8b-3cbf4314f9ef |
| 26 | 2.650836 | 1.466464 | 0.881933 | 2.929782 | 3789 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 2265.525265 | 7 | 17392 | 21181 | 1324 | 2229.561205 | 21181 | 106.37222604779124 | 372768768 | 96 | 28.035045 | 21181 | 343589699430922 | 343591962305351 | 343591964956187 | 343591931921909 | 343591961423418 | 343591933388373 | 343589702360704 | xfer-947e1adb-7842-46d9-b865-70a07d6f909a |
| 27 | 3.017333 | 2.621991 | 0.471187 | 3.57172 | 3799 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 649.59573 | 7 | 17392 | 21191 | 1325 | 612.62619 | 21191 | 109.74820038135677 | 374341632 | 96 | 27.287309 | 21191 | 343593213139077 | 343593859717474 | 343593862734807 | 343593829336987 | 343593859246287 | 343593831958978 | 343593216710797 | xfer-00759ac2-addc-4716-9983-3d484f37fe85 |
| 28 | 2.83588 | 1.611923 | 0.642583 | 3.288654 | 3827 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 2269.97943 | 7 | 17392 | 21219 | 1327 | 2234.057575 | 21219 | 109.6437992993817 | 377487360 | 96 | 27.542815 | 21219 | 343595111474468 | 343597378618018 | 343597381453898 | 343597348820697 | 343597377975435 | 343597350432620 | 343595114763122 | xfer-ac31cb90-7b9f-4bfb-8a39-b83668c9c229 |
| 29 | 2.598281 | 2.064127 | 0.607766 | 3.3877 | 15416 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 3011.682064 | 7 | 17392 | 32808 | 2051 | 2892.258285 | 32808 | 109.50957488227085 | 1516240896 | 96 | 110.765905 | 32808 | 343600650677233 | 343603659761016 | 343603662359297 | 343603546323218 | 343603659153250 | 343603548387345 | 343600654064933 | xfer-aada3659-fde4-4458-a5bc-94c10ca07899 |
| 30 | 2.688431 | 1.357478 | 0.584424 | 4.345204 | 15433 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 4668.338106 | 7 | 17392 | 32825 | 2052 | 4548.741312 | 32825 | 109.76651693625213 | 1517813760 | 96 | 110.621257 | 32825 | 343604950470092 | 343609616119767 | 343609618808198 | 343609503556608 | 343609615535343 | 343609504914086 | 343604954815296 | xfer-b71c9bcd-2122-4bcb-b1d3-756106b9b20a |
| 31 | 2.675303 | 14.913805 | 0.747011 | 3.301697 | 15479 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 3074.489807 | 7 | 17392 | 32871 | 2055 | 2893.557117 | 32871 | 76.46359553289831 | 1522532352 | 55392 | 159.294874 | 32871 | 343610909483629 | 343613981298133 | 343613983973436 | 343613806342443 | 343613980551122 | 343613821256248 | 343610912785326 | xfer-43467105-6488-4bc9-a37b-2f650259524c |
| 32 | 2.87329 | 3.545183 | 0.429664 | 2.861884 | 976 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 84.420309 | 7 | 0 | 976 | 61 | 63.928877 | 976 | 71.1926882297688 | 95944704 | 5856 | 10.781411 | 976 | 343923141873975 | 343923223420994 | 343923226294284 | 343923208664736 | 343923222991330 | 343923212209919 | 343923144735859 | xfer-7e279c04-512f-4f08-8960-7ad2b6e33fcf |
| 33 | 3.138747 | 2.183392 | 0.409532 | 2.725953 | 965 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 84.64734 | 7 | 0 | 965 | 61 | 65.379252 | 965 | 71.00135868358657 | 95944704 | 5856 | 10.810464 | 965 | 343924400841889 | 343924482350482 | 343924485489229 | 343924468947094 | 343924481940950 | 343924471130486 | 343924403567842 | xfer-f2343b70-d024-4c83-96ff-590fc1a8a9a7 |
| 34 | 2.406195 | 2.445786 | 0.436976 | 2.704908 | 972 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 84.823867 | 7 | 0 | 972 | 61 | 65.885681 | 972 | 70.13296046415304 | 95944704 | 5856 | 10.944321 | 972 | 343925661522710 | 343925743940382 | 343925746346577 | 343925730113299 | 343925743503406 | 343925732559085 | 343925664227618 | xfer-bb2b31e5-4ff7-409c-976a-f76eea44b515 |
| 35 | 3.595002 | 2.322323 | 1.033232 | 2.746885 | 982 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 87.151933 | 7 | 0 | 982 | 62 | 66.350372 | 982 | 70.25686089999576 | 97517568 | 5952 | 11.104119 | 982 | 343926922550555 | 343927006107486 | 343927009702488 | 343926991647812 | 343927005074254 | 343926993970135 | 343926925297440 | xfer-1c4d0cd5-3371-48c3-9d43-fb19272b9346 |
| 36 | 3.594882 | 2.939953 | 0.441969 | 2.742435 | 972 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 85.059198 | 7 | 0 | 972 | 61 | 64.269469 | 972 | 69.33366382156527 | 95944704 | 5856 | 11.07049 | 972 | 343928186677060 | 343928268141376 | 343928271736258 | 343928253688964 | 343928267699407 | 343928256628917 | 343928189419495 | xfer-82f41e4a-ebce-435d-ae64-9d085385a474 |
| 37 | 4.232759 | 6.040593 | 1.062792 | 2.993056 | 3839 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 281.649207 | 7 | 0 | 3839 | 240 | 224.455213 | 3839 | 70.45172968753798 | 377487360 | 23040 | 42.864794 | 3839 | 343931451915366 | 343931729331814 | 343931733564573 | 343931679363635 | 343931728269022 | 343931685404228 | 343931454908422 | xfer-14c09ca0-184a-430c-848e-0bee7a90a832 |
| 38 | 3.615952 | 5.927463 | 0.697803 | 3.01182 | 3781 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 276.929258 | 7 | 0 | 3781 | 237 | 221.153959 | 3781 | 70.13150462530673 | 372768768 | 22752 | 42.522261 | 3781 | 343932919215085 | 343933192528391 | 343933196144343 | 343933143380864 | 343933191830588 | 343933149308327 | 343932922226905 | xfer-3237e223-ba7e-437d-923b-4ec77495c7f7 |
| 39 | 2.447002 | 5.858704 | 0.495866 | 2.810846 | 3811 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 279.094026 | 7 | 0 | 3811 | 239 | 224.652015 | 3811 | 70.21584277020796 | 375914496 | 22944 | 42.829593 | 3811 | 343934384634236 | 343934661281260 | 343934663728262 | 343934612097097 | 343934660785394 | 343934617955801 | 343934387445082 | xfer-caee6ddc-a766-43bf-a77e-04052215a01f |
| 40 | 3.64398 | 5.597882 | 0.822779 | 2.946601 | 3815 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 279.546476 | 7 | 0 | 3815 | 239 | 223.590559 | 3815 | 70.02768021879314 | 375914496 | 22944 | 42.944675 | 3815 | 343935849174273 | 343936125076769 | 343936128720749 | 343936075711433 | 343936124253990 | 343936081309315 | 343935852120874 | xfer-f7a389fd-d54f-40fe-8127-9ba513e84e9d |
| 41 | 3.158833 | 5.423762 | 0.480652 | 3.399984 | 3810 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 278.521197 | 7 | 0 | 3810 | 239 | 223.086016 | 3810 | 69.98323250399388 | 375914496 | 22944 | 42.97195 | 3810 | 343937317705345 | 343937593067709 | 343937596226542 | 343937544191345 | 343937592587057 | 343937549615107 | 343937321105329 | xfer-dba542a7-c091-40c3-9256-9052733c5d24 |
| 42 | 2.802117 | 23.002515 | 0.720612 | 3.127717 | 15409 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1574.36384 | 7 | 0 | 15409 | 964 | 1360.445676 | 15409 | 65.8286370433163 | 1516240896 | 92544 | 184.265203 | 15409 | 343940799169801 | 343942370731524 | 343942373533641 | 343942162743194 | 343942370010912 | 343942185745709 | 343940802297518 | xfer-da6c5ecc-3e4b-43b1-9912-f3469c3d7cce |
| 43 | 2.758942 | 21.13861 | 0.691324 | 3.957685 | 15409 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1573.549032 | 7 | 0 | 15409 | 964 | 1360.44069 | 15409 | 65.72285498263587 | 1516240896 | 92544 | 184.561781 | 15409 | 343943602871270 | 343945173661360 | 343945176420302 | 343944967269645 | 343945172970036 | 343944988408255 | 343943606828955 | xfer-c84e0b7e-add7-46f0-b56e-324f69881210 |
| 44 | 3.382336 | 20.296695 | 0.817298 | 4.003918 | 15406 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1561.557037 | 7 | 0 | 15406 | 963 | 1359.195631 | 15406 | 69.69552213786865 | 1514668032 | 92448 | 173.861159 | 15406 | 343946402149766 | 343947960324467 | 343947963706803 | 343947765349315 | 343947959507169 | 343947785646010 | 343946406153684 | xfer-4b2cb6e7-14c2-464b-90f3-0fe2f69eb9f5 |
| 45 | 4.027914 | 20.186346 | 0.851076 | 4.357565 | 15423 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1561.46888 | 7 | 0 | 15423 | 964 | 1360.537359 | 15423 | 70.7248834956517 | 1516240896 | 92544 | 171.50862 | 15423 | 343949189682339 | 343950747123305 | 343950751151219 | 343950554577263 | 343950746272229 | 343950574763609 | 343949194039904 | xfer-8149fb3f-0181-45a9-96a1-71e3b5136cef |
| 46 | 3.404981 | 19.958405 | 0.599572 | 4.462838 | 15407 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1556.477919 | 7 | 0 | 15407 | 963 | 1357.418939 | 15407 | 71.0139960583517 | 1514668032 | 92448 | 170.633184 | 15407 | 343951975310914 | 343953528383852 | 343953531788833 | 343953337192691 | 343953527784280 | 343953357151096 | 343951979773752 | xfer-0d61490c-f4d1-41d8-a5c6-d3c3cb7458a2 |
| 47 | 2.78986 | 2.343292 | 0.424066 | 4.00012 | 11 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 187.404757 | 7 | 2128 | 2139 | 134 | 177.496682 | 2139 | 35.87563330928873 | 1572864 | 96 | 0.350737 | 2139 | 343956737870164 | 343956922485061 | 343956925274921 | 343956919366966 | 343956922060995 | 343956921710258 | 343956741870284 | xfer-d75ad3c4-ddd5-45e6-aede-e215396b93a0 |
| 48 | 3.537035 | 2.412897 | 0.449823 | 3.840924 | 978 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 97.132421 | 7 | 2128 | 3106 | 195 | 75.942308 | 3106 | 71.24939462624278 | 97517568 | 5952 | 10.949434 | 3106 | 343958952635219 | 343959046230605 | 343959049767640 | 343959032418451 | 343959045780782 | 343959034831348 | 343958956476143 | xfer-2b154013-682e-4689-a07c-a5b36b18117a |
| 49 | 3.954325 | 2.724032 | 1.589472 | 4.043707 | 959 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 249.556472 | 7 | 2144 | 3103 | 194 | 226.668644 | 3103 | 71.38368721287195 | 94371840 | 5760 | 10.576292 | 3103 | 343960234489131 | 343960480091278 | 343960484045603 | 343960465201482 | 343960478501806 | 343960467925514 | 343960238532838 | xfer-e517397a-fa5e-4c9f-9938-bbc0c04a8a05 |
| 50 | 3.054987 | 2.706184 | 0.439188 | 3.920979 | 952 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 98.80282 | 7 | 2144 | 3096 | 194 | 77.537773 | 3096 | 67.74896221715768 | 94371840 | 5760 | 11.143709 | 3096 | 343961669217996 | 343961764965829 | 343961768020816 | 343961750676748 | 343961764526641 | 343961753382932 | 343961673138975 | xfer-e45ed7f3-8a58-4b93-ad80-501bac02d42e |
| 51 | 3.911436 | 2.397406 | 0.543541 | 3.83426 | 980 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 200.252808 | 7 | 2128 | 3108 | 195 | 178.553877 | 3108 | 70.84272986685419 | 97517568 | 5952 | 11.012288 | 3108 | 343962953041906 | 343963149383278 | 343963153294714 | 343963135430043 | 343963148839737 | 343963137827449 | 343962956876166 | xfer-3fc542a4-c451-4af6-af54-dc2a36f24f2c |
| 52 | 1.329255 | 3.026022 | 0.999154 | 3.960324 | 956 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 96.933447 | 7 | 2128 | 3084 | 193 | 76.607202 | 3084 | 68.56244886023599 | 94371840 | 5760 | 11.01149 | 3084 | 343964337590244 | 343964433194436 | 343964434523691 | 343964418157770 | 343964432195282 | 343964421183792 | 343964341550568 | xfer-b2aca2e7-4875-427e-9f0b-d317aebaae0d |
| 53 | 2.955891 | 5.995731 | 0.529785 | 3.861272 | 3786 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 427.15141 | 7 | 2144 | 5930 | 371 | 371.196088 | 5930 | 69.98275474253029 | 372768768 | 22752 | 42.612643 | 5930 | 343967625545250 | 343968049740769 | 343968052696660 | 343968000602610 | 343968049210984 | 343968006598341 | 343967629406522 | xfer-c6998520-9685-4641-8bde-20d71a9eeb76 |
| 54 | 3.94437 | 6.033872 | 1.209924 | 3.92171 | 3803 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 327.251599 | 7 | 2144 | 5947 | 372 | 269.706977 | 5947 | 70.57266363748236 | 374341632 | 22848 | 42.434746 | 5947 | 343969245405347 | 343969568712576 | 343969572656946 | 343969519034034 | 343969567502652 | 343969525067906 | 343969249327057 | xfer-ab43d7a9-3560-4de7-8810-f85caad5105f |
| 55 | 3.942834 | 5.737844 | 0.645133 | 3.916515 | 3799 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 426.730381 | 7 | 2144 | 5943 | 372 | 370.10648 | 5943 | 70.66120256267023 | 374341632 | 22848 | 42.381575 | 5943 | 343970764516949 | 343971187304496 | 343971191247330 | 343971138539944 | 343971186659363 | 343971144277788 | 343970768433464 | xfer-96925457-781c-4da1-a02d-9964d3218248 |
| 56 | 2.818408 | 5.958399 | 0.482084 | 3.919752 | 3771 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 321.743945 | 7 | 2144 | 5915 | 370 | 266.54489 | 5915 | 70.66963627105798 | 371195904 | 22656 | 42.020412 | 5915 | 343972388100818 | 343972707026355 | 343972709844763 | 343972658565460 | 343972706544271 | 343972664523859 | 343972392020570 | xfer-9abc49fc-9fe8-44d3-9475-9ebfeb404f2e |
| 57 | 2.815404 | 5.865107 | 0.536795 | 3.900765 | 3789 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 426.210635 | 7 | 2144 | 5933 | 371 | 370.298539 | 5933 | 69.68613361327895 | 372768768 | 22752 | 42.794025 | 5933 | 343973904232146 | 343974327627377 | 343974330442781 | 343974278431450 | 343974327090582 | 343974284296557 | 343973908132911 | xfer-5b9bd505-a4eb-4f4c-9777-89004834d643 |
| 58 | 2.811201 | 21.427328 | 0.77726 | 4.507936 | 15437 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1754.970166 | 7 | 2128 | 17565 | 1098 | 1547.29026 | 17565 | 68.1565467548948 | 1517813760 | 92640 | 178.156181 | 17565 | 343977546544054 | 343979298703019 | 343979301514220 | 343979098342250 | 343979297925759 | 343979119769578 | 343977551051990 | xfer-1fde94a8-2b65-4ecf-aa50-6ce3ad65902a |
| 59 | 3.296259 | 19.253188 | 0.795063 | 6.392683 | 15393 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1874.860712 | 7 | 2144 | 17537 | 1097 | 1673.955666 | 17537 | 70.79217296719845 | 1514668032 | 92448 | 171.167853 | 17537 | 343980534882389 | 343982406446842 | 343982409743101 | 343982215230738 | 343982405651779 | 343982234483926 | 343980541275072 | xfer-d403b6b0-8765-409e-98bb-d709760a156a |
| 60 | 3.934183 | 20.818748 | 1.699325 | 4.649118 | 15435 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1751.325849 | 7 | 2128 | 17563 | 1098 | 1545.890274 | 17563 | 69.65076278979821 | 1517813760 | 92640 | 174.334201 | 17563 | 343983647638965 | 343985395030631 | 343985398964814 | 343985198178357 | 343985393331306 | 343985218997105 | 343983652288083 | xfer-7d521add-3226-4625-846c-09f2e099ea4b |
| 61 | 3.941944 | 20.143961 | 1.882404 | 4.569059 | 15444 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1879.360201 | 7 | 2144 | 17588 | 1100 | 1676.502591 | 17588 | 70.5378129169526 | 1519386624 | 92736 | 172.320242 | 17588 | 343986638163124 | 343988513581381 | 343988517523325 | 343988319234774 | 343988511698977 | 343988339378735 | 343986642732183 | xfer-abcde13a-0c68-47c7-9900-47d203989fa1 |
| 62 | 3.204475 | 20.38466 | 0.600685 | 4.740413 | 15404 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1745.632218 | 7 | 2144 | 17548 | 1097 | 1545.117825 | 17548 | 70.62041307309487 | 1514668032 | 92448 | 171.58416 | 17548 | 343989752326392 | 343991494754135 | 343991497958610 | 343991302184630 | 343991494153450 | 343991322569290 | 343989757066805 | xfer-4753edc9-b787-4fff-823e-0c5a2b43eb7c |
| 63 | 4.049909 | 0.862976 | 1.757309 | 4.010888 | 16 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 631.17912 | 7 | 8624 | 8640 | 540 | 620.132294 | 8640 | 34.403604707117545 | 1572864 | 96 | 0.365744 | 8640 | 343994723546776 | 343995350675987 | 343995354725896 | 343995347689958 | 343995348918678 | 343995348552934 | 343994727557664 | xfer-b72a0aee-2779-4405-91ed-600357c3af26 |
| 64 | 3.257195 | 2.892717 | 0.413018 | 3.929762 | 991 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 142.99823 | 7 | 8624 | 9615 | 601 | 121.500199 | 9615 | 70.88746144030638 | 97517568 | 5952 | 11.005339 | 9615 | 343997397668131 | 343997537409166 | 343997540666361 | 343997523098092 | 343997536996148 | 343997525990809 | 343997401597893 | xfer-5167f7bc-f5a6-4764-8aea-d41b821d3746 |
| 65 | 4.005567 | 2.234935 | 0.817449 | 3.95562 | 972 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 784.81566 | 7 | 8640 | 9612 | 601 | 762.926825 | 9612 | 70.57829878888458 | 95944704 | 5856 | 10.875264 | 9612 | 343998749758108 | 343999530568201 | 343999534573768 | 343999516640553 | 343999529750752 | 343999518875488 | 343998753713728 | xfer-3e0b7202-d03b-40b2-bad9-12ece5886c86 |
| 66 | 3.998898 | 2.781587 | 1.154842 | 4.052568 | 954 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 142.964138 | 7 | 8640 | 9594 | 600 | 120.31907 | 9594 | 70.84193153287463 | 94371840 | 5760 | 10.657173 | 9594 | 344000750974619 | 344000889939859 | 344000893938757 | 344000875346257 | 344000888785017 | 344000878127844 | 344000755027187 | xfer-3355bc7d-537a-4c50-9f47-29475ef20df3 |
| 67 | 3.503411 | 2.430847 | 0.497237 | 3.941937 | 967 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 756.368633 | 7 | 8640 | 9607 | 601 | 735.281575 | 9607 | 71.64312362593205 | 95944704 | 5856 | 10.713626 | 9607 | 344002102827857 | 344002855693079 | 344002859196490 | 344002842051369 | 344002855195842 | 344002844482216 | 344002106769794 | xfer-bb87cadd-1a12-4848-9105-e1e8fa07b45c |
| 68 | 4.019339 | 3.110151 | 1.261111 | 4.112686 | 963 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 143.047517 | 7 | 8640 | 9603 | 601 | 119.738752 | 9603 | 71.03412102639051 | 95944704 | 5856 | 10.805478 | 9603 | 344004069169337 | 344004208197515 | 344004212216854 | 344004193020775 | 344004206936404 | 344004196130926 | 344004073282023 | xfer-a16f34ad-e1d4-47ca-a310-64f98e42ba93 |
| 69 | 3.4453 | 5.497828 | 0.697261 | 4.130356 | 3813 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1064.857818 | 7 | 8640 | 12453 | 779 | 1007.327189 | 12453 | 68.72312476879509 | 375914496 | 22944 | 43.759884 | 12453 | 344007425300610 | 344008486713128 | 344008490158428 | 344008436758155 | 344008486015867 | 344008442255983 | 344007429430966 | xfer-03c4a7dc-fee5-494d-9291-059261244a55 |
| 70 | 4.056958 | 6.556157 | 0.735644 | 4.325568 | 3816 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 474.809695 | 7 | 8640 | 12456 | 779 | 416.461503 | 12456 | 70.47207858955359 | 375914496 | 22944 | 42.673865 | 12456 | 344009709045447 | 344010179798184 | 344010183855142 | 344010129832518 | 344010179062540 | 344010136388675 | 344009713371015 | xfer-66a3e89b-0ef6-47a3-bf1d-4d651b1467d7 |
| 71 | 2.792053 | 5.769235 | 0.568306 | 4.276771 | 3795 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1062.91823 | 7 | 8640 | 12435 | 778 | 1007.108164 | 12435 | 70.62433196574045 | 374341632 | 22848 | 42.403701 | 12435 | 344011403523025 | 344012463649202 | 344012466441255 | 344012414907960 | 344012463080896 | 344012420677195 | 344011407799796 | xfer-f6f81412-2410-45ff-a344-1e7dcab42dd3 |
| 72 | 2.93636 | 6.721208 | 0.607545 | 4.532909 | 3820 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 475.312769 | 7 | 8640 | 12460 | 779 | 417.75525 | 12460 | 70.33094818678526 | 375914496 | 22944 | 42.759497 | 12460 | 344013684813214 | 344014157189623 | 344014160125983 | 344014107101373 | 344014156582078 | 344014113822581 | 344013689346123 | xfer-2f217b73-f6a7-4364-8e22-9ea686cf03fc |
| 73 | 3.983707 | 5.57057 | 0.871657 | 4.170471 | 3793 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1062.286824 | 7 | 8640 | 12433 | 778 | 1005.143341 | 12433 | 70.38633901016657 | 374341632 | 22848 | 42.547078 | 12433 | 344015377875151 | 344016436178268 | 344016440161975 | 344016387188963 | 344016435306611 | 344016392759533 | 344015382045622 | xfer-afe23d6a-d173-4ddd-9f4f-5c667ad3256f |
| 74 | 3.922147 | 21.524908 | 2.015128 | 4.809325 | 15407 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 2357.084348 | 7 | 8640 | 24047 | 1503 | 2116.110056 | 24047 | 58.06029044634115 | 1514668032 | 92448 | 208.702784 | 24047 | 344019678368411 | 344022031530612 | 344022035452759 | 344021799287792 | 344022029515484 | 344021820812700 | 344019683177736 | xfer-33003a8f-b840-4b7e-92e0-b86117e9c987 |
| 75 | 3.856647 | 23.906271 | 0.719083 | 4.93606 | 15438 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 2926.607498 | 7 | 8640 | 24078 | 1505 | 2718.446051 | 24078 | 69.48766621702066 | 1517813760 | 92640 | 174.743386 | 24078 | 344023293025231 | 344026215776082 | 344026219632729 | 344026016407342 | 344026215056999 | 344026040313613 | 344023297961291 | xfer-8de0b399-c504-4f07-a966-d52231d9d9b8 |
| 76 | 2.777104 | 21.427507 | 0.702439 | 4.719515 | 15396 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 2317.351766 | 7 | 8640 | 24036 | 1503 | 2116.281928 | 24036 | 70.67844683529812 | 1514668032 | 92448 | 171.443273 | 24036 | 344027477037341 | 344029791612003 | 344029794389107 | 344029598038784 | 344029790909564 | 344029619466291 | 344027481756856 | xfer-6bb9c369-7d98-476d-8c10-8a55a3b637be |
| 77 | 3.430388 | 21.708904 | 0.676655 | 5.76684 | 15384 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 2916.354688 | 7 | 8640 | 24024 | 1502 | 2712.008868 | 24024 | 70.06569133340001 | 1513095168 | 92352 | 172.763033 | 24024 | 344031054178050 | 344033967102350 | 344033970532738 | 344033771953758 | 344033966425695 | 344033793662662 | 344031059944890 | xfer-9622617c-afc4-4f3d-acab-617802d71b42 |
| 78 | 4.029216 | 20.945537 | 1.673896 | 5.401789 | 15458 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 2337.444843 | 7 | 8640 | 24098 | 1507 | 2121.993214 | 24098 | 66.34458499236246 | 1520959488 | 92832 | 183.401191 | 24098 | 344035231741558 | 344037565157185 | 344037569186401 | 344037359136561 | 344037563483289 | 344037380082098 | 344035237143347 | xfer-4ee6ddf8-3538-41e0-ac9a-13e25dd6c781 |
| 79 | 3.933837 | 5.631332 | 26286 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_zmq_received | 5418.119478 | 5 | 8624 | 34910 | 2182 | 5023.187616 | 34910 | 34910 | 344040846080028 | 344046260265669 | 344046264199506 | 344045874898976 | 344040851711360 | xfer-b95f41e4-627a-4d9e-aa65-b98afafb06c0 | ||||||||
| 80 | 4.062967 | 5.096194 | 1.244185 | 4.667851 | 988 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 330.333453 | 7 | 34896 | 35884 | 2243 | 304.252371 | 35884 | 70.85819188847113 | 97517568 | 5952 | 11.009885 | 35884 | 344048361285763 | 344048687556249 | 344048691619216 | 344048670205985 | 344048686312064 | 344048675302179 | 344048365953614 | xfer-204c5e89-95b3-4a59-8bbb-c03a1eac864c |
| 81 | 2.933318 | 6.931342 | 984 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_zmq_received | 5330.989922 | 5 | 34912 | 35896 | 2244 | 5306.482294 | 35896 | 35896 | 344049993993743 | 344055322050347 | 344055324983665 | 344055307407379 | 344050000925085 | xfer-a530e3ee-0c55-4a85-b988-110fd1cd3e5f | ||||||||
| 82 | 4.099607 | 4.522637 | 1.671787 | 4.717849 | 964 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 327.05328 | 7 | 34912 | 35876 | 2243 | 301.171991 | 35876 | 70.61631704170853 | 95944704 | 5856 | 10.869409 | 35876 | 344056625957769 | 344056948911442 | 344056953011049 | 344056931847609 | 344056947239655 | 344056936370246 | 344056630675618 | xfer-972f5586-7436-4a56-b437-a504e6b94fc3 |
| 83 | 3.841064 | 4.577476 | 952 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_zmq_received | 5270.728238 | 5 | 34912 | 35864 | 2242 | 5247.707635 | 35864 | 35864 | 344058254091725 | 344063520978899 | 344063524819963 | 344063506376836 | 344058258669201 | xfer-35222c64-1117-4742-8ff8-9cf890498f1d | ||||||||
| 84 | 4.324731 | 5.236853 | 0.604215 | 4.951784 | 975 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 327.583453 | 7 | 34912 | 35887 | 2243 | 301.618389 | 35887 | 70.75906673632338 | 95944704 | 5856 | 10.847481 | 35887 | 344064830066353 | 344065153325075 | 344065157649806 | 344065136636526 | 344065152720860 | 344065141873379 | 344064835018137 | xfer-c979d8d1-17a5-4624-b46b-b19da335f930 |
| 85 | 4.054432 | 5.409207 | 3816 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_zmq_received | 6065.42777 | 5 | 34912 | 38728 | 2421 | 6003.743773 | 38728 | 38728 | 344068478934114 | 344074540307452 | 344074544361884 | 344074488087094 | 344068484343321 | xfer-922060d3-5606-41bd-a728-cb9286104c5f | ||||||||
| 86 | 4.161682 | 11.759598 | 0.912837 | 4.801053 | 3787 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1067.077186 | 7 | 34912 | 38699 | 2419 | 1002.993495 | 38699 | 70.25333448013419 | 372768768 | 22752 | 42.448521 | 38699 | 344075859994221 | 344076922909725 | 344076927071407 | 344076867788769 | 344076921996888 | 344076879548367 | 344075864795274 | xfer-989b23c9-0a4a-4f61-be7b-f47c2cd5c68c |
| 87 | 4.462714 | 4.792675 | 3802 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_zmq_received | 6058.89703 | 5 | 34912 | 38714 | 2420 | 5998.51813 | 38714 | 38714 | 344078241308449 | 344084295742765 | 344084300205479 | 344084244619254 | 344078246101124 | xfer-6d52cb9b-7e69-498a-a14d-c4ef18466455 | ||||||||
| 88 | 3.990764 | 8.428795 | 0.813159 | 4.765448 | 3803 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 1065.965013 | 7 | 34912 | 38715 | 2420 | 1005.500559 | 38715 | 70.52024551804482 | 374341632 | 22848 | 42.466288 | 38715 | 344085614679887 | 344086676654136 | 344086680644900 | 344086624945894 | 344086675840977 | 344086633374689 | 344085619445335 | xfer-2f329fc6-dce1-40e9-bd89-6afb67f09a63 |
| 89 | 2.980913 | 4.953564 | 3799 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_zmq_received | 6055.683733 | 5 | 34912 | 38711 | 2420 | 5996.9245 | 38711 | 38711 | 344087991204865 | 344094043907685 | 344094046888598 | 344093993082929 | 344087996158429 | xfer-2d30d1c4-5c17-4da2-88a1-c098e973f146 | ||||||||
| 90 | 3.278995 | 21.052319 | 0.651839 | 5.371544 | 15414 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 4632.479244 | 7 | 34912 | 50326 | 3146 | 4431.768172 | 50326 | 71.20324770939743 | 1516240896 | 92544 | 170.356375 | 50326 | 344097383047854 | 344102012248103 | 344102015527098 | 344101820187570 | 344102011596264 | 344101841239889 | 344097388419398 | xfer-3d9b6f40-000a-499b-9eeb-f2cd0158e3cc |
| 91 | 4.210185 | 5.418244 | 15459 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_zmq_received | 9666.724654 | 5 | 34912 | 50371 | 3149 | 9455.88659 | 50371 | 50371 | 344103365721735 | 344113028236204 | 344113032446389 | 344112827026569 | 344103371139979 | xfer-6e5b9dfc-5791-483e-bd58-ca0a820f6f20 | ||||||||
| 92 | 3.171092 | 22.117254 | 0.897336 | 5.724271 | 15402 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 4630.020597 | 7 | 34912 | 50314 | 3145 | 4426.413397 | 50314 | 70.57389951045633 | 1514668032 | 92448 | 171.697247 | 50314 | 344114386617482 | 344119013466987 | 344119016638079 | 344118818755150 | 344119012569651 | 344118840872404 | 344114392341753 | xfer-9194c65d-89ee-4cfe-9403-01c80df0690e |
| 93 | 4.01791 | 5.373968 | 15504 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_zmq_received | 9744.213969 | 5 | 34912 | 50416 | 3151 | 9537.277953 | 50416 | 50416 | 344120376076315 | 344130116272374 | 344130120290284 | 344129918728236 | 344120381450283 | xfer-b40b0fbf-9f16-4040-bbb7-867c5e4960c7 | ||||||||
| 94 | 3.188845 | 20.508939 | 0.621964 | 5.709689 | 15456 | D_get_num_matched,D_recv_complete,D_request_promoted,P_prefill_done,P_rdma_end,P_rdma_start,P_zmq_received | 4602.182752 | 7 | 34912 | 50368 | 3148 | 4399.785916 | 50368 | 70.51851488459253 | 1519386624 | 92736 | 172.367399 | 50368 | 344131502136745 | 344136101130652 | 344136104319497 | 344135907632350 | 344136100508688 | 344135928141289 | 344131507846434 | xfer-1d314942-867e-4677-bd2c-f920c7aaf618 |