gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
agentic-kvc/analysis/mb2/B_intra_kvboth.jsonl
Gahow Wang de164e5a64 MB2: pure KV-transfer cost on dash1 intra-node — Mooncake ~9.7 GB/s steady 
Full sweep result on dash1 GPU 0+1 with vanilla vLLM 0.18.1 +
mooncake-transfer-engine 0.3.11, kv_both connector. Per-stage decomposition
via the instrumentation patch (analyze_mb2.py pairs A's send_blocks with
B's receive_kv enter/finish by time window).

Steady-state (1k..32k tokens, 96 MiB..3 GiB KV):
   pure_transfer ≈ size / 9.7 GB/s
   rx_overhead   ≈ 2–3 ms (ZMQ handshake + P-side setup)
   bandwidth     ≈ 9.6–10.1 GB/s, very stable

Large-size regime (65k..131k tokens, 6..12 GiB):
   p50 bandwidth collapses to 3.4–4.5 GB/s
   max bandwidth still hits ~9.7 GB/s (some runs achieve it)
   p99 agentic request (11.5 GiB) lands here

Implication for §3.2 PD-disaggregation cost argument:
   median agentic decode = 50–200 ms (tool-call JSON output)
   median agentic-tail KV transfer (p99 11.5 GiB):
     best case (9.7 GB/s)  ≈ 1.19 s
     observed range         1.5 – 10 s
   ⇒ KV transfer is 8–100× larger than the decode it enables.

This is intra-node — the lower-bound transfer cost. Inter-node RDMA
will be slower; that's MB2 phase 2.

Adds:
- analyze_mb2.py: pair A.send_blocks ↔ B.receive_kv by time window;
  per-size aggregation (n, ms_p50, ms_min/max, GB/s_p50/max)
- plot_mb2.py: log-log transfer-time chart + bandwidth-vs-size chart
- analysis/mb2/A_intra_kvboth.jsonl, B_intra_kvboth.jsonl: raw events
  (51 + 102 events including the sanity preamble)
- analysis/mb2/intra_kvboth_breakdown.json: paired and aggregated
- figs/mb2_transfer_time_intra.png, figs/mb2_transfer_bw_intra.png

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 19:04:03 +08:00

103 lines
22 KiB
JSON
Raw Permalink Blame History

{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ad00672f263a6643-0-9479211a"], "t_start_unix": 1779879143.1678784, "tp_rank": 0, "t_log_unix": 1779879143.167884}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ad00672f263a6643-0-9479211a"], "duration_s": 0.03333390498301014, "t_start_unix": 1779879143.1678784, "tp_rank": 0, "t_log_unix": 1779879143.201217}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ace77e2b02f9f141-0-b3c061bc"], "t_start_unix": 1779879143.2968972, "tp_rank": 0, "t_log_unix": 1779879143.2969005}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ace77e2b02f9f141-0-b3c061bc"], "duration_s": 0.007019245007541031, "t_start_unix": 1779879143.2968972, "tp_rank": 0, "t_log_unix": 1779879143.3039184}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4a2366879c68ded-0-8ac4098e"], "t_start_unix": 1779879143.5146625, "tp_rank": 0, "t_log_unix": 1779879143.5146651}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4a2366879c68ded-0-8ac4098e"], "duration_s": 0.02278437599306926, "t_start_unix": 1779879143.5146625, "tp_rank": 0, "t_log_unix": 1779879143.537448}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8690cafcace0d5e2-0-b89f33d2"], "t_start_unix": 1779879143.6958342, "tp_rank": 0, "t_log_unix": 1779879143.6958375}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8690cafcace0d5e2-0-b89f33d2"], "duration_s": 0.022794076008722186, "t_start_unix": 1779879143.6958342, "tp_rank": 0, "t_log_unix": 1779879143.71863}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b087e2ec4cfa8eb7-0-b908f425"], "t_start_unix": 1779879144.3279662, "tp_rank": 0, "t_log_unix": 1779879144.3279696}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b087e2ec4cfa8eb7-0-b908f425"], "duration_s": 0.08753501297906041, "t_start_unix": 1779879144.3279662, "tp_rank": 0, "t_log_unix": 1779879144.415505}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a115d16ff5575e08-0-9fa81984"], "t_start_unix": 1779879145.040141, "tp_rank": 0, "t_log_unix": 1779879145.0401456}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a115d16ff5575e08-0-9fa81984"], "duration_s": 0.0860149699728936, "t_start_unix": 1779879145.040141, "tp_rank": 0, "t_log_unix": 1779879145.1261594}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9e585ed083951df5-0-b03f812b"], "t_start_unix": 1779879221.7062025, "tp_rank": 0, "t_log_unix": 1779879221.7062056}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9e585ed083951df5-0-b03f812b"], "duration_s": 0.002459956973325461, "t_start_unix": 1779879221.7062025, "tp_rank": 0, "t_log_unix": 1779879221.7086644}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9271d403c044eadd-0-9c3c4639"], "t_start_unix": 1779879221.7826598, "tp_rank": 0, "t_log_unix": 1779879221.782662}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9271d403c044eadd-0-9c3c4639"], "duration_s": 0.0020201010047458112, "t_start_unix": 1779879221.7826598, "tp_rank": 0, "t_log_unix": 1779879221.7846813}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82a580cefd3e2440-0-a383c3c4"], "t_start_unix": 1779879221.859549, "tp_rank": 0, "t_log_unix": 1779879221.8595514}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82a580cefd3e2440-0-a383c3c4"], "duration_s": 0.006836243963334709, "t_start_unix": 1779879221.859549, "tp_rank": 0, "t_log_unix": 1779879221.8663864}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a31cb4bc9e7f63d2-0-8f48aacd"], "t_start_unix": 1779879221.9419758, "tp_rank": 0, "t_log_unix": 1779879221.9419782}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a31cb4bc9e7f63d2-0-8f48aacd"], "duration_s": 0.00694335694424808, "t_start_unix": 1779879221.9419758, "tp_rank": 0, "t_log_unix": 1779879221.9489205}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a9dfc1a5b425d994-0-a0930098"], "t_start_unix": 1779879222.0232244, "tp_rank": 0, "t_log_unix": 1779879222.0232272}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a9dfc1a5b425d994-0-a0930098"], "duration_s": 0.006697195000015199, "t_start_unix": 1779879222.0232244, "tp_rank": 0, "t_log_unix": 1779879222.0299227}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9712857755af2efc-0-90b2dc9b"], "t_start_unix": 1779879222.1297998, "tp_rank": 0, "t_log_unix": 1779879222.1298025}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9712857755af2efc-0-90b2dc9b"], "duration_s": 0.01183948403922841, "t_start_unix": 1779879222.1297998, "tp_rank": 0, "t_log_unix": 1779879222.1416407}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4f0a10dee65acbe-0-a3c132fc"], "t_start_unix": 1779879222.243023, "tp_rank": 0, "t_log_unix": 1779879222.243025}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4f0a10dee65acbe-0-a3c132fc"], "duration_s": 0.01214482297655195, "t_start_unix": 1779879222.243023, "tp_rank": 0, "t_log_unix": 1779879222.2551687}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4c514b80b52a3f2-0-bcd24f8e"], "t_start_unix": 1779879222.3569698, "tp_rank": 0, "t_log_unix": 1779879222.356972}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b4c514b80b52a3f2-0-bcd24f8e"], "duration_s": 0.011961110983975232, "t_start_unix": 1779879222.3569698, "tp_rank": 0, "t_log_unix": 1779879222.368932}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ac7118d8090d181c-0-8af4adf0"], "t_start_unix": 1779879222.4715128, "tp_rank": 0, "t_log_unix": 1779879222.4715152}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-ac7118d8090d181c-0-8af4adf0"], "duration_s": 0.011788576026447117, "t_start_unix": 1779879222.4715128, "tp_rank": 0, "t_log_unix": 1779879222.4833028}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-85291bcb93aaf638-0-868db1a8"], "t_start_unix": 1779879222.5826046, "tp_rank": 0, "t_log_unix": 1779879222.5826073}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-85291bcb93aaf638-0-868db1a8"], "duration_s": 0.0118055299972184, "t_start_unix": 1779879222.5826046, "tp_rank": 0, "t_log_unix": 1779879222.5944116}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a448cf2e059ba0c9-0-a1360796"], "t_start_unix": 1779879222.750828, "tp_rank": 0, "t_log_unix": 1779879222.7508304}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a448cf2e059ba0c9-0-a1360796"], "duration_s": 0.0021119200391694903, "t_start_unix": 1779879222.750828, "tp_rank": 0, "t_log_unix": 1779879222.7529414}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b486fd9e945a4658-0-8bb561cd"], "t_start_unix": 1779879222.913044, "tp_rank": 0, "t_log_unix": 1779879222.9130466}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b486fd9e945a4658-0-8bb561cd"], "duration_s": 0.0022232600022107363, "t_start_unix": 1779879222.913044, "tp_rank": 0, "t_log_unix": 1779879222.9152684}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82da2bfe65f276c6-0-88d9a9a2"], "t_start_unix": 1779879223.0765986, "tp_rank": 0, "t_log_unix": 1779879223.0766027}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-82da2bfe65f276c6-0-88d9a9a2"], "duration_s": 0.022250515001360327, "t_start_unix": 1779879223.0765986, "tp_rank": 0, "t_log_unix": 1779879223.0988505}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-93bd777652eba5f3-0-9ec3d058"], "t_start_unix": 1779879223.2591784, "tp_rank": 0, "t_log_unix": 1779879223.2591808}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-93bd777652eba5f3-0-9ec3d058"], "duration_s": 0.022157608007546514, "t_start_unix": 1779879223.2591784, "tp_rank": 0, "t_log_unix": 1779879223.2813375}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81f950480a3cabf9-0-bbf8584f"], "t_start_unix": 1779879223.4402068, "tp_rank": 0, "t_log_unix": 1779879223.440209}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81f950480a3cabf9-0-bbf8584f"], "duration_s": 0.022589912987314165, "t_start_unix": 1779879223.4402068, "tp_rank": 0, "t_log_unix": 1779879223.4627984}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b109ed06b5882659-0-8d14993c"], "t_start_unix": 1779879223.7529812, "tp_rank": 0, "t_log_unix": 1779879223.752984}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b109ed06b5882659-0-8d14993c"], "duration_s": 0.043345845013391227, "t_start_unix": 1779879223.7529812, "tp_rank": 0, "t_log_unix": 1779879223.796329}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8a57776c81d64b2c-0-ace8fb2b"], "t_start_unix": 1779879224.0899644, "tp_rank": 0, "t_log_unix": 1779879224.0899673}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8a57776c81d64b2c-0-ace8fb2b"], "duration_s": 0.04341953102266416, "t_start_unix": 1779879224.0899644, "tp_rank": 0, "t_log_unix": 1779879224.1333857}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9b1a5dce18758450-0-b17b3649"], "t_start_unix": 1779879224.424807, "tp_rank": 0, "t_log_unix": 1779879224.42481}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9b1a5dce18758450-0-b17b3649"], "duration_s": 0.04336977802449837, "t_start_unix": 1779879224.424807, "tp_rank": 0, "t_log_unix": 1779879224.4681823}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8c7d412b85f43ed7-0-9dea4add"], "t_start_unix": 1779879224.7599711, "tp_rank": 0, "t_log_unix": 1779879224.7599735}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8c7d412b85f43ed7-0-9dea4add"], "duration_s": 0.043769759009592235, "t_start_unix": 1779879224.7599711, "tp_rank": 0, "t_log_unix": 1779879224.8037443}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8860308db3f010a5-0-ad51eb46"], "t_start_unix": 1779879225.0962389, "tp_rank": 0, "t_log_unix": 1779879225.0962446}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8860308db3f010a5-0-ad51eb46"], "duration_s": 0.043612666020635515, "t_start_unix": 1779879225.0962389, "tp_rank": 0, "t_log_unix": 1779879225.1398532}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-86cca1a2b9427801-0-ba41ade7"], "t_start_unix": 1779879225.7592747, "tp_rank": 0, "t_log_unix": 1779879225.759278}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-86cca1a2b9427801-0-ba41ade7"], "duration_s": 0.002386144013144076, "t_start_unix": 1779879225.7592747, "tp_rank": 0, "t_log_unix": 1779879225.7616625}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a208c6d804293be7-0-94d265ab"], "t_start_unix": 1779879226.384918, "tp_rank": 0, "t_log_unix": 1779879226.384921}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a208c6d804293be7-0-94d265ab"], "duration_s": 0.0023903060355223715, "t_start_unix": 1779879226.384918, "tp_rank": 0, "t_log_unix": 1779879226.3873098}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b53bea2317cc1211-0-8fcad8a8"], "t_start_unix": 1779879227.0092332, "tp_rank": 0, "t_log_unix": 1779879227.0092363}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b53bea2317cc1211-0-8fcad8a8"], "duration_s": 0.08524628396844491, "t_start_unix": 1779879227.0092332, "tp_rank": 0, "t_log_unix": 1779879227.094482}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9daf909593bbdf03-0-8fd7d50e"], "t_start_unix": 1779879227.7190688, "tp_rank": 0, "t_log_unix": 1779879227.719072}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9daf909593bbdf03-0-8fd7d50e"], "duration_s": 0.08596085698809475, "t_start_unix": 1779879227.7190688, "tp_rank": 0, "t_log_unix": 1779879227.8050315}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9ef40f3b6d736128-0-8e8e1c30"], "t_start_unix": 1779879228.4297745, "tp_rank": 0, "t_log_unix": 1779879228.4297774}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9ef40f3b6d736128-0-8e8e1c30"], "duration_s": 0.0860762019874528, "t_start_unix": 1779879228.4297745, "tp_rank": 0, "t_log_unix": 1779879228.5158527}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-851e5d7e3e83d7ea-0-a66a5e0b"], "t_start_unix": 1779879230.131392, "tp_rank": 0, "t_log_unix": 1779879230.1313956}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-851e5d7e3e83d7ea-0-a66a5e0b"], "duration_s": 0.1721468890318647, "t_start_unix": 1779879230.131392, "tp_rank": 0, "t_log_unix": 1779879230.3035412}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9be12af6a9ccccf5-0-af1230c7"], "t_start_unix": 1779879231.896075, "tp_rank": 0, "t_log_unix": 1779879231.896078}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9be12af6a9ccccf5-0-af1230c7"], "duration_s": 0.16974544001277536, "t_start_unix": 1779879231.896075, "tp_rank": 0, "t_log_unix": 1779879232.0658224}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b61b9b237366297b-0-9832f0e3"], "t_start_unix": 1779879233.6589305, "tp_rank": 0, "t_log_unix": 1779879233.6589334}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-b61b9b237366297b-0-9832f0e3"], "duration_s": 0.16975757898762822, "t_start_unix": 1779879233.6589305, "tp_rank": 0, "t_log_unix": 1779879233.8286898}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bae0d0efe47ece8f-0-affbc685"], "t_start_unix": 1779879235.4181106, "tp_rank": 0, "t_log_unix": 1779879235.418114}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bae0d0efe47ece8f-0-affbc685"], "duration_s": 0.1695251659839414, "t_start_unix": 1779879235.4181106, "tp_rank": 0, "t_log_unix": 1779879235.587638}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a34bc73c9cd2efc1-0-90d647fc"], "t_start_unix": 1779879237.1803744, "tp_rank": 0, "t_log_unix": 1779879237.1803775}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a34bc73c9cd2efc1-0-90d647fc"], "duration_s": 0.16962904302636161, "t_start_unix": 1779879237.1803744, "tp_rank": 0, "t_log_unix": 1779879237.3500054}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-89a36c12ee6b0ff3-0-9fddbc0f"], "t_start_unix": 1779879241.9859307, "tp_rank": 0, "t_log_unix": 1779879241.9859338}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-89a36c12ee6b0ff3-0-9fddbc0f"], "duration_s": 0.32203804596792907, "t_start_unix": 1779879241.9859307, "tp_rank": 0, "t_log_unix": 1779879242.3079708}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8d65512eb7e3c36c-0-8b23597c"], "t_start_unix": 1779879246.9755645, "tp_rank": 0, "t_log_unix": 1779879246.9755676}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-8d65512eb7e3c36c-0-8b23597c"], "duration_s": 0.3227974839974195, "t_start_unix": 1779879246.9755645, "tp_rank": 0, "t_log_unix": 1779879247.2983644}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a13c271ecbbca78b-0-b76a0370"], "t_start_unix": 1779879251.9618897, "tp_rank": 0, "t_log_unix": 1779879251.961893}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a13c271ecbbca78b-0-b76a0370"], "duration_s": 0.3240378479822539, "t_start_unix": 1779879251.9618897, "tp_rank": 0, "t_log_unix": 1779879252.2859304}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bada04ec8c556aca-0-a263d637"], "t_start_unix": 1779879256.9512377, "tp_rank": 0, "t_log_unix": 1779879256.9512408}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bada04ec8c556aca-0-a263d637"], "duration_s": 0.5924434679909609, "t_start_unix": 1779879256.9512377, "tp_rank": 0, "t_log_unix": 1779879257.5436878}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9641a077022e6123-0-8c3c0975"], "t_start_unix": 1779879262.2127163, "tp_rank": 0, "t_log_unix": 1779879262.2127194}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9641a077022e6123-0-8c3c0975"], "duration_s": 0.5644763479940593, "t_start_unix": 1779879262.2127163, "tp_rank": 0, "t_log_unix": 1779879262.777195}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bb3a4e5084af8c3a-0-bdfa0931"], "t_start_unix": 1779879278.1063075, "tp_rank": 0, "t_log_unix": 1779879278.106311}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bb3a4e5084af8c3a-0-bdfa0931"], "duration_s": 2.0784930550144054, "t_start_unix": 1779879278.1063075, "tp_rank": 0, "t_log_unix": 1779879280.1848085}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-91b951f85c93a71b-0-8396bee5"], "t_start_unix": 1779879295.600993, "tp_rank": 0, "t_log_unix": 1779879295.6009963}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-91b951f85c93a71b-0-8396bee5"], "duration_s": 2.2067435560165904, "t_start_unix": 1779879295.600993, "tp_rank": 0, "t_log_unix": 1779879297.8077443}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81d236ecb6aadadf-0-ac184d51"], "t_start_unix": 1779879313.2315958, "tp_rank": 0, "t_log_unix": 1779879313.2315989}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-81d236ecb6aadadf-0-ac184d51"], "duration_s": 1.9879729640088044, "t_start_unix": 1779879313.2315958, "tp_rank": 0, "t_log_unix": 1779879315.219571}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4c76c62b44c4295-0-b007a6ed"], "t_start_unix": 1779879330.6154163, "tp_rank": 0, "t_log_unix": 1779879330.6154196}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a4c76c62b44c4295-0-b007a6ed"], "duration_s": 0.9849357060156763, "t_start_unix": 1779879330.6154163, "tp_rank": 0, "t_log_unix": 1779879331.6003594}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a06d4b774a8af9a5-0-980e9d23"], "t_start_unix": 1779879346.990221, "tp_rank": 0, "t_log_unix": 1779879346.9902246}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a06d4b774a8af9a5-0-980e9d23"], "duration_s": 0.6725030990201049, "t_start_unix": 1779879346.990221, "tp_rank": 0, "t_log_unix": 1779879347.6627269}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bf0d435e06e3349f-0-8507c933"], "t_start_unix": 1779879402.7123013, "tp_rank": 0, "t_log_unix": 1779879402.7123044}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-bf0d435e06e3349f-0-8507c933"], "duration_s": 1.3384539679973386, "t_start_unix": 1779879402.7123013, "tp_rank": 0, "t_log_unix": 1779879404.0507588}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f87ae0fb0c7eec8-0-a8a1daea"], "t_start_unix": 1779879458.9232886, "tp_rank": 0, "t_log_unix": 1779879458.9232917}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f87ae0fb0c7eec8-0-a8a1daea"], "duration_s": 5.973284716019407, "t_start_unix": 1779879458.9232886, "tp_rank": 0, "t_log_unix": 1779879464.896582}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a62e48e40e6c6ad7-0-acca9741"], "t_start_unix": 1779879519.7647448, "tp_rank": 0, "t_log_unix": 1779879519.7647479}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-a62e48e40e6c6ad7-0-acca9741"], "duration_s": 10.056511385017075, "t_start_unix": 1779879519.7647448, "tp_rank": 0, "t_log_unix": 1779879529.8212643}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-824479d53bab40e4-0-af951a11"], "t_start_unix": 1779879584.888362, "tp_rank": 0, "t_log_unix": 1779879584.8883653}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-824479d53bab40e4-0-af951a11"], "duration_s": 2.925714804965537, "t_start_unix": 1779879584.888362, "tp_rank": 0, "t_log_unix": 1779879587.814085}
{"event": "receive_kv_enter", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f06f19c981c0b3f-0-b3afb370"], "t_start_unix": 1779879642.6076336, "tp_rank": 0, "t_log_unix": 1779879642.6076367}
{"event": "receive_kv_finish", "worker_addr": "tcp://172.27.123.142:44435", "req_ids": ["cmpl-9f06f19c981c0b3f-0-b3afb370"], "duration_s": 1.5183607729850337, "t_start_unix": 1779879642.6076336, "tp_rank": 0, "t_log_unix": 1779879644.1259985}
Reference in New Issue View Git Blame Copy Permalink