gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d5b276180d147571923679ee554e434c843a4be5
aituner/runs/opprof-phase6/phase6/cells/tp2_mns64/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

454 lines
56 KiB
Plaintext
Raw Blame History

(APIServer pid=2660457) INFO 07-12 15:16:40 [api_utils.py:339]
(APIServer pid=2660457) INFO 07-12 15:16:40 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2660457) INFO 07-12 15:16:40 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2660457) INFO 07-12 15:16:40 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2660457) INFO 07-12 15:16:40 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2660457) INFO 07-12 15:16:40 [api_utils.py:339]
(APIServer pid=2660457) INFO 07-12 15:16:40 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8513, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 2, 'max_num_batched_tokens': 8192, 'max_num_seqs': 64, 'shutdown_timeout': 120}
(APIServer pid=2660457) INFO 07-12 15:16:40 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2660457) INFO 07-12 15:16:40 [model.py:1725] Using max model len 40960
(APIServer pid=2660457) INFO 07-12 15:16:40 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2660457) INFO 07-12 15:16:40 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2660457) INFO 07-12 15:16:40 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2660457) INFO 07-12 15:16:44 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2661205) INFO 07-12 15:16:54 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2661205) WARNING 07-12 15:16:54 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2661205) INFO 07-12 15:16:54 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2
(Worker pid=2661569) INFO 07-12 15:17:05 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:41317 backend=nccl
(Worker pid=2661570) INFO 07-12 15:17:05 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:41317 backend=nccl
(Worker pid=2661569) INFO 07-12 15:17:06 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2661569) INFO 07-12 15:17:09 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2661569) INFO 07-12 15:17:09 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2661569) INFO 07-12 15:17:09 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2661569) INFO 07-12 15:17:10 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2661569) INFO 07-12 15:17:10 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2661569) INFO 07-12 15:17:10 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2661569) INFO 07-12 15:17:10 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2661569) INFO 07-12 15:17:10 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2661569) INFO 07-12 15:17:11 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1275.25 GiB.
(Worker_TP0 pid=2661569) INFO 07-12 15:17:11 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:09, 1.58it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:09, 1.40it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:02<00:09, 1.36it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:02<00:08, 1.36it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:03<00:08, 1.35it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:04<00:07, 1.36it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:05<00:06, 1.36it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:05<00:05, 1.35it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:06<00:05, 1.33it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:07<00:04, 1.33it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:08<00:03, 1.32it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:08<00:03, 1.30it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:09<00:02, 1.32it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:10<00:01, 1.33it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:11<00:00, 1.34it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:11<00:00, 1.74it/s]
(Worker_TP0 pid=2661569)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:11<00:00, 1.41it/s]
(Worker_TP0 pid=2661569)
(Worker_TP0 pid=2661569) INFO 07-12 15:17:22 [default_loader.py:430] Loading weights took 11.35 seconds
(Worker_TP0 pid=2661569) INFO 07-12 15:17:22 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2661569) INFO 07-12 15:17:23 [gpu_model_runner.py:5259] Model loading took 28.46 GiB memory and 11.752905 seconds
(Worker_TP0 pid=2661569) INFO 07-12 15:17:32 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/7a9372c548/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2661569) INFO 07-12 15:17:32 [backends.py:1148] Dynamo bytecode transform time: 9.56 s
(Worker_TP0 pid=2661569) INFO 07-12 15:17:32 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2661569) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2661569) return func(*args, **kwargs)
[rank0]:[W712 15:17:33.273268179 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2661569) INFO 07-12 15:17:33 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP0 pid=2661569) INFO 07-12 15:17:57 [backends.py:378] Cache the graph of compile range (1, 8192) for later use
(Worker_TP0 pid=2661569) INFO 07-12 15:18:04 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 30.15 s
(Worker_TP0 pid=2661569) INFO 07-12 15:18:18 [decorators.py:708] saved AOT compiled function to /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/7c6e0ddc6dcbaf895fc62cc90d8a628d2dfd8c5b38e2d7747d4762b6a26b182b/rank_0_0/model
(Worker_TP0 pid=2661569) INFO 07-12 15:18:18 [monitor.py:53] torch.compile took 55.50 s in total
(Worker_TP0 pid=2661569) INFO 07-12 15:18:19 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2661569) INFO 07-12 15:18:21 [monitor.py:81] Initial profiling/warmup run took 2.55 s
(EngineCore pid=2661205) INFO 07-12 15:18:24 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP1 pid=2661570) INFO 07-12 15:18:28 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=19 (largest=128), FULL=11 (largest=64)
(Worker_TP0 pid=2661569) INFO 07-12 15:18:28 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=19 (largest=128), FULL=11 (largest=64)
(Worker_TP0 pid=2661569) INFO 07-12 15:18:32 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2661570) INFO 07-12 15:18:32 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2661570) INFO 07-12 15:18:33 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.17 GiB total
(Worker_TP0 pid=2661569) INFO 07-12 15:18:33 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.17 GiB total
(Worker_TP1 pid=2661570) INFO 07-12 15:18:34 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9182 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9218. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=2661569) INFO 07-12 15:18:34 [gpu_worker.py:508] Available KV cache memory: 57.27 GiB
(Worker_TP0 pid=2661569) INFO 07-12 15:18:34 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9182 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9218. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2661205) INFO 07-12 15:18:34 [kv_cache_utils.py:2146] GPU KV cache size: 1,251,040 tokens
(EngineCore pid=2661205) INFO 07-12 15:18:34 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 30.54x
(Worker_TP0 pid=2661569) INFO 07-12 15:18:34 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2661569) INFO 07-12 15:18:34 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP1 pid=2661570) 2026-07-12 15:18:34,433 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2661569) 2026-07-12 15:18:34,434 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2661570) 2026-07-12 15:18:34,488 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2661569) 2026-07-12 15:18:34,489 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2661569)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 5%|▌ | 1/19 [00:00<00:02, 8.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:01, 8.66it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 3/19 [00:00<00:01, 8.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 21%|██ | 4/19 [00:00<00:01, 8.68it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:00<00:01, 8.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 32%|███▏ | 6/19 [00:00<00:01, 8.62it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 7/19 [00:00<00:01, 8.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:00<00:01, 8.62it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 9/19 [00:01<00:01, 8.67it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 10/19 [00:01<00:01, 8.67it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:01<00:00, 8.59it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:02<00:03, 2.22it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 68%|██████▊ | 13/19 [00:02<00:02, 2.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:03<00:02, 1.71it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 79%|███████▉ | 15/19 [00:03<00:01, 2.24it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 16/19 [00:03<00:01, 2.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:05<00:01, 1.37it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:06<00:00, 1.25it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:07<00:00, 1.15it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:07<00:00, 2.50it/s]
(Worker_TP0 pid=2661569)
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 9%|▉ | 1/11 [00:00<00:01, 9.49it/s]
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:00<00:00, 9.41it/s]
Capturing CUDA graphs (decode, FULL): 27%|██▋ | 3/11 [00:00<00:00, 9.08it/s]
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:00<00:00, 9.02it/s]
Capturing CUDA graphs (decode, FULL): 45%|████▌ | 5/11 [00:00<00:00, 8.97it/s]
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:00<00:00, 8.15it/s]
Capturing CUDA graphs (decode, FULL): 64%|██████▎ | 7/11 [00:00<00:00, 8.52it/s]
Capturing CUDA graphs (decode, FULL): 73%|███████▎ | 8/11 [00:00<00:00, 8.81it/s]
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:01<00:00, 8.94it/s]
Capturing CUDA graphs (decode, FULL): 91%|█████████ | 10/11 [00:01<00:00, 9.03it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:01<00:00, 9.16it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:01<00:00, 8.94it/s]
(Worker_TP1 pid=2661570) INFO 07-12 15:18:43 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2661569) INFO 07-12 15:18:43 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2661569) INFO 07-12 15:18:44 [gpu_model_runner.py:6660] Graph capturing finished in 10 secs, took 0.20 GiB
(Worker_TP0 pid=2661569) INFO 07-12 15:18:44 [gpu_worker.py:667] CUDA graph pool memory: 0.2 GiB (actual), 0.17 GiB (estimated), difference: 0.03 GiB (14.9%).
(Worker_TP1 pid=2661570) INFO 07-12 15:18:44 [gpu_worker.py:667] CUDA graph pool memory: 0.2 GiB (actual), 0.17 GiB (estimated), difference: 0.03 GiB (14.9%).
(Worker_TP0 pid=2661569) INFO 07-12 15:18:44 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP1 pid=2661570) INFO 07-12 15:18:44 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2661205) INFO 07-12 15:18:44 [core.py:337] init engine (profile, create kv cache, warmup model) took 81.68 s (compilation: 55.67 s)
(EngineCore pid=2661205) INFO 07-12 15:18:45 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp2_mns64/opprof/opprof-v1-dp0-pid2661205-1783869525995600880.jsonl
(EngineCore pid=2661205) INFO 07-12 15:18:46 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2661205) INFO 07-12 15:18:46 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2661205) INFO 07-12 15:18:47 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2660457) INFO 07-12 15:18:47 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2660457) WARNING 07-12 15:18:47 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2660457) INFO 07-12 15:18:47 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2660457) INFO 07-12 15:18:47 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8513
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:37] Available routes are:
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2660457) INFO 07-12 15:18:47 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2660457) INFO: Started server process [2660457]
(APIServer pid=2660457) INFO: Waiting for application startup.
(APIServer pid=2660457) INFO: Application startup complete.
(APIServer pid=2660457) INFO: 127.0.0.1:57528 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2661569) WARNING 07-12 15:18:54 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2660457) INFO: 127.0.0.1:55622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2661569) WARNING 07-12 15:18:56 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2660457) INFO 07-12 15:18:57 [loggers.py:273] Engine 000: Avg prompt throughput: 3062.3 tokens/s, Avg generation throughput: 3.4 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 7.2%
(APIServer pid=2660457) INFO 07-12 15:19:07 [loggers.py:273] Engine 000: Avg prompt throughput: 922.8 tokens/s, Avg generation throughput: 200.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 6.5%
(APIServer pid=2660457) INFO: 127.0.0.1:50272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:50284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:50294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:50296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:50298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:50306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:47984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO 07-12 15:19:17 [loggers.py:273] Engine 000: Avg prompt throughput: 7057.2 tokens/s, Avg generation throughput: 208.8 tokens/s, Running: 28 reqs, Waiting: 8 reqs, GPU KV cache usage: 6.9%, Prefix cache hit rate: 30.7%
(APIServer pid=2660457) INFO: 127.0.0.1:48040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO 07-12 15:19:27 [loggers.py:273] Engine 000: Avg prompt throughput: 13039.6 tokens/s, Avg generation throughput: 707.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.9%, Prefix cache hit rate: 21.3%
(APIServer pid=2660457) INFO: 127.0.0.1:43322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:43480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:40530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO 07-12 15:19:37 [loggers.py:273] Engine 000: Avg prompt throughput: 13455.8 tokens/s, Avg generation throughput: 735.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 16.6%
(APIServer pid=2660457) INFO 07-12 15:19:47 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 16.6%
(APIServer pid=2660457) INFO: 127.0.0.1:33930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:33946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:33952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:33966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:33978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO 07-12 15:19:57 [loggers.py:273] Engine 000: Avg prompt throughput: 36.2 tokens/s, Avg generation throughput: 441.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 33.0%
(APIServer pid=2660457) INFO: 127.0.0.1:54826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO 07-12 15:20:07 [loggers.py:273] Engine 000: Avg prompt throughput: 45.9 tokens/s, Avg generation throughput: 672.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 46.1%
(APIServer pid=2660457) INFO: 127.0.0.1:49422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:49494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:53992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO 07-12 15:20:17 [loggers.py:273] Engine 000: Avg prompt throughput: 9203.0 tokens/s, Avg generation throughput: 297.5 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.9%, Prefix cache hit rate: 43.0%
(APIServer pid=2660457) INFO: 127.0.0.1:54036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:54148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO: 127.0.0.1:44284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660457) INFO 07-12 15:20:27 [loggers.py:273] Engine 000: Avg prompt throughput: 6296.2 tokens/s, Avg generation throughput: 406.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 40.5%
(APIServer pid=2660457) INFO 07-12 15:20:37 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 40.5%
(APIServer pid=2660457) INFO 07-12 15:23:26 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2660457) INFO 07-12 15:23:26 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2660457) INFO 07-12 15:23:26 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2660457) INFO 07-12 15:23:26 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2661205) INFO 07-12 15:23:26 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2661205) INFO 07-12 15:23:26 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2661205) INFO 07-12 15:23:26 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2661205) INFO 07-12 15:23:26 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(EngineCore pid=2661205) INFO 07-12 15:23:26 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2
(Worker_TP0 pid=2661569) INFO 07-12 15:23:26 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(APIServer pid=2660457) INFO: Shutting down
(APIServer pid=2660457) INFO: Waiting for application shutdown.
(APIServer pid=2660457) INFO: Application shutdown complete.
(APIServer pid=2660457) INFO: Finished server process [2660457]
(EngineCore pid=2661205) INFO 07-12 15:23:30 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2660457) INFO 07-12 15:23:31 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2660457) INFO 07-12 15:23:31 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2660457) INFO 07-12 15:23:31 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink