gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1f32ae217e06adf477b24d7cf606b899e7ea66b0
aituner/runs/opprof-phase6/phase6/cells/tp4_mns8/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

609 lines
70 KiB
Plaintext
Raw Blame History

(APIServer pid=2670180) INFO 07-12 15:23:45 [api_utils.py:339]
(APIServer pid=2670180) INFO 07-12 15:23:45 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2670180) INFO 07-12 15:23:45 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2670180) INFO 07-12 15:23:45 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2670180) INFO 07-12 15:23:45 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2670180) INFO 07-12 15:23:45 [api_utils.py:339]
(APIServer pid=2670180) INFO 07-12 15:23:45 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8520, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 4, 'max_num_batched_tokens': 8192, 'max_num_seqs': 8, 'shutdown_timeout': 120}
(APIServer pid=2670180) INFO 07-12 15:23:45 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2670180) INFO 07-12 15:23:45 [model.py:1725] Using max model len 40960
(APIServer pid=2670180) INFO 07-12 15:23:45 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2670180) INFO 07-12 15:23:45 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2670180) INFO 07-12 15:23:45 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2670180) INFO 07-12 15:23:49 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2670761) INFO 07-12 15:23:58 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2670761) WARNING 07-12 15:23:58 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2670761) INFO 07-12 15:23:58 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4
(Worker pid=2671136) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:42089 backend=nccl
(Worker pid=2671138) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:42089 backend=nccl
(Worker pid=2671139) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:42089 backend=nccl
(Worker pid=2671137) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:42089 backend=nccl
(Worker pid=2671136) INFO 07-12 15:24:13 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2671136) INFO 07-12 15:24:16 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2671136) INFO 07-12 15:24:17 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2671136) INFO 07-12 15:24:17 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2671136) INFO 07-12 15:24:17 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2671136) INFO 07-12 15:24:17 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2671136) INFO 07-12 15:24:18 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2671136) INFO 07-12 15:24:18 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2671136) INFO 07-12 15:24:18 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2671136) INFO 07-12 15:24:18 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1278.21 GiB.
(Worker_TP0 pid=2671136) INFO 07-12 15:24:18 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:05, 2.54it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:00<00:05, 2.33it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:05, 2.29it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:01<00:05, 2.20it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:02<00:05, 2.14it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:02<00:04, 2.16it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:03<00:04, 2.19it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:03<00:03, 2.20it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:04<00:03, 2.21it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:04<00:02, 2.19it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:05<00:02, 2.16it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:05<00:01, 2.15it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:05<00:01, 2.14it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:06<00:00, 2.14it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:06<00:00, 2.13it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:06<00:00, 2.76it/s]
(Worker_TP0 pid=2671136)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:06<00:00, 2.29it/s]
(Worker_TP0 pid=2671136)
(Worker_TP0 pid=2671136) INFO 07-12 15:24:25 [default_loader.py:430] Loading weights took 7.01 seconds
(Worker_TP0 pid=2671136) INFO 07-12 15:24:25 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2671136) INFO 07-12 15:24:26 [gpu_model_runner.py:5259] Model loading took 14.25 GiB memory and 7.517402 seconds
(Worker_TP0 pid=2671136) INFO 07-12 15:24:36 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/fe5024cfcf/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2671136) INFO 07-12 15:24:36 [backends.py:1148] Dynamo bytecode transform time: 9.36 s
(Worker_TP0 pid=2671136) INFO 07-12 15:24:36 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2671136) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2671136) return func(*args, **kwargs)
[rank0]:[W712 15:24:36.630060232 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2671136) INFO 07-12 15:24:37 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP0 pid=2671136) INFO 07-12 15:25:00 [backends.py:378] Cache the graph of compile range (1, 512) for later use
(Worker_TP0 pid=2671136) INFO 07-12 15:25:07 [backends.py:378] Cache the graph of compile range (513, 8192) for later use
(Worker_TP0 pid=2671136) INFO 07-12 15:25:18 [backends.py:393] Compiling a graph for compile range (1, 512) takes 17.71 s
(Worker_TP0 pid=2671136) INFO 07-12 15:25:18 [backends.py:393] Compiling a graph for compile range (513, 8192) takes 18.61 s
(EngineCore pid=2670761) INFO 07-12 15:25:27 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=2671136) INFO 07-12 15:25:31 [decorators.py:708] saved AOT compiled function to /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/8f59e8024c8d60aba6fc658297e6ebafd45eb9986d8df3183f6b00fe04a9daae/rank_0_0/model
(Worker_TP0 pid=2671136) INFO 07-12 15:25:31 [monitor.py:53] torch.compile took 64.87 s in total
(Worker_TP0 pid=2671136) INFO 07-12 15:25:32 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2671136) INFO 07-12 15:25:34 [monitor.py:81] Initial profiling/warmup run took 2.35 s
(Worker_TP3 pid=2671139) INFO 07-12 15:25:41 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(Worker_TP1 pid=2671137) INFO 07-12 15:25:41 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(Worker_TP2 pid=2671138) INFO 07-12 15:25:41 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(Worker_TP0 pid=2671136) INFO 07-12 15:25:41 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(Worker_TP0 pid=2671136) (Worker_TP3 pid=2671139) INFO 07-12 15:25:46 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
INFO 07-12 15:25:46 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2671137) INFO 07-12 15:25:46 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP2 pid=2671138) INFO 07-12 15:25:46 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2671136) INFO 07-12 15:25:47 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.10 GiB total
(Worker_TP3 pid=2671139) INFO 07-12 15:25:47 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.10 GiB total
(Worker_TP2 pid=2671138) INFO 07-12 15:25:47 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.10 GiB total
(Worker_TP1 pid=2671137) INFO 07-12 15:25:47 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.10 GiB total
(Worker_TP0 pid=2671136) INFO 07-12 15:25:47 [gpu_worker.py:508] Available KV cache memory: 71.78 GiB
(Worker_TP0 pid=2671136) INFO 07-12 15:25:47 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9189 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9211. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP3 pid=2671139) INFO 07-12 15:25:47 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9189 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9211. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP2 pid=2671138) INFO 07-12 15:25:47 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9189 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9211. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP1 pid=2671137) INFO 07-12 15:25:47 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9189 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9211. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2670761) INFO 07-12 15:25:47 [kv_cache_utils.py:2146] GPU KV cache size: 3,136,096 tokens
(EngineCore pid=2670761) INFO 07-12 15:25:47 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 76.56x
(Worker_TP2 pid=2671138) (Worker_TP3 pid=2671139) (Worker_TP0 pid=2671136) INFO 07-12 15:25:47 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 15:25:47 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 15:25:47 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP1 pid=2671137) INFO 07-12 15:25:47 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP0 pid=2671136) INFO 07-12 15:25:47 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2671136) INFO 07-12 15:25:47 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP0 pid=2671136) 2026-07-12 15:25:47,997 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP2 pid=2671138) 2026-07-12 15:25:47,997 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP3 pid=2671139) 2026-07-12 15:25:47,997 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2671137) 2026-07-12 15:25:47,997 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP2 pid=2671138) 2026-07-12 15:25:48,053 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP3 pid=2671139) 2026-07-12 15:25:48,055 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=2671137) 2026-07-12 15:25:48,055 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2671136) 2026-07-12 15:25:48,056 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2671136)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/5 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 20%|██ | 1/5 [00:00<00:00, 7.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 40%|████ | 2/5 [00:00<00:00, 8.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 60%|██████ | 3/5 [00:00<00:00, 8.09it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 80%|████████ | 4/5 [00:01<00:00, 1.76it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:02<00:00, 1.28it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:02<00:00, 1.79it/s]
(Worker_TP0 pid=2671136)
Capturing CUDA graphs (decode, FULL): 0%| | 0/4 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 25%|██▌ | 1/4 [00:00<00:00, 8.97it/s]
Capturing CUDA graphs (decode, FULL): 50%|█████ | 2/4 [00:00<00:00, 9.15it/s]
Capturing CUDA graphs (decode, FULL): 75%|███████▌ | 3/4 [00:00<00:00, 9.40it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 4/4 [00:00<00:00, 9.50it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 4/4 [00:00<00:00, 9.39it/s]
(Worker_TP0 pid=2671136) INFO 07-12 15:25:51 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP2 pid=2671138) INFO 07-12 15:25:51 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2671139) INFO 07-12 15:25:51 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2671137) INFO 07-12 15:25:51 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2671139) INFO 07-12 15:25:52 [gpu_worker.py:667] CUDA graph pool memory: 0.06 GiB (actual), 0.1 GiB (estimated), difference: 0.04 GiB (65.6%).
(Worker_TP1 pid=2671137) INFO 07-12 15:25:52 [gpu_worker.py:667] CUDA graph pool memory: 0.06 GiB (actual), 0.1 GiB (estimated), difference: 0.04 GiB (65.6%).
(Worker_TP2 pid=2671138) INFO 07-12 15:25:52 [gpu_worker.py:667] CUDA graph pool memory: 0.06 GiB (actual), 0.1 GiB (estimated), difference: 0.04 GiB (65.6%).
(Worker_TP0 pid=2671136) INFO 07-12 15:25:52 [gpu_model_runner.py:6660] Graph capturing finished in 4 secs, took 0.06 GiB
(Worker_TP0 pid=2671136) INFO 07-12 15:25:52 [gpu_worker.py:667] CUDA graph pool memory: 0.06 GiB (actual), 0.1 GiB (estimated), difference: 0.04 GiB (65.6%).
(Worker_TP0 pid=2671136) INFO 07-12 15:25:52 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP3 pid=2671139) INFO 07-12 15:25:52 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP1 pid=2671137) INFO 07-12 15:25:52 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP2 pid=2671138) INFO 07-12 15:25:52 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2670761) INFO 07-12 15:25:52 [core.py:337] init engine (profile, create kv cache, warmup model) took 86.30 s (compilation: 64.98 s)
(EngineCore pid=2670761) INFO 07-12 15:25:54 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp4_mns8/opprof/opprof-v1-dp0-pid2670761-1783869954643149539.jsonl
(EngineCore pid=2670761) INFO 07-12 15:25:54 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2670761) INFO 07-12 15:25:54 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2670761) INFO 07-12 15:25:55 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2670180) INFO 07-12 15:25:55 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2670180) WARNING 07-12 15:25:55 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2670180) INFO 07-12 15:25:56 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2670180) INFO 07-12 15:25:56 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8520
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:37] Available routes are:
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2670180) INFO 07-12 15:25:56 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2670180) INFO: Started server process [2670180]
(APIServer pid=2670180) INFO: Waiting for application startup.
(APIServer pid=2670180) INFO: Application startup complete.
(APIServer pid=2670180) INFO: 127.0.0.1:41106 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2671136) WARNING 07-12 15:26:08 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2670180) INFO: 127.0.0.1:47740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2671136) WARNING 07-12 15:26:09 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2670180) INFO: 127.0.0.1:47792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:36950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:36958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:26:16 [loggers.py:273] Engine 000: Avg prompt throughput: 4681.2 tokens/s, Avg generation throughput: 204.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 10.3%
(APIServer pid=2670180) INFO: 127.0.0.1:39042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:26:26 [loggers.py:273] Engine 000: Avg prompt throughput: 980.7 tokens/s, Avg generation throughput: 193.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 47.9%
(APIServer pid=2670180) INFO: 127.0.0.1:39248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:26:36 [loggers.py:273] Engine 000: Avg prompt throughput: 2675.8 tokens/s, Avg generation throughput: 204.2 tokens/s, Running: 6 reqs, Waiting: 35 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 43.7%
(APIServer pid=2670180) INFO: 127.0.0.1:46794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:46950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:40152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:40154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:40156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:40172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:26:46 [loggers.py:273] Engine 000: Avg prompt throughput: 2482.1 tokens/s, Avg generation throughput: 246.8 tokens/s, Running: 8 reqs, Waiting: 41 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 39.5%
(APIServer pid=2670180) INFO 07-12 15:26:56 [loggers.py:273] Engine 000: Avg prompt throughput: 14928.5 tokens/s, Avg generation throughput: 584.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 25.4%
(APIServer pid=2670180) INFO 07-12 15:27:06 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 25.4%
(APIServer pid=2670180) INFO: 127.0.0.1:55020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:55022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:55024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:55026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:55032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:55046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:55054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:27:16 [loggers.py:273] Engine 000: Avg prompt throughput: 24.2 tokens/s, Avg generation throughput: 376.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 41.0%
(APIServer pid=2670180) INFO: 127.0.0.1:37274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:37418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:27:26 [loggers.py:273] Engine 000: Avg prompt throughput: 35.3 tokens/s, Avg generation throughput: 606.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 54.0%
(APIServer pid=2670180) INFO: 127.0.0.1:57632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:58972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:27:36 [loggers.py:273] Engine 000: Avg prompt throughput: 10795.3 tokens/s, Avg generation throughput: 632.4 tokens/s, Running: 8 reqs, Waiting: 5 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 51.1%
(APIServer pid=2670180) INFO: 127.0.0.1:58984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:59100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:27:46 [loggers.py:273] Engine 000: Avg prompt throughput: 11915.9 tokens/s, Avg generation throughput: 562.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 45.6%
(APIServer pid=2670180) INFO: 127.0.0.1:39658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:39854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:27:56 [loggers.py:273] Engine 000: Avg prompt throughput: 15604.1 tokens/s, Avg generation throughput: 735.0 tokens/s, Running: 8 reqs, Waiting: 2 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 40.0%
(APIServer pid=2670180) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:28:06 [loggers.py:273] Engine 000: Avg prompt throughput: 12575.4 tokens/s, Avg generation throughput: 678.7 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 36.4%
(APIServer pid=2670180) INFO: 127.0.0.1:43542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO: 127.0.0.1:43686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670180) INFO 07-12 15:28:16 [loggers.py:273] Engine 000: Avg prompt throughput: 4778.2 tokens/s, Avg generation throughput: 261.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 35.3%
(APIServer pid=2670180) INFO 07-12 15:28:26 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 35.3%
(APIServer pid=2670180) INFO 07-12 15:30:28 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2670180) INFO 07-12 15:30:28 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2670180) INFO 07-12 15:30:28 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2670180) INFO 07-12 15:30:28 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2670761) INFO 07-12 15:30:28 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2670761) INFO 07-12 15:30:28 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2670761) INFO 07-12 15:30:28 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2670761) INFO 07-12 15:30:28 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(EngineCore pid=2670761) INFO 07-12 15:30:28 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=4
(Worker_TP0 pid=2671136) INFO 07-12 15:30:28 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(APIServer pid=2670180) INFO: Shutting down
(APIServer pid=2670180) INFO: Waiting for application shutdown.
(APIServer pid=2670180) INFO: Application shutdown complete.
(APIServer pid=2670180) INFO: Finished server process [2670180]
(EngineCore pid=2670761) INFO 07-12 15:30:32 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2670180) INFO 07-12 15:30:34 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2670180) INFO 07-12 15:30:34 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2670180) INFO 07-12 15:30:34 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink