gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8c930ba3a1f6e7b6481ff0a310e45a30c877988e
aituner/runs/opprof-phase6/phase6/cells/tp2_mns16/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

539 lines
63 KiB
Plaintext
Raw Blame History

(APIServer pid=2660455) INFO 07-12 15:16:40 [api_utils.py:339]
(APIServer pid=2660455) INFO 07-12 15:16:40 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2660455) INFO 07-12 15:16:40 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2660455) INFO 07-12 15:16:40 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2660455) INFO 07-12 15:16:40 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2660455) INFO 07-12 15:16:40 [api_utils.py:339]
(APIServer pid=2660455) INFO 07-12 15:16:40 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8511, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 2, 'max_num_batched_tokens': 8192, 'max_num_seqs': 16, 'shutdown_timeout': 120}
(APIServer pid=2660455) INFO 07-12 15:16:40 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2660455) INFO 07-12 15:16:40 [model.py:1725] Using max model len 40960
(APIServer pid=2660455) INFO 07-12 15:16:40 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2660455) INFO 07-12 15:16:40 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2660455) INFO 07-12 15:16:40 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2660455) INFO 07-12 15:16:43 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2661159) INFO 07-12 15:16:53 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2661159) WARNING 07-12 15:16:53 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2661159) INFO 07-12 15:16:53 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2
(Worker pid=2661557) INFO 07-12 15:17:04 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:48921 backend=nccl
(Worker pid=2661558) INFO 07-12 15:17:05 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:48921 backend=nccl
(Worker pid=2661557) INFO 07-12 15:17:06 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2661557) INFO 07-12 15:17:08 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2661557) INFO 07-12 15:17:09 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2661557) INFO 07-12 15:17:09 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2661557) INFO 07-12 15:17:09 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2661557) INFO 07-12 15:17:09 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2661557) INFO 07-12 15:17:10 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2661557) INFO 07-12 15:17:10 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2661557) INFO 07-12 15:17:10 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2661557) INFO 07-12 15:17:10 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1275.69 GiB.
(Worker_TP0 pid=2661557) INFO 07-12 15:17:10 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:08, 1.71it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:09, 1.55it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:08, 1.52it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:02<00:08, 1.48it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:03<00:07, 1.47it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:04<00:06, 1.45it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:04<00:06, 1.44it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:05<00:05, 1.45it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:06<00:04, 1.44it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:06<00:04, 1.44it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:07<00:03, 1.43it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:08<00:02, 1.43it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:08<00:02, 1.42it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:09<00:01, 1.42it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:10<00:00, 1.44it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.87it/s]
(Worker_TP0 pid=2661557)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.52it/s]
(Worker_TP0 pid=2661557)
(Worker_TP0 pid=2661557) INFO 07-12 15:17:21 [default_loader.py:430] Loading weights took 10.50 seconds
(Worker_TP0 pid=2661557) INFO 07-12 15:17:21 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2661557) INFO 07-12 15:17:21 [gpu_model_runner.py:5259] Model loading took 28.46 GiB memory and 10.959169 seconds
(Worker_TP0 pid=2661557) INFO 07-12 15:17:31 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/71caca9a09/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2661557) INFO 07-12 15:17:31 [backends.py:1148] Dynamo bytecode transform time: 9.70 s
(Worker_TP0 pid=2661557) INFO 07-12 15:17:31 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2661557) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2661557) return func(*args, **kwargs)
[rank0]:[W712 15:17:32.191117064 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2661557) INFO 07-12 15:17:32 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP0 pid=2661557) INFO 07-12 15:17:56 [backends.py:378] Cache the graph of compile range (1, 8192) for later use
(Worker_TP0 pid=2661557) INFO 07-12 15:18:03 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 30.28 s
(Worker_TP0 pid=2661557) INFO 07-12 15:18:18 [decorators.py:708] saved AOT compiled function to /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/b1870a24d902681201272cd6a2355f94635aa2439d53d1b45755c3823de597b6/rank_0_0/model
(Worker_TP0 pid=2661557) INFO 07-12 15:18:18 [monitor.py:53] torch.compile took 56.43 s in total
(Worker_TP0 pid=2661557) INFO 07-12 15:18:18 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2661557) INFO 07-12 15:18:20 [monitor.py:81] Initial profiling/warmup run took 2.40 s
(EngineCore pid=2661159) INFO 07-12 15:18:22 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=2661557) INFO 07-12 15:18:28 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP1 pid=2661558) INFO 07-12 15:18:28 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP0 pid=2661557) INFO 07-12 15:18:33 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2661558) INFO 07-12 15:18:33 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2661558) INFO 07-12 15:18:34 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.08 GiB total
(Worker_TP0 pid=2661557) INFO 07-12 15:18:34 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.08 GiB total
(Worker_TP1 pid=2661558) INFO 07-12 15:18:34 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9191 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9209. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=2661557) INFO 07-12 15:18:35 [gpu_worker.py:508] Available KV cache memory: 57.35 GiB
(Worker_TP0 pid=2661557) INFO 07-12 15:18:35 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9191 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9209. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2661159) INFO 07-12 15:18:35 [kv_cache_utils.py:2146] GPU KV cache size: 1,252,864 tokens
(EngineCore pid=2661159) INFO 07-12 15:18:35 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 30.59x
(Worker_TP1 pid=2661558) 2026-07-12 15:18:35,109 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2661557) INFO 07-12 15:18:35 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2661557) INFO 07-12 15:18:35 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP0 pid=2661557) 2026-07-12 15:18:35,114 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2661558) 2026-07-12 15:18:35,169 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2661557) 2026-07-12 15:18:35,170 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2661557)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 14%|█▍ | 1/7 [00:00<00:00, 7.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 29%|██▊ | 2/7 [00:00<00:00, 8.09it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 43%|████▎ | 3/7 [00:00<00:00, 8.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 57%|█████▋ | 4/7 [00:00<00:00, 8.19it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 71%|███████▏ | 5/7 [00:02<00:01, 1.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 86%|████████▌ | 6/7 [00:03<00:00, 1.17it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:04<00:00, 1.09it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:04<00:00, 1.58it/s]
(Worker_TP0 pid=2661557)
Capturing CUDA graphs (decode, FULL): 0%| | 0/5 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 20%|██ | 1/5 [00:00<00:00, 8.34it/s]
Capturing CUDA graphs (decode, FULL): 40%|████ | 2/5 [00:00<00:00, 8.87it/s]
Capturing CUDA graphs (decode, FULL): 60%|██████ | 3/5 [00:00<00:00, 9.15it/s]
Capturing CUDA graphs (decode, FULL): 80%|████████ | 4/5 [00:00<00:00, 9.30it/s](Worker_TP1 pid=2661558) INFO 07-12 15:18:40 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
Capturing CUDA graphs (decode, FULL): 100%|██████████| 5/5 [00:00<00:00, 9.17it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 5/5 [00:00<00:00, 9.09it/s]
(Worker_TP0 pid=2661557) INFO 07-12 15:18:40 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2661558) INFO 07-12 15:18:41 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.08 GiB (estimated), difference: 0.01 GiB (14.0%).
(Worker_TP0 pid=2661557) INFO 07-12 15:18:41 [gpu_model_runner.py:6660] Graph capturing finished in 6 secs, took 0.10 GiB
(Worker_TP0 pid=2661557) INFO 07-12 15:18:41 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.08 GiB (estimated), difference: 0.01 GiB (14.0%).
(Worker_TP1 pid=2661558) INFO 07-12 15:18:41 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP0 pid=2661557) INFO 07-12 15:18:41 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2661159) INFO 07-12 15:18:41 [core.py:337] init engine (profile, create kv cache, warmup model) took 79.74 s (compilation: 56.45 s)
(EngineCore pid=2661159) INFO 07-12 15:18:42 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp2_mns16/opprof/opprof-v1-dp0-pid2661159-1783869522696890561.jsonl
(EngineCore pid=2661159) INFO 07-12 15:18:42 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2661159) INFO 07-12 15:18:42 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2661159) INFO 07-12 15:18:43 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2660455) INFO 07-12 15:18:43 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2660455) WARNING 07-12 15:18:43 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2660455) INFO 07-12 15:18:44 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2660455) INFO 07-12 15:18:44 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8511
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:37] Available routes are:
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2660455) INFO 07-12 15:18:44 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2660455) INFO: Started server process [2660455]
(APIServer pid=2660455) INFO: Waiting for application startup.
(APIServer pid=2660455) INFO: Application startup complete.
(APIServer pid=2660455) INFO: 127.0.0.1:51420 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2661557) WARNING 07-12 15:18:54 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2660455) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2661557) WARNING 07-12 15:18:56 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2660455) INFO: 127.0.0.1:57912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:57942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:19:04 [loggers.py:273] Engine 000: Avg prompt throughput: 5201.1 tokens/s, Avg generation throughput: 204.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 4.4%
(APIServer pid=2660455) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:49390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:49398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:19:14 [loggers.py:273] Engine 000: Avg prompt throughput: 25.9 tokens/s, Avg generation throughput: 193.9 tokens/s, Running: 5 reqs, Waiting: 1 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 47.6%
(APIServer pid=2660455) INFO: 127.0.0.1:52274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:52532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:19:24 [loggers.py:273] Engine 000: Avg prompt throughput: 8173.6 tokens/s, Avg generation throughput: 281.9 tokens/s, Running: 16 reqs, Waiting: 27 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 32.6%
(APIServer pid=2660455) INFO: 127.0.0.1:48558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:19:34 [loggers.py:273] Engine 000: Avg prompt throughput: 5964.6 tokens/s, Avg generation throughput: 336.0 tokens/s, Running: 11 reqs, Waiting: 12 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 27.1%
(APIServer pid=2660455) INFO 07-12 15:19:44 [loggers.py:273] Engine 000: Avg prompt throughput: 3010.9 tokens/s, Avg generation throughput: 276.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 25.0%
(APIServer pid=2660455) INFO: 127.0.0.1:41688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:41700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:41702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:41708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:41716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:19:54 [loggers.py:273] Engine 000: Avg prompt throughput: 21.7 tokens/s, Avg generation throughput: 239.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 38.8%
(APIServer pid=2660455) INFO: 127.0.0.1:44642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:44888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:20:04 [loggers.py:273] Engine 000: Avg prompt throughput: 46.9 tokens/s, Avg generation throughput: 670.8 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 56.6%
(APIServer pid=2660455) INFO: 127.0.0.1:34566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:20:14 [loggers.py:273] Engine 000: Avg prompt throughput: 8386.5 tokens/s, Avg generation throughput: 478.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 50.9%
(APIServer pid=2660455) INFO: 127.0.0.1:33752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:33998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:20:24 [loggers.py:273] Engine 000: Avg prompt throughput: 12462.8 tokens/s, Avg generation throughput: 461.2 tokens/s, Running: 16 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 44.1%
(APIServer pid=2660455) INFO: 127.0.0.1:58370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:20:34 [loggers.py:273] Engine 000: Avg prompt throughput: 10574.6 tokens/s, Avg generation throughput: 559.7 tokens/s, Running: 16 reqs, Waiting: 8 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 39.5%
(APIServer pid=2660455) INFO: 127.0.0.1:58290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:58516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:20:44 [loggers.py:273] Engine 000: Avg prompt throughput: 14020.4 tokens/s, Avg generation throughput: 680.9 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 35.0%
(APIServer pid=2660455) INFO: 127.0.0.1:34264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO: 127.0.0.1:34462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660455) INFO 07-12 15:20:54 [loggers.py:273] Engine 000: Avg prompt throughput: 9210.7 tokens/s, Avg generation throughput: 352.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 32.7%
(APIServer pid=2660455) INFO 07-12 15:21:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 32.7%
(APIServer pid=2660455) INFO 07-12 15:23:14 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2660455) INFO 07-12 15:23:14 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2660455) INFO 07-12 15:23:14 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2660455) INFO 07-12 15:23:14 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2661159) INFO 07-12 15:23:14 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2661159) INFO 07-12 15:23:14 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2661159) INFO 07-12 15:23:14 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2661159) INFO 07-12 15:23:14 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(EngineCore pid=2661159) INFO 07-12 15:23:14 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2
(Worker_TP0 pid=2661557) INFO 07-12 15:23:14 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(APIServer pid=2660455) INFO: Shutting down
(APIServer pid=2660455) INFO: Waiting for application shutdown.
(APIServer pid=2660455) INFO: Application shutdown complete.
(APIServer pid=2660455) INFO: Finished server process [2660455]
(EngineCore pid=2661159) INFO 07-12 15:23:18 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2660455) INFO 07-12 15:23:20 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2660455) INFO 07-12 15:23:20 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2660455) INFO 07-12 15:23:20 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink