gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e5fd463f05e3351ad931ed2773289b46056d157f
aituner/runs/opprof-phase6/phase6/cells/tp2_mns32/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

1314 lines
138 KiB
Plaintext
Raw Blame History

(APIServer pid=2660456) INFO 07-12 15:16:40 [api_utils.py:339]
(APIServer pid=2660456) INFO 07-12 15:16:40 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2660456) INFO 07-12 15:16:40 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2660456) INFO 07-12 15:16:40 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2660456) INFO 07-12 15:16:40 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2660456) INFO 07-12 15:16:40 [api_utils.py:339]
(APIServer pid=2660456) INFO 07-12 15:16:40 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8512, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 2, 'max_num_batched_tokens': 8192, 'max_num_seqs': 32, 'shutdown_timeout': 120}
(APIServer pid=2660456) INFO 07-12 15:16:40 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2660456) INFO 07-12 15:16:40 [model.py:1725] Using max model len 40960
(APIServer pid=2660456) INFO 07-12 15:16:40 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2660456) INFO 07-12 15:16:40 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2660456) INFO 07-12 15:16:40 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2660456) INFO 07-12 15:16:43 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2661056) INFO 07-12 15:16:52 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2661056) WARNING 07-12 15:16:52 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2661056) INFO 07-12 15:16:52 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2
(Worker pid=2661549) INFO 07-12 15:17:04 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:34189 backend=nccl
(Worker pid=2661550) INFO 07-12 15:17:04 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:34189 backend=nccl
(Worker pid=2661549) INFO 07-12 15:17:05 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2661549) INFO 07-12 15:17:08 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2661549) INFO 07-12 15:17:09 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2661549) INFO 07-12 15:17:09 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2661549) INFO 07-12 15:17:09 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2661549) INFO 07-12 15:17:09 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2661549) INFO 07-12 15:17:10 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2661549) INFO 07-12 15:17:10 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2661549) INFO 07-12 15:17:10 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2661549) INFO 07-12 15:17:10 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1275.68 GiB.
(Worker_TP0 pid=2661549) INFO 07-12 15:17:10 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:08, 1.72it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:09, 1.55it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:08, 1.52it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:02<00:08, 1.47it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:03<00:07, 1.45it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:04<00:06, 1.45it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:04<00:06, 1.44it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:05<00:05, 1.45it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:06<00:04, 1.44it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:06<00:04, 1.44it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:07<00:03, 1.43it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:08<00:02, 1.43it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:08<00:02, 1.42it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:09<00:01, 1.42it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:10<00:00, 1.41it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.82it/s]
(Worker_TP0 pid=2661549)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.52it/s]
(Worker_TP0 pid=2661549)
(Worker_TP0 pid=2661549) INFO 07-12 15:17:21 [default_loader.py:430] Loading weights took 10.57 seconds
(Worker_TP0 pid=2661549) INFO 07-12 15:17:21 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2661549) INFO 07-12 15:17:21 [gpu_model_runner.py:5259] Model loading took 28.46 GiB memory and 11.028235 seconds
(Worker_TP0 pid=2661549) INFO 07-12 15:17:32 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/d2f6c70834/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2661549) INFO 07-12 15:17:32 [backends.py:1148] Dynamo bytecode transform time: 9.62 s
(Worker_TP0 pid=2661549) INFO 07-12 15:17:32 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2661549) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2661549) return func(*args, **kwargs)
[rank0]:[W712 15:17:32.373050677 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2661549) INFO 07-12 15:17:32 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP0 pid=2661549) INFO 07-12 15:17:56 [backends.py:378] Cache the graph of compile range (1, 8192) for later use
(Worker_TP0 pid=2661549) INFO 07-12 15:18:04 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 30.42 s
(Worker_TP0 pid=2661549) INFO 07-12 15:18:18 [decorators.py:708] saved AOT compiled function to /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/597ec81617fac86c912b3de6d61fe206016c47a08580acb57fd5cb1fd717f71f/rank_0_0/model
(Worker_TP0 pid=2661549) INFO 07-12 15:18:18 [monitor.py:53] torch.compile took 56.16 s in total
(Worker_TP0 pid=2661549) INFO 07-12 15:18:18 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2661549) INFO 07-12 15:18:21 [monitor.py:81] Initial profiling/warmup run took 2.33 s
(EngineCore pid=2661056) INFO 07-12 15:18:23 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP1 pid=2661550) INFO 07-12 15:18:28 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=11 (largest=64), FULL=7 (largest=32)
(Worker_TP0 pid=2661549) INFO 07-12 15:18:28 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=11 (largest=64), FULL=7 (largest=32)
(Worker_TP1 pid=2661550) INFO 07-12 15:18:33 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2661549) INFO 07-12 15:18:33 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2661550) INFO 07-12 15:18:34 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.11 GiB total
(Worker_TP0 pid=2661549) INFO 07-12 15:18:34 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.11 GiB total
(Worker_TP1 pid=2661550) INFO 07-12 15:18:34 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9188 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9212. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=2661549) INFO 07-12 15:18:34 [gpu_worker.py:508] Available KV cache memory: 57.32 GiB
(Worker_TP0 pid=2661549) INFO 07-12 15:18:34 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9188 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9212. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2661056) INFO 07-12 15:18:34 [kv_cache_utils.py:2146] GPU KV cache size: 1,252,272 tokens
(EngineCore pid=2661056) INFO 07-12 15:18:34 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 30.57x
(Worker_TP0 pid=2661549) INFO 07-12 15:18:35 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2661549) INFO 07-12 15:18:35 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP0 pid=2661549) 2026-07-12 15:18:35,016 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2661550) 2026-07-12 15:18:35,019 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2661549) 2026-07-12 15:18:35,073 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=2661550) 2026-07-12 15:18:35,076 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2661549)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 9%|▉ | 1/11 [00:00<00:01, 8.39it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 18%|█▊ | 2/11 [00:00<00:01, 8.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 27%|██▋ | 3/11 [00:00<00:00, 8.57it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 36%|███▋ | 4/11 [00:00<00:00, 8.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 45%|████▌ | 5/11 [00:00<00:00, 8.59it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 55%|█████▍ | 6/11 [00:00<00:00, 8.51it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 64%|██████▎ | 7/11 [00:00<00:00, 8.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 73%|███████▎ | 8/11 [00:02<00:01, 2.17it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 82%|████████▏ | 9/11 [00:03<00:01, 1.18it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 91%|█████████ | 10/11 [00:04<00:00, 1.11it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:05<00:00, 1.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:05<00:00, 1.90it/s]
(Worker_TP0 pid=2661549)
Capturing CUDA graphs (decode, FULL): 0%| | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 14%|█▍ | 1/7 [00:00<00:00, 8.70it/s]
Capturing CUDA graphs (decode, FULL): 29%|██▊ | 2/7 [00:00<00:00, 9.04it/s]
Capturing CUDA graphs (decode, FULL): 43%|████▎ | 3/7 [00:00<00:00, 9.14it/s]
Capturing CUDA graphs (decode, FULL): 57%|█████▋ | 4/7 [00:00<00:00, 8.25it/s]
Capturing CUDA graphs (decode, FULL): 71%|███████▏ | 5/7 [00:00<00:00, 8.67it/s]
Capturing CUDA graphs (decode, FULL): 86%|████████▌ | 6/7 [00:00<00:00, 8.98it/s](Worker_TP1 pid=2661550) INFO 07-12 15:18:42 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:00<00:00, 9.15it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:00<00:00, 8.93it/s]
(Worker_TP0 pid=2661549) INFO 07-12 15:18:42 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2661550) INFO 07-12 15:18:42 [gpu_worker.py:667] CUDA graph pool memory: 0.13 GiB (actual), 0.11 GiB (estimated), difference: 0.02 GiB (14.9%).
(Worker_TP0 pid=2661549) INFO 07-12 15:18:42 [gpu_model_runner.py:6660] Graph capturing finished in 7 secs, took 0.13 GiB
(Worker_TP0 pid=2661549) INFO 07-12 15:18:42 [gpu_worker.py:667] CUDA graph pool memory: 0.13 GiB (actual), 0.11 GiB (estimated), difference: 0.02 GiB (14.9%).
(Worker_TP1 pid=2661550) INFO 07-12 15:18:42 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP0 pid=2661549) INFO 07-12 15:18:42 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2661056) INFO 07-12 15:18:43 [core.py:337] init engine (profile, create kv cache, warmup model) took 80.87 s (compilation: 56.16 s)
(EngineCore pid=2661056) INFO 07-12 15:18:44 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp2_mns32/opprof/opprof-v1-dp0-pid2661056-1783869524249984543.jsonl
(EngineCore pid=2661056) INFO 07-12 15:18:44 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2661056) INFO 07-12 15:18:44 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2661056) INFO 07-12 15:18:45 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2660456) INFO 07-12 15:18:45 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2660456) WARNING 07-12 15:18:45 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2660456) INFO 07-12 15:18:45 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2660456) INFO 07-12 15:18:45 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8512
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:37] Available routes are:
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2660456) INFO 07-12 15:18:45 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2660456) INFO: Started server process [2660456]
(APIServer pid=2660456) INFO: Waiting for application startup.
(APIServer pid=2660456) INFO: Application startup complete.
(APIServer pid=2660456) INFO: 127.0.0.1:45172 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2661549) WARNING 07-12 15:18:54 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2660456) INFO: 127.0.0.1:59062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:18:56 [loggers.py:273] Engine 000: Avg prompt throughput: 2972.0 tokens/s, Avg generation throughput: 2.4 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 5.7%
(APIServer pid=2660456) INFO: 127.0.0.1:59150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2661549) WARNING 07-12 15:18:56 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2660456) INFO 07-12 15:19:06 [loggers.py:273] Engine 000: Avg prompt throughput: 973.3 tokens/s, Avg generation throughput: 202.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 6.5%
(APIServer pid=2660456) INFO: 127.0.0.1:54136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:54140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:54148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:54158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:54164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:54174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:19:16 [loggers.py:273] Engine 000: Avg prompt throughput: 7223.5 tokens/s, Avg generation throughput: 195.5 tokens/s, Running: 26 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 32.0%
(APIServer pid=2660456) INFO: 127.0.0.1:39370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:39586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:33992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:19:26 [loggers.py:273] Engine 000: Avg prompt throughput: 4048.6 tokens/s, Avg generation throughput: 396.7 tokens/s, Running: 27 reqs, Waiting: 30 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 26.5%
(APIServer pid=2660456) INFO: 127.0.0.1:34114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:34250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:19:36 [loggers.py:273] Engine 000: Avg prompt throughput: 17055.4 tokens/s, Avg generation throughput: 758.7 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 18.3%
(APIServer pid=2660456) INFO 07-12 15:19:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 57.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 18.3%
(APIServer pid=2660456) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:45126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:45134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:19:56 [loggers.py:273] Engine 000: Avg prompt throughput: 39.3 tokens/s, Avg generation throughput: 465.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 37.6%
(APIServer pid=2660456) INFO: 127.0.0.1:52866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:52994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:53120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:20:06 [loggers.py:273] Engine 000: Avg prompt throughput: 60.5 tokens/s, Avg generation throughput: 865.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 56.2%
(APIServer pid=2660456) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:20:16 [loggers.py:273] Engine 000: Avg prompt throughput: 15699.3 tokens/s, Avg generation throughput: 530.7 tokens/s, Running: 32 reqs, Waiting: 4 reqs, GPU KV cache usage: 8.0%, Prefix cache hit rate: 47.1%
(APIServer pid=2660456) INFO: 127.0.0.1:36316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:20:26 [loggers.py:273] Engine 000: Avg prompt throughput: 13096.2 tokens/s, Avg generation throughput: 720.3 tokens/s, Running: 31 reqs, Waiting: 3 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 42.4%
(APIServer pid=2660456) INFO 07-12 15:20:36 [loggers.py:273] Engine 000: Avg prompt throughput: 827.5 tokens/s, Avg generation throughput: 246.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 42.4%
(APIServer pid=2660456) INFO 07-12 15:20:46 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 42.4%
(APIServer pid=2660456) INFO: 127.0.0.1:41526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:41700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:21:06 [loggers.py:273] Engine 000: Avg prompt throughput: 54.9 tokens/s, Avg generation throughput: 674.8 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 50.2%
(APIServer pid=2660456) INFO: 127.0.0.1:43328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:43626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:21:16 [loggers.py:273] Engine 000: Avg prompt throughput: 62.6 tokens/s, Avg generation throughput: 881.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 57.9%
(APIServer pid=2660456) INFO: 127.0.0.1:60026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:21:26 [loggers.py:273] Engine 000: Avg prompt throughput: 63.6 tokens/s, Avg generation throughput: 784.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 62.5%
(APIServer pid=2660456) INFO: 127.0.0.1:42502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:21:36 [loggers.py:273] Engine 000: Avg prompt throughput: 7295.9 tokens/s, Avg generation throughput: 695.6 tokens/s, Running: 23 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 62.2%
(APIServer pid=2660456) INFO: 127.0.0.1:56702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:56954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:35998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:21:46 [loggers.py:273] Engine 000: Avg prompt throughput: 16977.4 tokens/s, Avg generation throughput: 876.8 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 57.6%
(APIServer pid=2660456) INFO: 127.0.0.1:36150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:21:56 [loggers.py:273] Engine 000: Avg prompt throughput: 19309.1 tokens/s, Avg generation throughput: 653.3 tokens/s, Running: 31 reqs, Waiting: 10 reqs, GPU KV cache usage: 9.2%, Prefix cache hit rate: 52.6%
(APIServer pid=2660456) INFO: 127.0.0.1:50618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:22:06 [loggers.py:273] Engine 000: Avg prompt throughput: 8322.2 tokens/s, Avg generation throughput: 476.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 51.1%
(APIServer pid=2660456) INFO: 127.0.0.1:51376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:22:16 [loggers.py:273] Engine 000: Avg prompt throughput: 52.7 tokens/s, Avg generation throughput: 622.0 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 54.3%
(APIServer pid=2660456) INFO: 127.0.0.1:50776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:51054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:22:26 [loggers.py:273] Engine 000: Avg prompt throughput: 60.8 tokens/s, Avg generation throughput: 884.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 58.0%
(APIServer pid=2660456) INFO: 127.0.0.1:48410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:48644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:22:36 [loggers.py:273] Engine 000: Avg prompt throughput: 63.4 tokens/s, Avg generation throughput: 778.4 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 60.6%
(APIServer pid=2660456) INFO: 127.0.0.1:49630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:49856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:22:46 [loggers.py:273] Engine 000: Avg prompt throughput: 46.9 tokens/s, Avg generation throughput: 817.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 63.0%
(APIServer pid=2660456) INFO: 127.0.0.1:42516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:42758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:59992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:22:56 [loggers.py:273] Engine 000: Avg prompt throughput: 60.0 tokens/s, Avg generation throughput: 877.8 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 67.3%
(APIServer pid=2660456) INFO: 127.0.0.1:60126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:60346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:36996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:23:06 [loggers.py:273] Engine 000: Avg prompt throughput: 50.4 tokens/s, Avg generation throughput: 843.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 73.6%
(APIServer pid=2660456) INFO: 127.0.0.1:37146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO: 127.0.0.1:37240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2660456) INFO 07-12 15:23:16 [loggers.py:273] Engine 000: Avg prompt throughput: 6.1 tokens/s, Avg generation throughput: 181.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 74.0%
(APIServer pid=2660456) INFO 07-12 15:23:20 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2660456) INFO 07-12 15:23:20 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2660456) INFO 07-12 15:23:20 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2660456) INFO 07-12 15:23:20 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2661056) INFO 07-12 15:23:20 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2661056) INFO 07-12 15:23:20 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2661056) INFO 07-12 15:23:20 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2661056) INFO 07-12 15:23:20 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(EngineCore pid=2661056) INFO 07-12 15:23:20 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2
(Worker_TP0 pid=2661549) INFO 07-12 15:23:20 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(APIServer pid=2660456) INFO: Shutting down
(APIServer pid=2660456) INFO: Waiting for application shutdown.
(APIServer pid=2660456) INFO: Application shutdown complete.
(APIServer pid=2660456) INFO: Finished server process [2660456]
(EngineCore pid=2661056) INFO 07-12 15:23:24 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2660456) INFO 07-12 15:23:26 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2660456) INFO 07-12 15:23:26 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2660456) INFO 07-12 15:23:26 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink