gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
34e1f4c144641c2b2fcc308a2b776de09f747a24
aituner/runs/opprof-phase6/phase6/cells/tp4_mns16/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

2004 lines
203 KiB
Plaintext
Raw Blame History

(APIServer pid=2670181) INFO 07-12 15:23:45 [api_utils.py:339]
(APIServer pid=2670181) INFO 07-12 15:23:45 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2670181) INFO 07-12 15:23:45 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2670181) INFO 07-12 15:23:45 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2670181) INFO 07-12 15:23:45 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2670181) INFO 07-12 15:23:45 [api_utils.py:339]
(APIServer pid=2670181) INFO 07-12 15:23:45 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8521, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 4, 'max_num_batched_tokens': 8192, 'max_num_seqs': 16, 'shutdown_timeout': 120}
(APIServer pid=2670181) INFO 07-12 15:23:45 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2670181) INFO 07-12 15:23:45 [model.py:1725] Using max model len 40960
(APIServer pid=2670181) INFO 07-12 15:23:45 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2670181) INFO 07-12 15:23:45 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2670181) INFO 07-12 15:23:45 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2670181) INFO 07-12 15:23:50 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2670831) INFO 07-12 15:23:59 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2670831) WARNING 07-12 15:23:59 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2670831) INFO 07-12 15:23:59 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4
(Worker pid=2671149) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:44927 backend=nccl
(Worker pid=2671150) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:44927 backend=nccl
(Worker pid=2671152) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:44927 backend=nccl
(Worker pid=2671151) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:44927 backend=nccl
(Worker pid=2671149) INFO 07-12 15:24:13 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2671149) INFO 07-12 15:24:16 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2671149) INFO 07-12 15:24:17 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2671149) INFO 07-12 15:24:17 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2671149) INFO 07-12 15:24:17 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2671149) INFO 07-12 15:24:17 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2671149) INFO 07-12 15:24:18 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2671149) INFO 07-12 15:24:18 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2671149) INFO 07-12 15:24:18 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2671149) INFO 07-12 15:24:18 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1278.20 GiB.
(Worker_TP0 pid=2671149) INFO 07-12 15:24:18 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:06, 2.50it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:00<00:06, 2.32it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:05, 2.28it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:01<00:05, 2.20it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:02<00:05, 2.14it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:02<00:04, 2.16it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:03<00:04, 2.12it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:03<00:03, 2.13it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:04<00:03, 2.11it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:04<00:02, 2.10it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:05<00:02, 2.08it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:05<00:01, 2.07it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:06<00:01, 2.08it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:06<00:00, 2.08it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:07<00:00, 2.07it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:07<00:00, 2.68it/s]
(Worker_TP0 pid=2671149)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:07<00:00, 2.23it/s]
(Worker_TP0 pid=2671149)
(Worker_TP0 pid=2671149) INFO 07-12 15:24:25 [default_loader.py:430] Loading weights took 7.19 seconds
(Worker_TP0 pid=2671149) INFO 07-12 15:24:25 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2671149) INFO 07-12 15:24:26 [gpu_model_runner.py:5259] Model loading took 14.25 GiB memory and 7.624056 seconds
(Worker_TP0 pid=2671149) INFO 07-12 15:24:36 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/b2c463430f/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2671149) INFO 07-12 15:24:36 [backends.py:1148] Dynamo bytecode transform time: 9.78 s
(Worker_TP0 pid=2671149) INFO 07-12 15:24:36 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2671149) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2671149) return func(*args, **kwargs)
[rank0]:[W712 15:24:37.189729240 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2671149) INFO 07-12 15:24:38 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP0 pid=2671149) INFO 07-12 15:25:00 [backends.py:378] Cache the graph of compile range (1, 512) for later use
(Worker_TP0 pid=2671149) INFO 07-12 15:25:07 [backends.py:378] Cache the graph of compile range (513, 8192) for later use
(Worker_TP0 pid=2671149) INFO 07-12 15:25:18 [backends.py:393] Compiling a graph for compile range (1, 512) takes 18.23 s
(Worker_TP0 pid=2671149) INFO 07-12 15:25:19 [backends.py:393] Compiling a graph for compile range (513, 8192) takes 19.17 s
(EngineCore pid=2670831) INFO 07-12 15:25:27 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=2671149) INFO 07-12 15:25:34 [decorators.py:708] saved AOT compiled function to /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/b6644fd152ee2e01fa41a99c8d3a889e0c30ce9c4fc77a4648d72085fd0ab076/rank_0_0/model
(Worker_TP0 pid=2671149) INFO 07-12 15:25:34 [monitor.py:53] torch.compile took 67.28 s in total
(Worker_TP0 pid=2671149) INFO 07-12 15:25:34 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2671149) INFO 07-12 15:25:36 [monitor.py:81] Initial profiling/warmup run took 2.53 s
(Worker_TP1 pid=2671150) INFO 07-12 15:25:43 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP3 pid=2671152) INFO 07-12 15:25:43 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP0 pid=2671149) INFO 07-12 15:25:43 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP2 pid=2671151) INFO 07-12 15:25:44 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP3 pid=2671152) INFO 07-12 15:25:49 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2671149) INFO 07-12 15:25:49 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP2 pid=2671151) INFO 07-12 15:25:49 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2671150) INFO 07-12 15:25:49 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2671152) INFO 07-12 15:25:50 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.12 GiB total
(Worker_TP2 pid=2671151) INFO 07-12 15:25:50 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.12 GiB total
(Worker_TP0 pid=2671149) INFO 07-12 15:25:50 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.12 GiB total
(Worker_TP1 pid=2671150) INFO 07-12 15:25:50 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.12 GiB total
(Worker_TP3 pid=2671152) INFO 07-12 15:25:51 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9187 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9213. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP2 pid=2671151) INFO 07-12 15:25:51 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9187 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9213. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=2671149) INFO 07-12 15:25:51 [gpu_worker.py:508] Available KV cache memory: 71.76 GiB
(Worker_TP0 pid=2671149) INFO 07-12 15:25:51 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9187 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9213. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP1 pid=2671150) INFO 07-12 15:25:51 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9187 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9213. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2670831) INFO 07-12 15:25:51 [kv_cache_utils.py:2146] GPU KV cache size: 3,135,328 tokens
(EngineCore pid=2670831) INFO 07-12 15:25:51 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 76.55x
(Worker_TP1 pid=2671150) (Worker_TP3 pid=2671152) (Worker_TP0 pid=2671149) INFO 07-12 15:25:51 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 15:25:51 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 15:25:51 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP2 pid=2671151) INFO 07-12 15:25:51 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP0 pid=2671149) INFO 07-12 15:25:51 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2671149) INFO 07-12 15:25:51 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP0 pid=2671149) 2026-07-12 15:25:51,386 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2671150) 2026-07-12 15:25:51,388 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP3 pid=2671152) 2026-07-12 15:25:51,389 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP2 pid=2671151) 2026-07-12 15:25:51,391 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2671149) 2026-07-12 15:25:51,444 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP3 pid=2671152) 2026-07-12 15:25:51,444 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=2671150) 2026-07-12 15:25:51,444 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP2 pid=2671151) 2026-07-12 15:25:51,454 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2671149)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 14%|█▍ | 1/7 [00:00<00:00, 8.17it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 29%|██▊ | 2/7 [00:00<00:00, 8.31it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 43%|████▎ | 3/7 [00:00<00:00, 8.32it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 57%|█████▋ | 4/7 [00:00<00:00, 8.23it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 71%|███████▏ | 5/7 [00:02<00:01, 1.40it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 86%|████████▌ | 6/7 [00:03<00:00, 1.06it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:04<00:00, 1.00s/it]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:04<00:00, 1.47it/s]
(Worker_TP0 pid=2671149)
Capturing CUDA graphs (decode, FULL): 0%| | 0/5 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 20%|██ | 1/5 [00:00<00:00, 8.07it/s]
Capturing CUDA graphs (decode, FULL): 40%|████ | 2/5 [00:00<00:00, 8.64it/s]
Capturing CUDA graphs (decode, FULL): 60%|██████ | 3/5 [00:00<00:00, 8.65it/s]
Capturing CUDA graphs (decode, FULL): 80%|████████ | 4/5 [00:00<00:00, 8.82it/s](Worker_TP3 pid=2671152) INFO 07-12 15:25:57 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2671150) INFO 07-12 15:25:57 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
Capturing CUDA graphs (decode, FULL): 100%|██████████| 5/5 [00:00<00:00, 8.91it/s](Worker_TP2 pid=2671151) INFO 07-12 15:25:57 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
Capturing CUDA graphs (decode, FULL): 100%|██████████| 5/5 [00:00<00:00, 8.78it/s]
(Worker_TP0 pid=2671149) INFO 07-12 15:25:57 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP2 pid=2671151) INFO 07-12 15:25:57 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.12 GiB (estimated), difference: 0.02 GiB (24.0%).
(Worker_TP0 pid=2671149) INFO 07-12 15:25:57 [gpu_model_runner.py:6660] Graph capturing finished in 6 secs, took 0.10 GiB
(Worker_TP0 pid=2671149) INFO 07-12 15:25:57 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.12 GiB (estimated), difference: 0.02 GiB (24.0%).
(Worker_TP1 pid=2671150) INFO 07-12 15:25:57 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.12 GiB (estimated), difference: 0.02 GiB (24.0%).
(Worker_TP3 pid=2671152) INFO 07-12 15:25:57 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.12 GiB (estimated), difference: 0.02 GiB (24.0%).
(Worker_TP2 pid=2671151) INFO 07-12 15:25:57 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP0 pid=2671149) INFO 07-12 15:25:57 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP3 pid=2671152) INFO 07-12 15:25:57 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP1 pid=2671150) INFO 07-12 15:25:57 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2670831) INFO 07-12 15:25:58 [core.py:337] init engine (profile, create kv cache, warmup model) took 91.66 s (compilation: 67.59 s)
(EngineCore pid=2670831) INFO 07-12 15:25:59 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp4_mns16/opprof/opprof-v1-dp0-pid2670831-1783869959685052499.jsonl
(EngineCore pid=2670831) INFO 07-12 15:25:59 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2670831) INFO 07-12 15:25:59 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2670831) INFO 07-12 15:26:00 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2670181) INFO 07-12 15:26:00 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2670181) WARNING 07-12 15:26:00 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2670181) INFO 07-12 15:26:01 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2670181) INFO 07-12 15:26:01 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8521
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:37] Available routes are:
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2670181) INFO 07-12 15:26:01 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2670181) INFO: Started server process [2670181]
(APIServer pid=2670181) INFO: Waiting for application startup.
(APIServer pid=2670181) INFO: Application startup complete.
(APIServer pid=2670181) INFO: 127.0.0.1:52448 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2671149) WARNING 07-12 15:26:07 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2670181) INFO: 127.0.0.1:52486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2671149) WARNING 07-12 15:26:09 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2670181) INFO 07-12 15:26:11 [loggers.py:273] Engine 000: Avg prompt throughput: 3440.1 tokens/s, Avg generation throughput: 176.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 8.7%
(APIServer pid=2670181) INFO: 127.0.0.1:41402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:26:21 [loggers.py:273] Engine 000: Avg prompt throughput: 3.6 tokens/s, Avg generation throughput: 29.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 36.8%
(APIServer pid=2670181) INFO: 127.0.0.1:41440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:26:31 [loggers.py:273] Engine 000: Avg prompt throughput: 8683.1 tokens/s, Avg generation throughput: 361.6 tokens/s, Running: 15 reqs, Waiting: 46 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 29.7%
(APIServer pid=2670181) INFO: 127.0.0.1:43914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:26:41 [loggers.py:273] Engine 000: Avg prompt throughput: 4660.7 tokens/s, Avg generation throughput: 384.7 tokens/s, Running: 16 reqs, Waiting: 47 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 26.9%
(APIServer pid=2670181) INFO: 127.0.0.1:46866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:26:51 [loggers.py:273] Engine 000: Avg prompt throughput: 10502.8 tokens/s, Avg generation throughput: 642.1 tokens/s, Running: 16 reqs, Waiting: 46 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 22.4%
(APIServer pid=2670181) INFO: 127.0.0.1:59938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:27:01 [loggers.py:273] Engine 000: Avg prompt throughput: 22748.9 tokens/s, Avg generation throughput: 1002.3 tokens/s, Running: 14 reqs, Waiting: 4 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 19.4%
(APIServer pid=2670181) INFO: 127.0.0.1:47740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:27:11 [loggers.py:273] Engine 000: Avg prompt throughput: 965.1 tokens/s, Avg generation throughput: 387.3 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 29.6%
(APIServer pid=2670181) INFO: 127.0.0.1:47306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:48000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:48018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:48038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:48054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:27:21 [loggers.py:273] Engine 000: Avg prompt throughput: 66.8 tokens/s, Avg generation throughput: 1174.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 44.9%
(APIServer pid=2670181) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:44718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:27:31 [loggers.py:273] Engine 000: Avg prompt throughput: 2812.9 tokens/s, Avg generation throughput: 1126.7 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 56.5%
(APIServer pid=2670181) INFO: 127.0.0.1:41590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:27:41 [loggers.py:273] Engine 000: Avg prompt throughput: 24105.0 tokens/s, Avg generation throughput: 1022.5 tokens/s, Running: 15 reqs, Waiting: 7 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 48.8%
(APIServer pid=2670181) INFO: 127.0.0.1:55196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:27:51 [loggers.py:273] Engine 000: Avg prompt throughput: 22524.8 tokens/s, Avg generation throughput: 1023.9 tokens/s, Running: 15 reqs, Waiting: 20 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 43.4%
(APIServer pid=2670181) INFO: 127.0.0.1:36832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:36998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:37012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:37022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:37028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:28:01 [loggers.py:273] Engine 000: Avg prompt throughput: 11029.0 tokens/s, Avg generation throughput: 677.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 41.5%
(APIServer pid=2670181) INFO 07-12 15:28:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 41.5%
(APIServer pid=2670181) INFO: 127.0.0.1:50876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:28:21 [loggers.py:273] Engine 000: Avg prompt throughput: 139.7 tokens/s, Avg generation throughput: 558.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 45.7%
(APIServer pid=2670181) INFO: 127.0.0.1:54756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:54994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:55586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:28:31 [loggers.py:273] Engine 000: Avg prompt throughput: 1060.4 tokens/s, Avg generation throughput: 1343.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 51.1%
(APIServer pid=2670181) INFO: 127.0.0.1:52350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:52996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:53122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:28:41 [loggers.py:273] Engine 000: Avg prompt throughput: 764.0 tokens/s, Avg generation throughput: 1192.7 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 57.3%
(APIServer pid=2670181) INFO: 127.0.0.1:41440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:28:51 [loggers.py:273] Engine 000: Avg prompt throughput: 923.5 tokens/s, Avg generation throughput: 1110.0 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 60.8%
(APIServer pid=2670181) INFO: 127.0.0.1:33564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:33998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:29:01 [loggers.py:273] Engine 000: Avg prompt throughput: 4728.6 tokens/s, Avg generation throughput: 1278.0 tokens/s, Running: 16 reqs, Waiting: 6 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 65.8%
(APIServer pid=2670181) INFO: 127.0.0.1:46660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:46996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:29:11 [loggers.py:273] Engine 000: Avg prompt throughput: 20163.7 tokens/s, Avg generation throughput: 1082.4 tokens/s, Running: 16 reqs, Waiting: 26 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 64.7%
(APIServer pid=2670181) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:29:21 [loggers.py:273] Engine 000: Avg prompt throughput: 16744.3 tokens/s, Avg generation throughput: 871.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 65.2%
(APIServer pid=2670181) INFO: 127.0.0.1:57984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:57996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:58314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:29:31 [loggers.py:273] Engine 000: Avg prompt throughput: 34.7 tokens/s, Avg generation throughput: 525.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 65.7%
(APIServer pid=2670181) INFO: 127.0.0.1:34696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:34996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:35552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:29:41 [loggers.py:273] Engine 000: Avg prompt throughput: 79.3 tokens/s, Avg generation throughput: 1305.7 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 65.7%
(APIServer pid=2670181) INFO: 127.0.0.1:41866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:29:51 [loggers.py:273] Engine 000: Avg prompt throughput: 79.5 tokens/s, Avg generation throughput: 1154.3 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 69.3%
(APIServer pid=2670181) INFO: 127.0.0.1:47300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:30:01 [loggers.py:273] Engine 000: Avg prompt throughput: 79.1 tokens/s, Avg generation throughput: 1087.7 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 76.8%
(APIServer pid=2670181) INFO: 127.0.0.1:41428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:41994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:42152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:30:11 [loggers.py:273] Engine 000: Avg prompt throughput: 96.5 tokens/s, Avg generation throughput: 1385.7 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 85.4%
(APIServer pid=2670181) INFO: 127.0.0.1:50390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:50998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:51218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:30:21 [loggers.py:273] Engine 000: Avg prompt throughput: 87.6 tokens/s, Avg generation throughput: 1324.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 85.6%
(APIServer pid=2670181) INFO: 127.0.0.1:59650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2670181) INFO 07-12 15:30:31 [loggers.py:273] Engine 000: Avg prompt throughput: 4968.8 tokens/s, Avg generation throughput: 717.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 84.4%
(APIServer pid=2670181) INFO 07-12 15:30:34 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2670181) INFO 07-12 15:30:34 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2670181) INFO 07-12 15:30:34 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2670181) INFO 07-12 15:30:34 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2670831) INFO 07-12 15:30:34 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2670831) INFO 07-12 15:30:34 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2670831) INFO 07-12 15:30:34 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2670831) INFO 07-12 15:30:34 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(EngineCore pid=2670831) INFO 07-12 15:30:34 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=4
(Worker_TP0 pid=2671149) INFO 07-12 15:30:34 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(APIServer pid=2670181) INFO: Shutting down
(APIServer pid=2670181) INFO: Waiting for application shutdown.
(APIServer pid=2670181) INFO: Application shutdown complete.
(APIServer pid=2670181) INFO: Finished server process [2670181]
(EngineCore pid=2670831) INFO 07-12 15:30:39 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2670181) INFO 07-12 15:30:41 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2670181) INFO 07-12 15:30:41 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2670181) INFO 07-12 15:30:41 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink