gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
823c550e53807b632f4792ce71d39dd21875655e
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp4_mns32/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

2746 lines
274 KiB
Plaintext
Raw Blame History

(APIServer pid=2711716) INFO 07-12 16:14:28 [api_utils.py:339]
(APIServer pid=2711716) INFO 07-12 16:14:28 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2711716) INFO 07-12 16:14:28 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2711716) INFO 07-12 16:14:28 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2711716) INFO 07-12 16:14:28 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2711716) INFO 07-12 16:14:28 [api_utils.py:339]
(APIServer pid=2711716) INFO 07-12 16:14:28 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8700, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 4, 'max_num_batched_tokens': 8192, 'max_num_seqs': 32, 'shutdown_timeout': 120}
(APIServer pid=2711716) INFO 07-12 16:14:38 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2711716) INFO 07-12 16:14:38 [model.py:1725] Using max model len 40960
(APIServer pid=2711716) INFO 07-12 16:14:38 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2711716) INFO 07-12 16:14:38 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2711716) INFO 07-12 16:14:38 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2711716) INFO 07-12 16:14:40 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2712385) INFO 07-12 16:14:49 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2712385) WARNING 07-12 16:14:49 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2712385) INFO 07-12 16:14:49 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4
(Worker pid=2712610) INFO 07-12 16:15:00 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:41619 backend=nccl
(Worker pid=2712613) INFO 07-12 16:15:00 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:41619 backend=nccl
(Worker pid=2712612) INFO 07-12 16:15:00 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:41619 backend=nccl
(Worker pid=2712611) INFO 07-12 16:15:00 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:41619 backend=nccl
(Worker pid=2712610) INFO 07-12 16:15:01 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2712610) INFO 07-12 16:15:03 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2712610) INFO 07-12 16:15:04 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2712610) INFO 07-12 16:15:04 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2712610) INFO 07-12 16:15:04 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2712610) INFO 07-12 16:15:04 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2712610) INFO 07-12 16:15:05 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2712610) INFO 07-12 16:15:05 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2712610) INFO 07-12 16:15:05 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2712610) INFO 07-12 16:15:05 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.29 GiB.
(Worker_TP0 pid=2712610) INFO 07-12 16:15:05 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:05, 2.54it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:00<00:06, 2.31it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:05, 2.27it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:01<00:05, 2.20it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:02<00:05, 2.17it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:02<00:04, 2.15it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:03<00:04, 2.15it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:03<00:03, 2.17it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:04<00:03, 2.18it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:04<00:02, 2.17it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:05<00:02, 2.16it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:05<00:01, 2.17it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:05<00:01, 2.15it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:06<00:00, 2.14it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:06<00:00, 2.12it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:07<00:00, 2.76it/s]
(Worker_TP0 pid=2712610)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:07<00:00, 2.28it/s]
(Worker_TP0 pid=2712610)
(Worker_TP0 pid=2712610) INFO 07-12 16:15:12 [default_loader.py:430] Loading weights took 7.03 seconds
(Worker_TP0 pid=2712610) INFO 07-12 16:15:12 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2712610) INFO 07-12 16:15:13 [gpu_model_runner.py:5259] Model loading took 14.25 GiB memory and 7.462085 seconds
(Worker_TP0 pid=2712610) INFO 07-12 16:15:23 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/edf2aacbf3/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2712610) INFO 07-12 16:15:23 [backends.py:1148] Dynamo bytecode transform time: 9.39 s
(Worker_TP0 pid=2712610) INFO 07-12 16:15:23 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2712610) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2712610) return func(*args, **kwargs)
[rank0]:[W712 16:15:23.601736403 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2712610) INFO 07-12 16:15:24 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP0 pid=2712610) INFO 07-12 16:15:43 [backends.py:378] Cache the graph of compile range (1, 512) for later use
(Worker_TP0 pid=2712610) INFO 07-12 16:15:49 [backends.py:378] Cache the graph of compile range (513, 8192) for later use
(Worker_TP0 pid=2712610) INFO 07-12 16:16:00 [backends.py:393] Compiling a graph for compile range (1, 512) takes 16.64 s
(Worker_TP0 pid=2712610) INFO 07-12 16:16:01 [backends.py:393] Compiling a graph for compile range (513, 8192) takes 17.58 s
(Worker_TP0 pid=2712610) INFO 07-12 16:16:12 [decorators.py:708] saved AOT compiled function to /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/c71c90179be357a284fa861db73684add6ff06446713e1fbd590185a5bbffe56/rank_0_0/model
(Worker_TP0 pid=2712610) INFO 07-12 16:16:12 [monitor.py:53] torch.compile took 59.12 s in total
(Worker_TP0 pid=2712610) INFO 07-12 16:16:13 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_H20.json for MoE layer.
(EngineCore pid=2712385) INFO 07-12 16:16:14 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=2712610) INFO 07-12 16:16:15 [monitor.py:81] Initial profiling/warmup run took 2.35 s
(Worker_TP2 pid=2712612) INFO 07-12 16:16:22 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=11 (largest=64), FULL=7 (largest=32)
(Worker_TP3 pid=2712613) INFO 07-12 16:16:22 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=11 (largest=64), FULL=7 (largest=32)
(Worker_TP1 pid=2712611) INFO 07-12 16:16:22 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=11 (largest=64), FULL=7 (largest=32)
(Worker_TP0 pid=2712610) INFO 07-12 16:16:22 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=11 (largest=64), FULL=7 (largest=32)
(Worker_TP1 pid=2712611) INFO 07-12 16:16:26 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2712610) INFO 07-12 16:16:26 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2712613) INFO 07-12 16:16:26 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP2 pid=2712612) INFO 07-12 16:16:26 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2712611) INFO 07-12 16:16:27 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.16 GiB total
(Worker_TP3 pid=2712613) INFO 07-12 16:16:27 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.16 GiB total
(Worker_TP2 pid=2712612) INFO 07-12 16:16:27 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.16 GiB total
(Worker_TP0 pid=2712610) INFO 07-12 16:16:27 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.16 GiB total
(Worker_TP2 pid=2712612) INFO 07-12 16:16:28 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9184 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9216. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP1 pid=2712611) INFO 07-12 16:16:28 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9184 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9216. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP3 pid=2712613) INFO 07-12 16:16:28 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9184 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9216. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=2712610) INFO 07-12 16:16:28 [gpu_worker.py:508] Available KV cache memory: 71.73 GiB
(Worker_TP0 pid=2712610) INFO 07-12 16:16:28 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9184 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9216. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2712385) INFO 07-12 16:16:28 [kv_cache_utils.py:2146] GPU KV cache size: 3,133,792 tokens
(EngineCore pid=2712385) INFO 07-12 16:16:28 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 76.51x
(Worker_TP3 pid=2712613) (Worker_TP0 pid=2712610) (Worker_TP1 pid=2712611) INFO 07-12 16:16:28 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 16:16:28 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 16:16:28 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP2 pid=2712612) INFO 07-12 16:16:28 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP0 pid=2712610) INFO 07-12 16:16:28 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2712610) INFO 07-12 16:16:28 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP2 pid=2712612) 2026-07-12 16:16:28,515 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2712610) 2026-07-12 16:16:28,515 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2712611) 2026-07-12 16:16:28,515 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP3 pid=2712613) 2026-07-12 16:16:28,515 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2712611) 2026-07-12 16:16:28,571 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP2 pid=2712612) 2026-07-12 16:16:28,572 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2712610) 2026-07-12 16:16:28,573 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP3 pid=2712613) 2026-07-12 16:16:28,574 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2712610)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 9%|▉ | 1/11 [00:00<00:01, 8.48it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 18%|█▊ | 2/11 [00:00<00:01, 8.44it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 27%|██▋ | 3/11 [00:00<00:00, 8.24it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 36%|███▋ | 4/11 [00:00<00:00, 8.23it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 45%|████▌ | 5/11 [00:00<00:00, 8.27it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 55%|█████▍ | 6/11 [00:00<00:00, 8.24it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 64%|██████▎ | 7/11 [00:01<00:01, 2.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 73%|███████▎ | 8/11 [00:03<00:02, 1.44it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 82%|████████▏ | 9/11 [00:05<00:02, 1.09s/it]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 91%|█████████ | 10/11 [00:06<00:01, 1.12s/it]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:07<00:00, 1.10s/it]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:07<00:00, 1.51it/s]
(Worker_TP0 pid=2712610)
Capturing CUDA graphs (decode, FULL): 0%| | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 14%|█▍ | 1/7 [00:00<00:00, 8.17it/s]
Capturing CUDA graphs (decode, FULL): 29%|██▊ | 2/7 [00:00<00:00, 7.85it/s]
Capturing CUDA graphs (decode, FULL): 43%|████▎ | 3/7 [00:00<00:00, 7.88it/s]
Capturing CUDA graphs (decode, FULL): 57%|█████▋ | 4/7 [00:00<00:00, 8.16it/s]
Capturing CUDA graphs (decode, FULL): 71%|███████▏ | 5/7 [00:00<00:00, 8.61it/s]
Capturing CUDA graphs (decode, FULL): 86%|████████▌ | 6/7 [00:00<00:00, 8.84it/s](Worker_TP2 pid=2712612) INFO 07-12 16:16:37 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2712613) INFO 07-12 16:16:37 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2712611) INFO 07-12 16:16:37 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:00<00:00, 8.98it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:00<00:00, 8.59it/s]
(Worker_TP0 pid=2712610) INFO 07-12 16:16:37 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2712611) INFO 07-12 16:16:37 [gpu_worker.py:667] CUDA graph pool memory: 0.13 GiB (actual), 0.16 GiB (estimated), difference: 0.03 GiB (19.4%).
(Worker_TP0 pid=2712610) INFO 07-12 16:16:37 [gpu_model_runner.py:6660] Graph capturing finished in 9 secs, took 0.13 GiB
(Worker_TP0 pid=2712610) INFO 07-12 16:16:37 [gpu_worker.py:667] CUDA graph pool memory: 0.13 GiB (actual), 0.16 GiB (estimated), difference: 0.03 GiB (19.4%).
(Worker_TP2 pid=2712612) INFO 07-12 16:16:37 [gpu_worker.py:667] CUDA graph pool memory: 0.13 GiB (actual), 0.16 GiB (estimated), difference: 0.03 GiB (19.4%).
(Worker_TP3 pid=2712613) INFO 07-12 16:16:37 [gpu_worker.py:667] CUDA graph pool memory: 0.13 GiB (actual), 0.16 GiB (estimated), difference: 0.03 GiB (19.4%).
(Worker_TP1 pid=2712611) INFO 07-12 16:16:37 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP0 pid=2712610) INFO 07-12 16:16:37 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP2 pid=2712612) INFO 07-12 16:16:37 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP3 pid=2712613) INFO 07-12 16:16:37 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2712385) INFO 07-12 16:16:38 [core.py:337] init engine (profile, create kv cache, warmup model) took 84.76 s (compilation: 59.19 s)
(EngineCore pid=2712385) INFO 07-12 16:16:40 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp4_mns32/opprof/opprof-v1-dp0-pid2712385-1783873000008589576.jsonl
(EngineCore pid=2712385) INFO 07-12 16:16:40 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2712385) INFO 07-12 16:16:40 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2712385) INFO 07-12 16:16:41 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2711716) INFO 07-12 16:16:41 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2711716) WARNING 07-12 16:16:41 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2711716) INFO 07-12 16:16:41 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2711716) INFO 07-12 16:16:41 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8700
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:37] Available routes are:
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2711716) INFO 07-12 16:16:41 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2711716) INFO: Started server process [2711716]
(APIServer pid=2711716) INFO: Waiting for application startup.
(APIServer pid=2711716) INFO: Application startup complete.
(APIServer pid=2711716) INFO: 127.0.0.1:47424 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2712610) WARNING 07-12 16:16:48 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2711716) INFO: 127.0.0.1:47456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:47570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2712610) WARNING 07-12 16:16:50 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2711716) INFO 07-12 16:16:52 [loggers.py:273] Engine 000: Avg prompt throughput: 3065.7 tokens/s, Avg generation throughput: 30.2 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 8.6%
(APIServer pid=2711716) INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:17:02 [loggers.py:273] Engine 000: Avg prompt throughput: 3395.0 tokens/s, Avg generation throughput: 358.1 tokens/s, Running: 16 reqs, Waiting: 1 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 39.3%
(APIServer pid=2711716) INFO: 127.0.0.1:41880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:17:12 [loggers.py:273] Engine 000: Avg prompt throughput: 9891.6 tokens/s, Avg generation throughput: 413.3 tokens/s, Running: 32 reqs, Waiting: 31 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 27.1%
(APIServer pid=2711716) INFO: 127.0.0.1:40332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:17:22 [loggers.py:273] Engine 000: Avg prompt throughput: 12003.2 tokens/s, Avg generation throughput: 819.0 tokens/s, Running: 32 reqs, Waiting: 18 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 22.7%
(APIServer pid=2711716) INFO 07-12 16:17:32 [loggers.py:273] Engine 000: Avg prompt throughput: 4203.1 tokens/s, Avg generation throughput: 434.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 22.4%
(APIServer pid=2711716) INFO: 127.0.0.1:57962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:58618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:17:42 [loggers.py:273] Engine 000: Avg prompt throughput: 66.0 tokens/s, Avg generation throughput: 997.1 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 49.8%
(APIServer pid=2711716) INFO: 127.0.0.1:36486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:36994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:17:52 [loggers.py:273] Engine 000: Avg prompt throughput: 13834.8 tokens/s, Avg generation throughput: 1025.1 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 51.0%
(APIServer pid=2711716) INFO: 127.0.0.1:40898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:40992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:18:02 [loggers.py:273] Engine 000: Avg prompt throughput: 27250.1 tokens/s, Avg generation throughput: 1315.4 tokens/s, Running: 30 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 41.6%
(APIServer pid=2711716) INFO: 127.0.0.1:52080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:18:12 [loggers.py:273] Engine 000: Avg prompt throughput: 25441.2 tokens/s, Avg generation throughput: 1129.6 tokens/s, Running: 32 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 35.8%
(APIServer pid=2711716) INFO: 127.0.0.1:55158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:18:22 [loggers.py:273] Engine 000: Avg prompt throughput: 27415.2 tokens/s, Avg generation throughput: 1307.4 tokens/s, Running: 32 reqs, Waiting: 9 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 32.0%
(APIServer pid=2711716) INFO: 127.0.0.1:54444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:18:32 [loggers.py:273] Engine 000: Avg prompt throughput: 27206.8 tokens/s, Avg generation throughput: 1307.0 tokens/s, Running: 32 reqs, Waiting: 8 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 29.0%
(APIServer pid=2711716) INFO: 127.0.0.1:53638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:18:42 [loggers.py:273] Engine 000: Avg prompt throughput: 3678.2 tokens/s, Avg generation throughput: 458.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 29.3%
(APIServer pid=2711716) INFO: 127.0.0.1:60728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:60992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:32986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:33154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:18:52 [loggers.py:273] Engine 000: Avg prompt throughput: 69.9 tokens/s, Avg generation throughput: 1114.4 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 35.9%
(APIServer pid=2711716) INFO: 127.0.0.1:45952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:19:02 [loggers.py:273] Engine 000: Avg prompt throughput: 83.7 tokens/s, Avg generation throughput: 1300.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 43.3%
(APIServer pid=2711716) INFO: 127.0.0.1:51866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:19:12 [loggers.py:273] Engine 000: Avg prompt throughput: 86.9 tokens/s, Avg generation throughput: 1277.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 49.7%
(APIServer pid=2711716) INFO: 127.0.0.1:43984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:19:22 [loggers.py:273] Engine 000: Avg prompt throughput: 77.7 tokens/s, Avg generation throughput: 1110.3 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 56.5%
(APIServer pid=2711716) INFO: 127.0.0.1:37614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:38374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:19:32 [loggers.py:273] Engine 000: Avg prompt throughput: 97.0 tokens/s, Avg generation throughput: 1374.5 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 59.2%
(APIServer pid=2711716) INFO: 127.0.0.1:53866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:19:42 [loggers.py:273] Engine 000: Avg prompt throughput: 76.7 tokens/s, Avg generation throughput: 1283.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 62.6%
(APIServer pid=2711716) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:19:52 [loggers.py:273] Engine 000: Avg prompt throughput: 136.7 tokens/s, Avg generation throughput: 459.6 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 66.1%
(APIServer pid=2711716) INFO: 127.0.0.1:41320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:20:02 [loggers.py:273] Engine 000: Avg prompt throughput: 779.9 tokens/s, Avg generation throughput: 1228.1 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 74.4%
(APIServer pid=2711716) INFO: 127.0.0.1:57080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:48998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:20:12 [loggers.py:273] Engine 000: Avg prompt throughput: 1035.1 tokens/s, Avg generation throughput: 1235.0 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 83.5%
(APIServer pid=2711716) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:54996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:20:22 [loggers.py:273] Engine 000: Avg prompt throughput: 473.3 tokens/s, Avg generation throughput: 1165.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 90.7%
(APIServer pid=2711716) INFO: 127.0.0.1:55102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:55790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:20:32 [loggers.py:273] Engine 000: Avg prompt throughput: 93.0 tokens/s, Avg generation throughput: 1269.8 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 99.0%
(APIServer pid=2711716) INFO: 127.0.0.1:51026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:20:42 [loggers.py:273] Engine 000: Avg prompt throughput: 693.8 tokens/s, Avg generation throughput: 1485.8 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 98.8%
(APIServer pid=2711716) INFO: 127.0.0.1:43386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:43908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:20:52 [loggers.py:273] Engine 000: Avg prompt throughput: 421.9 tokens/s, Avg generation throughput: 836.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 98.7%
(APIServer pid=2711716) INFO: 127.0.0.1:44746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:44988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:45992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:21:02 [loggers.py:273] Engine 000: Avg prompt throughput: 49.5 tokens/s, Avg generation throughput: 765.5 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 98.7%
(APIServer pid=2711716) INFO: 127.0.0.1:46120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:21:12 [loggers.py:273] Engine 000: Avg prompt throughput: 87.4 tokens/s, Avg generation throughput: 1371.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 98.7%
(APIServer pid=2711716) INFO: 127.0.0.1:52292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:53006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:21:22 [loggers.py:273] Engine 000: Avg prompt throughput: 86.2 tokens/s, Avg generation throughput: 1299.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 98.7%
(APIServer pid=2711716) INFO: 127.0.0.1:50500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:50990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:51190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:56988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:21:32 [loggers.py:273] Engine 000: Avg prompt throughput: 74.5 tokens/s, Avg generation throughput: 1101.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 98.7%
(APIServer pid=2711716) INFO: 127.0.0.1:57146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:57826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:21:42 [loggers.py:273] Engine 000: Avg prompt throughput: 98.0 tokens/s, Avg generation throughput: 1309.8 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 98.8%
(APIServer pid=2711716) INFO: 127.0.0.1:41202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:41996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:42018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:21:52 [loggers.py:273] Engine 000: Avg prompt throughput: 87.6 tokens/s, Avg generation throughput: 1389.2 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 99.2%
(APIServer pid=2711716) INFO: 127.0.0.1:37284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO: 127.0.0.1:37544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2711716) INFO 07-12 16:21:55 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2711716) INFO 07-12 16:21:55 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2711716) INFO 07-12 16:21:55 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2711716) INFO 07-12 16:21:55 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2712385) INFO 07-12 16:21:55 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2712385) INFO 07-12 16:21:55 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2712385) INFO 07-12 16:21:55 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2712385) INFO 07-12 16:21:55 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(EngineCore pid=2712385) INFO 07-12 16:21:55 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=4
(Worker_TP0 pid=2712610) INFO 07-12 16:21:55 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(APIServer pid=2711716) INFO: Shutting down
(APIServer pid=2711716) INFO: Waiting for application shutdown.
(APIServer pid=2711716) INFO: Application shutdown complete.
(APIServer pid=2711716) INFO: Finished server process [2711716]
(EngineCore pid=2712385) INFO 07-12 16:21:59 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2711716) INFO 07-12 16:22:01 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2711716) INFO 07-12 16:22:01 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2711716) INFO 07-12 16:22:01 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink