gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
bb698b5de1e163811d3d464eff8cb55bb96d0d35
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp4_mns64/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

2745 lines
275 KiB
Plaintext
Raw Blame History

(APIServer pid=2720467) INFO 07-12 16:22:14 [api_utils.py:339]
(APIServer pid=2720467) INFO 07-12 16:22:14 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2720467) INFO 07-12 16:22:14 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2720467) INFO 07-12 16:22:14 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2720467) INFO 07-12 16:22:14 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2720467) INFO 07-12 16:22:14 [api_utils.py:339]
(APIServer pid=2720467) INFO 07-12 16:22:14 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8701, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 4, 'max_num_batched_tokens': 8192, 'max_num_seqs': 64, 'shutdown_timeout': 120}
(APIServer pid=2720467) INFO 07-12 16:22:24 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2720467) INFO 07-12 16:22:24 [model.py:1725] Using max model len 40960
(APIServer pid=2720467) INFO 07-12 16:22:24 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2720467) INFO 07-12 16:22:24 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2720467) INFO 07-12 16:22:24 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2720467) INFO 07-12 16:22:26 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2721119) INFO 07-12 16:22:36 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2721119) WARNING 07-12 16:22:36 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2721119) INFO 07-12 16:22:36 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4
(Worker pid=2721339) INFO 07-12 16:22:47 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:52671 backend=nccl
(Worker pid=2721340) INFO 07-12 16:22:47 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:52671 backend=nccl
(Worker pid=2721341) (Worker pid=2721342) INFO 07-12 16:22:47 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:52671 backend=nccl
INFO 07-12 16:22:47 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:52671 backend=nccl
(Worker pid=2721339) INFO 07-12 16:22:48 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2721339) INFO 07-12 16:22:50 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2721339) INFO 07-12 16:22:50 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2721339) INFO 07-12 16:22:50 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2721339) INFO 07-12 16:22:51 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.21 GiB.
(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:05, 2.67it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:00<00:05, 2.35it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:05, 2.30it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:01<00:05, 2.24it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:02<00:04, 2.24it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:02<00:04, 2.22it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:03<00:04, 2.24it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:03<00:03, 2.27it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:03<00:03, 2.29it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:04<00:02, 2.31it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:04<00:02, 2.32it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:05<00:01, 2.32it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:05<00:01, 2.32it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:06<00:00, 2.30it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:06<00:00, 2.30it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:06<00:00, 2.99it/s]
(Worker_TP0 pid=2721339)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:06<00:00, 2.41it/s]
(Worker_TP0 pid=2721339)
(Worker_TP0 pid=2721339) INFO 07-12 16:22:58 [default_loader.py:430] Loading weights took 6.64 seconds
(Worker_TP0 pid=2721339) INFO 07-12 16:22:58 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2721339) INFO 07-12 16:22:59 [gpu_model_runner.py:5259] Model loading took 14.25 GiB memory and 7.076691 seconds
(Worker_TP0 pid=2721339) INFO 07-12 16:23:08 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/3c5dbd2718/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2721339) INFO 07-12 16:23:08 [backends.py:1148] Dynamo bytecode transform time: 9.34 s
(Worker_TP0 pid=2721339) INFO 07-12 16:23:08 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2721339) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2721339) return func(*args, **kwargs)
[rank0]:[W712 16:23:09.446348029 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2721339) INFO 07-12 16:23:10 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP0 pid=2721339) INFO 07-12 16:23:29 [backends.py:378] Cache the graph of compile range (1, 512) for later use
(Worker_TP0 pid=2721339) INFO 07-12 16:23:35 [backends.py:378] Cache the graph of compile range (513, 8192) for later use
(Worker_TP0 pid=2721339) INFO 07-12 16:23:45 [backends.py:393] Compiling a graph for compile range (1, 512) takes 16.61 s
(Worker_TP0 pid=2721339) INFO 07-12 16:23:46 [backends.py:393] Compiling a graph for compile range (513, 8192) takes 17.54 s
(Worker_TP0 pid=2721339) INFO 07-12 16:23:57 [decorators.py:708] saved AOT compiled function to /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/1d197159751aea999e7b74e4e78c90fea389c74aac270707bb81c303ea14759d/rank_0_0/model
(Worker_TP0 pid=2721339) INFO 07-12 16:23:57 [monitor.py:53] torch.compile took 58.34 s in total
(Worker_TP0 pid=2721339) INFO 07-12 16:23:58 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_H20.json for MoE layer.
(EngineCore pid=2721119) INFO 07-12 16:24:00 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0 pid=2721339) INFO 07-12 16:24:01 [monitor.py:81] Initial profiling/warmup run took 3.12 s
(Worker_TP2 pid=2721341) INFO 07-12 16:24:08 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=19 (largest=128), FULL=11 (largest=64)
(Worker_TP1 pid=2721340) INFO 07-12 16:24:08 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=19 (largest=128), FULL=11 (largest=64)
(Worker_TP0 pid=2721339) INFO 07-12 16:24:08 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=19 (largest=128), FULL=11 (largest=64)
(Worker_TP3 pid=2721342) INFO 07-12 16:24:08 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=19 (largest=128), FULL=11 (largest=64)
(Worker_TP2 pid=2721341) INFO 07-12 16:24:11 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2721339) INFO 07-12 16:24:11 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2721340) INFO 07-12 16:24:11 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2721342) INFO 07-12 16:24:11 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP2 pid=2721341) INFO 07-12 16:24:12 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.23 GiB total
(Worker_TP3 pid=2721342) INFO 07-12 16:24:12 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.23 GiB total
(Worker_TP1 pid=2721340) INFO 07-12 16:24:12 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.23 GiB total
(Worker_TP0 pid=2721339) INFO 07-12 16:24:12 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.23 GiB total
(Worker_TP2 pid=2721341) INFO 07-12 16:24:13 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9176 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9224. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP3 pid=2721342) INFO 07-12 16:24:13 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9176 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9224. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP1 pid=2721340) INFO 07-12 16:24:13 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9176 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9224. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=2721339) INFO 07-12 16:24:13 [gpu_worker.py:508] Available KV cache memory: 71.63 GiB
(Worker_TP0 pid=2721339) INFO 07-12 16:24:13 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9176 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9224. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2721119) INFO 07-12 16:24:13 [kv_cache_utils.py:2146] GPU KV cache size: 3,129,712 tokens
(EngineCore pid=2721119) INFO 07-12 16:24:13 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 76.41x
(Worker_TP2 pid=2721341) INFO 07-12 16:24:13 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP0 pid=2721339) (Worker_TP3 pid=2721342) INFO 07-12 16:24:13 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 16:24:13 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP1 pid=2721340) INFO 07-12 16:24:13 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP0 pid=2721339) INFO 07-12 16:24:13 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2721339) INFO 07-12 16:24:13 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP1 pid=2721340) 2026-07-12 16:24:13,311 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP3 pid=2721342) 2026-07-12 16:24:13,311 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2721339) 2026-07-12 16:24:13,311 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP2 pid=2721341) 2026-07-12 16:24:13,311 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP2 pid=2721341) 2026-07-12 16:24:13,364 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2721339) 2026-07-12 16:24:13,366 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=2721340) 2026-07-12 16:24:13,367 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP3 pid=2721342) 2026-07-12 16:24:13,369 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2721339)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 5%|▌ | 1/19 [00:00<00:02, 7.49it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:02, 7.96it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 3/19 [00:00<00:02, 7.96it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 21%|██ | 4/19 [00:00<00:01, 7.93it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:00<00:01, 7.90it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 32%|███▏ | 6/19 [00:00<00:01, 7.90it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 7/19 [00:00<00:01, 7.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:01<00:01, 7.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 9/19 [00:01<00:01, 7.82it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 10/19 [00:01<00:01, 7.85it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:01<00:01, 7.72it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:02<00:03, 2.21it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 68%|██████▊ | 13/19 [00:02<00:02, 2.82it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:02<00:01, 3.50it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 79%|███████▉ | 15/19 [00:04<00:02, 1.72it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 16/19 [00:05<00:02, 1.28it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:07<00:02, 1.10s/it]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:08<00:01, 1.11s/it]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:09<00:00, 1.18s/it]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:09<00:00, 1.96it/s]
(Worker_TP0 pid=2721339)
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 9%|▉ | 1/11 [00:00<00:01, 7.64it/s]
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:00<00:01, 8.34it/s]
Capturing CUDA graphs (decode, FULL): 27%|██▋ | 3/11 [00:00<00:00, 8.75it/s]
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:00<00:00, 8.67it/s]
Capturing CUDA graphs (decode, FULL): 45%|████▌ | 5/11 [00:00<00:00, 8.76it/s]
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:00<00:00, 8.84it/s]
Capturing CUDA graphs (decode, FULL): 64%|██████▎ | 7/11 [00:00<00:00, 8.89it/s]
Capturing CUDA graphs (decode, FULL): 73%|███████▎ | 8/11 [00:00<00:00, 8.29it/s]
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:01<00:00, 8.01it/s]
Capturing CUDA graphs (decode, FULL): 91%|█████████ | 10/11 [00:01<00:00, 8.37it/s](Worker_TP1 pid=2721340) INFO 07-12 16:24:24 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:01<00:00, 8.67it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:01<00:00, 8.54it/s]
(Worker_TP2 pid=2721341) INFO 07-12 16:24:24 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2721339) INFO 07-12 16:24:24 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2721342) INFO 07-12 16:24:24 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2721342) INFO 07-12 16:24:25 [gpu_worker.py:667] CUDA graph pool memory: 0.2 GiB (actual), 0.23 GiB (estimated), difference: 0.03 GiB (14.7%).
(Worker_TP1 pid=2721340) INFO 07-12 16:24:25 [gpu_worker.py:667] CUDA graph pool memory: 0.2 GiB (actual), 0.23 GiB (estimated), difference: 0.03 GiB (14.7%).
(Worker_TP2 pid=2721341) INFO 07-12 16:24:25 [gpu_worker.py:667] CUDA graph pool memory: 0.2 GiB (actual), 0.23 GiB (estimated), difference: 0.03 GiB (14.7%).
(Worker_TP0 pid=2721339) INFO 07-12 16:24:25 [gpu_model_runner.py:6660] Graph capturing finished in 12 secs, took 0.20 GiB
(Worker_TP0 pid=2721339) INFO 07-12 16:24:25 [gpu_worker.py:667] CUDA graph pool memory: 0.2 GiB (actual), 0.23 GiB (estimated), difference: 0.03 GiB (14.7%).
(Worker_TP1 pid=2721340) INFO 07-12 16:24:25 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP2 pid=2721341) INFO 07-12 16:24:25 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP3 pid=2721342) INFO 07-12 16:24:25 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP0 pid=2721339) INFO 07-12 16:24:25 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2721119) INFO 07-12 16:24:25 [core.py:337] init engine (profile, create kv cache, warmup model) took 86.65 s (compilation: 59.08 s)
(EngineCore pid=2721119) INFO 07-12 16:24:27 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp4_mns64/opprof/opprof-v1-dp0-pid2721119-1783873467764669604.jsonl
(EngineCore pid=2721119) INFO 07-12 16:24:27 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2721119) INFO 07-12 16:24:27 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2721119) INFO 07-12 16:24:28 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2720467) INFO 07-12 16:24:28 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2720467) WARNING 07-12 16:24:29 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2720467) INFO 07-12 16:24:29 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2720467) INFO 07-12 16:24:29 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8701
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:37] Available routes are:
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2720467) INFO 07-12 16:24:29 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2720467) INFO: Started server process [2720467]
(APIServer pid=2720467) INFO: Waiting for application startup.
(APIServer pid=2720467) INFO: Application startup complete.
(APIServer pid=2720467) INFO: 127.0.0.1:47258 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2721339) WARNING 07-12 16:24:36 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2720467) INFO: 127.0.0.1:47290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2721339) WARNING 07-12 16:24:38 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2720467) INFO 07-12 16:24:39 [loggers.py:273] Engine 000: Avg prompt throughput: 3067.8 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 8.6%
(APIServer pid=2720467) INFO: 127.0.0.1:60460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:24:49 [loggers.py:273] Engine 000: Avg prompt throughput: 369.7 tokens/s, Avg generation throughput: 344.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 54.2%
(APIServer pid=2720467) INFO: 127.0.0.1:60582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:24:59 [loggers.py:273] Engine 000: Avg prompt throughput: 16640.1 tokens/s, Avg generation throughput: 656.9 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 24.8%
(APIServer pid=2720467) INFO: 127.0.0.1:41974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:41976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:41990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:25:09 [loggers.py:273] Engine 000: Avg prompt throughput: 12484.0 tokens/s, Avg generation throughput: 1034.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 22.4%
(APIServer pid=2720467) INFO: 127.0.0.1:59816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:25:19 [loggers.py:273] Engine 000: Avg prompt throughput: 31.7 tokens/s, Avg generation throughput: 417.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 40.0%
(APIServer pid=2720467) INFO: 127.0.0.1:60130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:25:29 [loggers.py:273] Engine 000: Avg prompt throughput: 70.2 tokens/s, Avg generation throughput: 1196.0 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 56.8%
(APIServer pid=2720467) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:50772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:57998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:25:39 [loggers.py:273] Engine 000: Avg prompt throughput: 24213.4 tokens/s, Avg generation throughput: 844.9 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.2%, Prefix cache hit rate: 47.0%
(APIServer pid=2720467) INFO: 127.0.0.1:58688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:25:49 [loggers.py:273] Engine 000: Avg prompt throughput: 25505.0 tokens/s, Avg generation throughput: 1421.0 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 39.5%
(APIServer pid=2720467) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:49872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:51994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:25:59 [loggers.py:273] Engine 000: Avg prompt throughput: 28564.0 tokens/s, Avg generation throughput: 1185.3 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 34.2%
(APIServer pid=2720467) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:52634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:26:09 [loggers.py:273] Engine 000: Avg prompt throughput: 27030.5 tokens/s, Avg generation throughput: 1360.5 tokens/s, Running: 41 reqs, Waiting: 2 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 30.6%
(APIServer pid=2720467) INFO: 127.0.0.1:47228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:54392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:26:19 [loggers.py:273] Engine 000: Avg prompt throughput: 19481.4 tokens/s, Avg generation throughput: 1075.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 28.6%
(APIServer pid=2720467) INFO: 127.0.0.1:53546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:26:29 [loggers.py:273] Engine 000: Avg prompt throughput: 40.2 tokens/s, Avg generation throughput: 624.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 33.8%
(APIServer pid=2720467) INFO: 127.0.0.1:53976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:53984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:26:39 [loggers.py:273] Engine 000: Avg prompt throughput: 83.4 tokens/s, Avg generation throughput: 1356.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 39.7%
(APIServer pid=2720467) INFO: 127.0.0.1:60828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:26:49 [loggers.py:273] Engine 000: Avg prompt throughput: 84.8 tokens/s, Avg generation throughput: 1189.5 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 47.3%
(APIServer pid=2720467) INFO: 127.0.0.1:42904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:26:59 [loggers.py:273] Engine 000: Avg prompt throughput: 76.4 tokens/s, Avg generation throughput: 1097.8 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 53.4%
(APIServer pid=2720467) INFO: 127.0.0.1:56950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:56972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:42992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:27:09 [loggers.py:273] Engine 000: Avg prompt throughput: 91.4 tokens/s, Avg generation throughput: 1320.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 59.3%
(APIServer pid=2720467) INFO: 127.0.0.1:43352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:43374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:27:19 [loggers.py:273] Engine 000: Avg prompt throughput: 87.0 tokens/s, Avg generation throughput: 1344.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 59.7%
(APIServer pid=2720467) INFO: 127.0.0.1:33520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:27:29 [loggers.py:273] Engine 000: Avg prompt throughput: 32.8 tokens/s, Avg generation throughput: 567.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 62.6%
(APIServer pid=2720467) INFO: 127.0.0.1:34328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:27:39 [loggers.py:273] Engine 000: Avg prompt throughput: 250.3 tokens/s, Avg generation throughput: 1108.8 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 70.4%
(APIServer pid=2720467) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:55898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:47992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:27:49 [loggers.py:273] Engine 000: Avg prompt throughput: 1673.6 tokens/s, Avg generation throughput: 1349.4 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 80.5%
(APIServer pid=2720467) INFO: 127.0.0.1:48354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:48370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:27:59 [loggers.py:273] Engine 000: Avg prompt throughput: 477.9 tokens/s, Avg generation throughput: 1274.1 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 89.3%
(APIServer pid=2720467) INFO: 127.0.0.1:60500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:28:09 [loggers.py:273] Engine 000: Avg prompt throughput: 79.8 tokens/s, Avg generation throughput: 1120.3 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 96.0%
(APIServer pid=2720467) INFO: 127.0.0.1:37642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:45994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:28:19 [loggers.py:273] Engine 000: Avg prompt throughput: 653.3 tokens/s, Avg generation throughput: 1395.2 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 98.8%
(APIServer pid=2720467) INFO: 127.0.0.1:46594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:46626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:58960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:28:29 [loggers.py:273] Engine 000: Avg prompt throughput: 498.8 tokens/s, Avg generation throughput: 1390.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 98.7%
(APIServer pid=2720467) INFO: 127.0.0.1:38954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:28:39 [loggers.py:273] Engine 000: Avg prompt throughput: 30.7 tokens/s, Avg generation throughput: 420.0 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 98.7%
(APIServer pid=2720467) INFO: 127.0.0.1:39304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:59996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:28:49 [loggers.py:273] Engine 000: Avg prompt throughput: 70.6 tokens/s, Avg generation throughput: 1225.0 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 98.7%
(APIServer pid=2720467) INFO: 127.0.0.1:60666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:36992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:28:59 [loggers.py:273] Engine 000: Avg prompt throughput: 82.9 tokens/s, Avg generation throughput: 1260.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 98.7%
(APIServer pid=2720467) INFO: 127.0.0.1:37422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:32986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:29:09 [loggers.py:273] Engine 000: Avg prompt throughput: 79.4 tokens/s, Avg generation throughput: 1147.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 98.7%
(APIServer pid=2720467) INFO: 127.0.0.1:33018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:33048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:34988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:29:19 [loggers.py:273] Engine 000: Avg prompt throughput: 89.6 tokens/s, Avg generation throughput: 1242.8 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 98.7%
(APIServer pid=2720467) INFO: 127.0.0.1:35724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:35778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:39992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:29:29 [loggers.py:273] Engine 000: Avg prompt throughput: 102.5 tokens/s, Avg generation throughput: 1517.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 98.8%
(APIServer pid=2720467) INFO: 127.0.0.1:40328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:40354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:37988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2720467) INFO 07-12 16:29:37 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2720467) INFO 07-12 16:29:37 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2720467) INFO 07-12 16:29:37 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2720467) INFO 07-12 16:29:37 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2721119) INFO 07-12 16:29:37 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2721119) INFO 07-12 16:29:37 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2721119) INFO 07-12 16:29:37 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2721119) INFO 07-12 16:29:37 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(EngineCore pid=2721119) INFO 07-12 16:29:37 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=4
(Worker_TP0 pid=2721339) INFO 07-12 16:29:37 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(APIServer pid=2720467) INFO: Shutting down
(APIServer pid=2720467) INFO: Waiting for application shutdown.
(APIServer pid=2720467) INFO: Application shutdown complete.
(APIServer pid=2720467) INFO: Finished server process [2720467]
(EngineCore pid=2721119) INFO 07-12 16:29:41 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2720467) INFO 07-12 16:29:43 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2720467) INFO 07-12 16:29:43 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2720467) INFO 07-12 16:29:43 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink