gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
34e1f4c144641c2b2fcc308a2b776de09f747a24
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp4_mns16/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

3544 lines
350 KiB
Plaintext
Raw Blame History

(APIServer pid=2741195) INFO 07-12 16:41:58 [api_utils.py:339]
(APIServer pid=2741195) INFO 07-12 16:41:58 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2741195) INFO 07-12 16:41:58 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2741195) INFO 07-12 16:41:58 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2741195) INFO 07-12 16:41:58 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2741195) INFO 07-12 16:41:58 [api_utils.py:339]
(APIServer pid=2741195) INFO 07-12 16:41:58 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8704, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 4, 'max_num_batched_tokens': 8192, 'max_num_seqs': 16, 'shutdown_timeout': 120}
(APIServer pid=2741195) INFO 07-12 16:41:58 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2741195) INFO 07-12 16:41:58 [model.py:1725] Using max model len 40960
(APIServer pid=2741195) INFO 07-12 16:41:58 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2741195) INFO 07-12 16:41:58 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2741195) INFO 07-12 16:41:58 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2741195) INFO 07-12 16:42:00 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2741574) INFO 07-12 16:42:10 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2741574) WARNING 07-12 16:42:10 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2741574) INFO 07-12 16:42:10 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4
(Worker pid=2741794) INFO 07-12 16:42:21 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:44217 backend=nccl
(Worker pid=2741795) INFO 07-12 16:42:21 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:44217 backend=nccl
(Worker pid=2741796) INFO 07-12 16:42:21 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:44217 backend=nccl
(Worker pid=2741797) INFO 07-12 16:42:21 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:44217 backend=nccl
(Worker pid=2741794) INFO 07-12 16:42:22 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2741794) INFO 07-12 16:42:24 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2741794) INFO 07-12 16:42:25 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2741794) INFO 07-12 16:42:25 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2741794) INFO 07-12 16:42:25 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2741794) INFO 07-12 16:42:25 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2741794) INFO 07-12 16:42:26 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2741794) INFO 07-12 16:42:26 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2741794) INFO 07-12 16:42:26 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2741794) INFO 07-12 16:42:26 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.26 GiB.
(Worker_TP0 pid=2741794) INFO 07-12 16:42:26 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:05, 2.56it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:00<00:05, 2.36it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:05, 2.36it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:01<00:05, 2.27it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:02<00:04, 2.23it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:02<00:04, 2.18it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:03<00:04, 2.17it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:03<00:03, 2.19it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:04<00:03, 2.19it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:04<00:02, 2.20it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:04<00:02, 2.21it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:05<00:01, 2.20it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:05<00:01, 2.20it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:06<00:00, 2.19it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:06<00:00, 2.19it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:06<00:00, 2.83it/s]
(Worker_TP0 pid=2741794)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:06<00:00, 2.32it/s]
(Worker_TP0 pid=2741794)
(Worker_TP0 pid=2741794) INFO 07-12 16:42:33 [default_loader.py:430] Loading weights took 6.89 seconds
(Worker_TP0 pid=2741794) INFO 07-12 16:42:33 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2741794) INFO 07-12 16:42:33 [gpu_model_runner.py:5259] Model loading took 14.25 GiB memory and 7.339504 seconds
(Worker_TP0 pid=2741794) INFO 07-12 16:42:37 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/b2c463430f/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2741794) INFO 07-12 16:42:37 [backends.py:1148] Dynamo bytecode transform time: 3.42 s
(Worker_TP0 pid=2741794) INFO 07-12 16:42:37 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2741794) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2741794) return func(*args, **kwargs)
[rank0]:[W712 16:42:38.013689126 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2741794) INFO 07-12 16:42:38 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP0 pid=2741794) INFO 07-12 16:42:44 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 512) from the cache, took 3.701 s
(Worker_TP3 pid=2741797) INFO 07-12 16:42:44 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/b6644fd152ee2e01fa41a99c8d3a889e0c30ce9c4fc77a4648d72085fd0ab076/rank_3_0/model
(Worker_TP0 pid=2741794) INFO 07-12 16:42:44 [backends.py:292] Directly load the compiled graph(s) for compile range (513, 8192) from the cache, took 3.801 s
(Worker_TP2 pid=2741796) INFO 07-12 16:42:44 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/b6644fd152ee2e01fa41a99c8d3a889e0c30ce9c4fc77a4648d72085fd0ab076/rank_2_0/model
(Worker_TP1 pid=2741795) INFO 07-12 16:42:44 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/b6644fd152ee2e01fa41a99c8d3a889e0c30ce9c4fc77a4648d72085fd0ab076/rank_1_0/model
(Worker_TP0 pid=2741794) INFO 07-12 16:42:44 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/b6644fd152ee2e01fa41a99c8d3a889e0c30ce9c4fc77a4648d72085fd0ab076/rank_0_0/model
(Worker_TP0 pid=2741794) INFO 07-12 16:42:44 [monitor.py:53] torch.compile took 9.97 s in total
(Worker_TP0 pid=2741794) INFO 07-12 16:42:44 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2741794) INFO 07-12 16:42:45 [monitor.py:81] Initial profiling/warmup run took 1.71 s
(Worker_TP1 pid=2741795) INFO 07-12 16:42:46 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP3 pid=2741797) INFO 07-12 16:42:46 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP0 pid=2741794) INFO 07-12 16:42:46 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP2 pid=2741796) INFO 07-12 16:42:46 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP1 pid=2741795) INFO 07-12 16:42:50 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2741794) INFO 07-12 16:42:50 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2741797) INFO 07-12 16:42:50 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP2 pid=2741796) INFO 07-12 16:42:50 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2741795) INFO 07-12 16:42:50 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.09 GiB total
(Worker_TP0 pid=2741794) INFO 07-12 16:42:50 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.09 GiB total
(Worker_TP3 pid=2741797) INFO 07-12 16:42:50 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.09 GiB total
(Worker_TP2 pid=2741796) INFO 07-12 16:42:50 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.09 GiB total
(Worker_TP1 pid=2741795) INFO 07-12 16:42:51 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9190 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9210. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=2741794) INFO 07-12 16:42:51 [gpu_worker.py:508] Available KV cache memory: 71.79 GiB
(Worker_TP0 pid=2741794) INFO 07-12 16:42:51 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9190 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9210. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP3 pid=2741797) INFO 07-12 16:42:51 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9190 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9210. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP2 pid=2741796) INFO 07-12 16:42:51 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9190 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9210. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2741574) INFO 07-12 16:42:51 [kv_cache_utils.py:2146] GPU KV cache size: 3,136,608 tokens
(EngineCore pid=2741574) INFO 07-12 16:42:51 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 76.58x
(Worker_TP1 pid=2741795) (Worker_TP3 pid=2741797) (Worker_TP0 pid=2741794) (Worker_TP2 pid=2741796) INFO 07-12 16:42:51 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 16:42:51 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 16:42:51 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 16:42:51 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP0 pid=2741794) INFO 07-12 16:42:51 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2741794) INFO 07-12 16:42:51 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP1 pid=2741795) 2026-07-12 16:42:51,552 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP3 pid=2741797) 2026-07-12 16:42:51,552 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2741794) 2026-07-12 16:42:51,552 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP2 pid=2741796) 2026-07-12 16:42:51,553 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2741795) 2026-07-12 16:42:51,606 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP3 pid=2741797) 2026-07-12 16:42:51,608 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2741794) 2026-07-12 16:42:51,608 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP2 pid=2741796) 2026-07-12 16:42:51,613 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2741794)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 14%|█▍ | 1/7 [00:00<00:00, 8.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 29%|██▊ | 2/7 [00:00<00:00, 8.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 43%|████▎ | 3/7 [00:00<00:00, 8.97it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 57%|█████▋ | 4/7 [00:00<00:00, 9.03it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 71%|███████▏ | 5/7 [00:01<00:01, 1.81it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 86%|████████▌ | 6/7 [00:03<00:00, 1.13it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:04<00:00, 1.04s/it]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:04<00:00, 1.50it/s]
(Worker_TP0 pid=2741794)
Capturing CUDA graphs (decode, FULL): 0%| | 0/5 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 20%|██ | 1/5 [00:00<00:00, 7.80it/s]
Capturing CUDA graphs (decode, FULL): 40%|████ | 2/5 [00:00<00:00, 8.55it/s]
Capturing CUDA graphs (decode, FULL): 60%|██████ | 3/5 [00:00<00:00, 9.12it/s]
Capturing CUDA graphs (decode, FULL): 80%|████████ | 4/5 [00:00<00:00, 9.31it/s](Worker_TP1 pid=2741795) INFO 07-12 16:42:57 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
Capturing CUDA graphs (decode, FULL): 100%|██████████| 5/5 [00:00<00:00, 9.54it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 5/5 [00:00<00:00, 9.22it/s]
(Worker_TP0 pid=2741794) INFO 07-12 16:42:57 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP2 pid=2741796) INFO 07-12 16:42:57 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2741797) INFO 07-12 16:42:57 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2741795) INFO 07-12 16:42:57 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.09 GiB (estimated), difference: 0.0 GiB (4.0%).
(Worker_TP0 pid=2741794) INFO 07-12 16:42:57 [gpu_model_runner.py:6660] Graph capturing finished in 6 secs, took 0.10 GiB
(Worker_TP0 pid=2741794) INFO 07-12 16:42:57 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.09 GiB (estimated), difference: 0.0 GiB (4.0%).
(Worker_TP2 pid=2741796) INFO 07-12 16:42:57 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.09 GiB (estimated), difference: 0.0 GiB (4.0%).
(Worker_TP3 pid=2741797) INFO 07-12 16:42:57 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.09 GiB (estimated), difference: 0.0 GiB (4.0%).
(Worker_TP1 pid=2741795) INFO 07-12 16:42:57 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP2 pid=2741796) INFO 07-12 16:42:57 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP0 pid=2741794) INFO 07-12 16:42:57 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP3 pid=2741797) INFO 07-12 16:42:57 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2741574) INFO 07-12 16:42:57 [core.py:337] init engine (profile, create kv cache, warmup model) took 24.03 s (compilation: 9.97 s)
(EngineCore pid=2741574) INFO 07-12 16:42:59 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp4_mns16/opprof/opprof-v1-dp0-pid2741574-1783874579483097755.jsonl
(EngineCore pid=2741574) INFO 07-12 16:42:59 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2741574) INFO 07-12 16:42:59 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2741574) INFO 07-12 16:43:00 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2741195) INFO 07-12 16:43:00 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2741195) WARNING 07-12 16:43:00 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2741195) INFO 07-12 16:43:01 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2741195) INFO 07-12 16:43:01 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8704
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:37] Available routes are:
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2741195) INFO 07-12 16:43:01 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2741195) INFO: Started server process [2741195]
(APIServer pid=2741195) INFO: Waiting for application startup.
(APIServer pid=2741195) INFO: Application startup complete.
(APIServer pid=2741195) INFO: 127.0.0.1:48360 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2741794) WARNING 07-12 16:43:07 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2741195) INFO: 127.0.0.1:48382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2741794) WARNING 07-12 16:43:08 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2741195) INFO: 127.0.0.1:48444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:43:11 [loggers.py:273] Engine 000: Avg prompt throughput: 3360.2 tokens/s, Avg generation throughput: 98.6 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 8.7%
(APIServer pid=2741195) INFO: 127.0.0.1:50422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:43:21 [loggers.py:273] Engine 000: Avg prompt throughput: 2858.2 tokens/s, Avg generation throughput: 268.4 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 40.5%
(APIServer pid=2741195) INFO: 127.0.0.1:41270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:43:31 [loggers.py:273] Engine 000: Avg prompt throughput: 10004.1 tokens/s, Avg generation throughput: 438.1 tokens/s, Running: 16 reqs, Waiting: 48 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 26.8%
(APIServer pid=2741195) INFO: 127.0.0.1:33122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:33592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:43:41 [loggers.py:273] Engine 000: Avg prompt throughput: 13243.9 tokens/s, Avg generation throughput: 989.1 tokens/s, Running: 16 reqs, Waiting: 19 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 22.8%
(APIServer pid=2741195) INFO 07-12 16:43:51 [loggers.py:273] Engine 000: Avg prompt throughput: 4072.6 tokens/s, Avg generation throughput: 337.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 22.5%
(APIServer pid=2741195) INFO: 127.0.0.1:35634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:44:01 [loggers.py:273] Engine 000: Avg prompt throughput: 66.7 tokens/s, Avg generation throughput: 1062.9 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 49.5%
(APIServer pid=2741195) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:44:11 [loggers.py:273] Engine 000: Avg prompt throughput: 10834.1 tokens/s, Avg generation throughput: 1137.9 tokens/s, Running: 16 reqs, Waiting: 11 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 52.6%
(APIServer pid=2741195) INFO: 127.0.0.1:54664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:55428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:44:21 [loggers.py:273] Engine 000: Avg prompt throughput: 22052.4 tokens/s, Avg generation throughput: 1016.1 tokens/s, Running: 15 reqs, Waiting: 31 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 44.6%
(APIServer pid=2741195) INFO: 127.0.0.1:32796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:32886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:44:31 [loggers.py:273] Engine 000: Avg prompt throughput: 14337.8 tokens/s, Avg generation throughput: 610.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 40.5%
(APIServer pid=2741195) INFO: 127.0.0.1:51462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:44:41 [loggers.py:273] Engine 000: Avg prompt throughput: 41.7 tokens/s, Avg generation throughput: 664.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 47.3%
(APIServer pid=2741195) INFO: 127.0.0.1:38388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:38998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:39344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:44:51 [loggers.py:273] Engine 000: Avg prompt throughput: 87.0 tokens/s, Avg generation throughput: 1343.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 54.9%
(APIServer pid=2741195) INFO: 127.0.0.1:43562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:45:01 [loggers.py:273] Engine 000: Avg prompt throughput: 84.4 tokens/s, Avg generation throughput: 1250.4 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 62.2%
(APIServer pid=2741195) INFO: 127.0.0.1:50640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:45:11 [loggers.py:273] Engine 000: Avg prompt throughput: 9753.3 tokens/s, Avg generation throughput: 922.6 tokens/s, Running: 16 reqs, Waiting: 4 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 61.8%
(APIServer pid=2741195) INFO: 127.0.0.1:59378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:45:21 [loggers.py:273] Engine 000: Avg prompt throughput: 23168.5 tokens/s, Avg generation throughput: 1060.4 tokens/s, Running: 16 reqs, Waiting: 17 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 56.9%
(APIServer pid=2741195) INFO: 127.0.0.1:44428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:45010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:45012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:45:31 [loggers.py:273] Engine 000: Avg prompt throughput: 20307.7 tokens/s, Avg generation throughput: 1133.8 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 53.4%
(APIServer pid=2741195) INFO: 127.0.0.1:41644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:45:41 [loggers.py:273] Engine 000: Avg prompt throughput: 26.4 tokens/s, Avg generation throughput: 371.6 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 54.9%
(APIServer pid=2741195) INFO: 127.0.0.1:42574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:45:51 [loggers.py:273] Engine 000: Avg prompt throughput: 69.7 tokens/s, Avg generation throughput: 1170.4 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 60.9%
(APIServer pid=2741195) INFO: 127.0.0.1:50274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:46:01 [loggers.py:273] Engine 000: Avg prompt throughput: 79.7 tokens/s, Avg generation throughput: 1228.1 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 66.8%
(APIServer pid=2741195) INFO: 127.0.0.1:53842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:54574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:46:11 [loggers.py:273] Engine 000: Avg prompt throughput: 75.9 tokens/s, Avg generation throughput: 1116.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 67.7%
(APIServer pid=2741195) INFO: 127.0.0.1:51598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:46:21 [loggers.py:273] Engine 000: Avg prompt throughput: 84.2 tokens/s, Avg generation throughput: 1173.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 72.8%
(APIServer pid=2741195) INFO: 127.0.0.1:34402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:46:31 [loggers.py:273] Engine 000: Avg prompt throughput: 98.9 tokens/s, Avg generation throughput: 1437.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 82.2%
(APIServer pid=2741195) INFO: 127.0.0.1:43438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:46:41 [loggers.py:273] Engine 000: Avg prompt throughput: 19352.5 tokens/s, Avg generation throughput: 938.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 75.9%
(APIServer pid=2741195) INFO: 127.0.0.1:60560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:46:51 [loggers.py:273] Engine 000: Avg prompt throughput: 32.4 tokens/s, Avg generation throughput: 474.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 76.4%
(APIServer pid=2741195) INFO: 127.0.0.1:41598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:42426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:47:01 [loggers.py:273] Engine 000: Avg prompt throughput: 76.8 tokens/s, Avg generation throughput: 1221.5 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 75.4%
(APIServer pid=2741195) INFO: 127.0.0.1:43716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:47:11 [loggers.py:273] Engine 000: Avg prompt throughput: 75.4 tokens/s, Avg generation throughput: 1150.0 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 76.2%
(APIServer pid=2741195) INFO: 127.0.0.1:48992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:48994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:47:21 [loggers.py:273] Engine 000: Avg prompt throughput: 75.5 tokens/s, Avg generation throughput: 1105.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 83.9%
(APIServer pid=2741195) INFO: 127.0.0.1:40542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:47:31 [loggers.py:273] Engine 000: Avg prompt throughput: 90.6 tokens/s, Avg generation throughput: 1241.1 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 92.4%
(APIServer pid=2741195) INFO: 127.0.0.1:40650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:41566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:47:41 [loggers.py:273] Engine 000: Avg prompt throughput: 93.4 tokens/s, Avg generation throughput: 1379.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 93.4%
(APIServer pid=2741195) INFO: 127.0.0.1:59898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:60400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:47:51 [loggers.py:273] Engine 000: Avg prompt throughput: 43.4 tokens/s, Avg generation throughput: 788.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 93.5%
(APIServer pid=2741195) INFO: 127.0.0.1:53130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:48:01 [loggers.py:273] Engine 000: Avg prompt throughput: 233.9 tokens/s, Avg generation throughput: 797.1 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 93.3%
(APIServer pid=2741195) INFO: 127.0.0.1:36736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:48:11 [loggers.py:273] Engine 000: Avg prompt throughput: 1435.6 tokens/s, Avg generation throughput: 1358.7 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 92.7%
(APIServer pid=2741195) INFO: 127.0.0.1:43620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:48:21 [loggers.py:273] Engine 000: Avg prompt throughput: 328.4 tokens/s, Avg generation throughput: 1296.6 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 92.7%
(APIServer pid=2741195) INFO: 127.0.0.1:49266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:49996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:50028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:48:31 [loggers.py:273] Engine 000: Avg prompt throughput: 457.2 tokens/s, Avg generation throughput: 1100.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 93.2%
(APIServer pid=2741195) INFO: 127.0.0.1:46720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:48:41 [loggers.py:273] Engine 000: Avg prompt throughput: 3369.2 tokens/s, Avg generation throughput: 1321.4 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 97.9%
(APIServer pid=2741195) INFO: 127.0.0.1:58334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:58996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:59228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:48:51 [loggers.py:273] Engine 000: Avg prompt throughput: 1804.7 tokens/s, Avg generation throughput: 1387.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 97.4%
(APIServer pid=2741195) INFO: 127.0.0.1:46778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:46986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:47044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:49:01 [loggers.py:273] Engine 000: Avg prompt throughput: 31.9 tokens/s, Avg generation throughput: 530.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 97.4%
(APIServer pid=2741195) INFO: 127.0.0.1:52622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:52988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:53282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:49:11 [loggers.py:273] Engine 000: Avg prompt throughput: 73.3 tokens/s, Avg generation throughput: 1185.5 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 97.3%
(APIServer pid=2741195) INFO: 127.0.0.1:34258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:34988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:35020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:49:21 [loggers.py:273] Engine 000: Avg prompt throughput: 85.5 tokens/s, Avg generation throughput: 1325.9 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 97.3%
(APIServer pid=2741195) INFO: 127.0.0.1:51102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:51900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:49:31 [loggers.py:273] Engine 000: Avg prompt throughput: 87.1 tokens/s, Avg generation throughput: 1273.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 97.4%
(APIServer pid=2741195) INFO: 127.0.0.1:40256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:40988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:49:41 [loggers.py:273] Engine 000: Avg prompt throughput: 83.5 tokens/s, Avg generation throughput: 1142.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 97.4%
(APIServer pid=2741195) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:43996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:44734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:49:51 [loggers.py:273] Engine 000: Avg prompt throughput: 101.1 tokens/s, Avg generation throughput: 1444.4 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 97.5%
(APIServer pid=2741195) INFO: 127.0.0.1:36490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:36990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO: 127.0.0.1:37172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2741195) INFO 07-12 16:50:01 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2741195) INFO 07-12 16:50:01 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2741195) INFO 07-12 16:50:01 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2741195) INFO 07-12 16:50:01 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2741574) INFO 07-12 16:50:01 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2741574) INFO 07-12 16:50:01 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2741574) INFO 07-12 16:50:01 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2741574) INFO 07-12 16:50:01 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(Worker_TP0 pid=2741794) INFO 07-12 16:50:01 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(EngineCore pid=2741574) INFO 07-12 16:50:01 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=4
(APIServer pid=2741195) INFO: Shutting down
(APIServer pid=2741195) INFO: Waiting for application shutdown.
(APIServer pid=2741195) INFO: Application shutdown complete.
(APIServer pid=2741195) INFO: Finished server process [2741195]
(EngineCore pid=2741574) INFO 07-12 16:50:05 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2741195) INFO 07-12 16:50:06 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2741195) INFO 07-12 16:50:06 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2741195) INFO 07-12 16:50:06 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink