gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
bb698b5de1e163811d3d464eff8cb55bb96d0d35
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp4_mns8/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

1970 lines
202 KiB
Plaintext
Raw Blame History

(APIServer pid=2762128) INFO 07-12 17:02:37 [api_utils.py:339]
(APIServer pid=2762128) INFO 07-12 17:02:37 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2762128) INFO 07-12 17:02:37 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2762128) INFO 07-12 17:02:37 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2762128) INFO 07-12 17:02:37 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2762128) INFO 07-12 17:02:37 [api_utils.py:339]
(APIServer pid=2762128) INFO 07-12 17:02:37 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8707, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 4, 'max_num_batched_tokens': 8192, 'max_num_seqs': 8, 'shutdown_timeout': 120}
(APIServer pid=2762128) INFO 07-12 17:02:37 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2762128) INFO 07-12 17:02:37 [model.py:1725] Using max model len 40960
(APIServer pid=2762128) INFO 07-12 17:02:37 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2762128) INFO 07-12 17:02:37 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2762128) INFO 07-12 17:02:37 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2762128) INFO 07-12 17:02:40 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2762505) INFO 07-12 17:02:49 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2762505) WARNING 07-12 17:02:49 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2762505) INFO 07-12 17:02:49 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4
(Worker pid=2762723) INFO 07-12 17:03:01 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:59219 backend=nccl
(Worker pid=2762726) INFO 07-12 17:03:01 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:59219 backend=nccl
(Worker pid=2762724) INFO 07-12 17:03:01 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:59219 backend=nccl
(Worker pid=2762725) INFO 07-12 17:03:01 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:59219 backend=nccl
(Worker pid=2762723) INFO 07-12 17:03:02 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2762723) INFO 07-12 17:03:04 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2762723) INFO 07-12 17:03:04 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2762723) INFO 07-12 17:03:04 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2762723) INFO 07-12 17:03:05 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2762723) INFO 07-12 17:03:05 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2762723) INFO 07-12 17:03:05 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2762723) INFO 07-12 17:03:05 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2762723) INFO 07-12 17:03:05 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2762723) INFO 07-12 17:03:06 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.19 GiB.
(Worker_TP0 pid=2762723) INFO 07-12 17:03:06 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:05, 2.65it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:00<00:05, 2.44it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:05, 2.43it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:01<00:05, 2.37it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:02<00:04, 2.35it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:02<00:04, 2.31it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:02<00:03, 2.31it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:03<00:03, 2.33it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:03<00:02, 2.34it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:04<00:02, 2.35it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:04<00:02, 2.35it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:05<00:01, 2.35it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:05<00:01, 2.34it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:05<00:00, 2.32it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:06<00:00, 2.32it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:06<00:00, 3.01it/s]
(Worker_TP0 pid=2762723)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:06<00:00, 2.46it/s]
(Worker_TP0 pid=2762723)
(Worker_TP0 pid=2762723) INFO 07-12 17:03:13 [default_loader.py:430] Loading weights took 6.51 seconds
(Worker_TP0 pid=2762723) INFO 07-12 17:03:13 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2762723) INFO 07-12 17:03:13 [gpu_model_runner.py:5259] Model loading took 14.25 GiB memory and 7.585942 seconds
(Worker_TP0 pid=2762723) INFO 07-12 17:03:17 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/fe5024cfcf/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2762723) INFO 07-12 17:03:17 [backends.py:1148] Dynamo bytecode transform time: 3.22 s
(Worker_TP0 pid=2762723) INFO 07-12 17:03:17 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2762723) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2762723) return func(*args, **kwargs)
[rank0]:[W712 17:03:18.211683852 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2762723) INFO 07-12 17:03:19 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP2 pid=2762725) INFO 07-12 17:03:24 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/8f59e8024c8d60aba6fc658297e6ebafd45eb9986d8df3183f6b00fe04a9daae/rank_2_0/model
(Worker_TP3 pid=2762726) INFO 07-12 17:03:24 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/8f59e8024c8d60aba6fc658297e6ebafd45eb9986d8df3183f6b00fe04a9daae/rank_3_0/model
(Worker_TP0 pid=2762723) INFO 07-12 17:03:24 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 512) from the cache, took 3.744 s
(Worker_TP1 pid=2762724) INFO 07-12 17:03:24 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/8f59e8024c8d60aba6fc658297e6ebafd45eb9986d8df3183f6b00fe04a9daae/rank_1_0/model
(Worker_TP0 pid=2762723) INFO 07-12 17:03:24 [backends.py:292] Directly load the compiled graph(s) for compile range (513, 8192) from the cache, took 3.831 s
(Worker_TP0 pid=2762723) INFO 07-12 17:03:24 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/8f59e8024c8d60aba6fc658297e6ebafd45eb9986d8df3183f6b00fe04a9daae/rank_0_0/model
(Worker_TP0 pid=2762723) INFO 07-12 17:03:24 [monitor.py:53] torch.compile took 9.98 s in total
(Worker_TP0 pid=2762723) INFO 07-12 17:03:24 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2762723) INFO 07-12 17:03:24 [monitor.py:81] Initial profiling/warmup run took 0.24 s
(Worker_TP0 pid=2762723) INFO 07-12 17:03:25 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(Worker_TP2 pid=2762725) INFO 07-12 17:03:25 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(Worker_TP1 pid=2762724) INFO 07-12 17:03:25 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(Worker_TP3 pid=2762726) INFO 07-12 17:03:25 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(Worker_TP3 pid=2762726) INFO 07-12 17:03:26 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2762723) INFO 07-12 17:03:26 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP2 pid=2762725) INFO 07-12 17:03:26 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2762724) INFO 07-12 17:03:26 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2762726) INFO 07-12 17:03:27 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.08 GiB total
(Worker_TP0 pid=2762723) INFO 07-12 17:03:27 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.08 GiB total
(Worker_TP2 pid=2762725) INFO 07-12 17:03:27 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.08 GiB total
(Worker_TP1 pid=2762724) INFO 07-12 17:03:27 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.08 GiB total
(Worker_TP3 pid=2762726) INFO 07-12 17:03:27 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9192 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9208. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=2762723) INFO 07-12 17:03:27 [gpu_worker.py:508] Available KV cache memory: 71.81 GiB
(Worker_TP0 pid=2762723) INFO 07-12 17:03:27 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9192 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9208. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP2 pid=2762725) INFO 07-12 17:03:27 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9192 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9208. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP1 pid=2762724) INFO 07-12 17:03:27 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9192 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9208. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2762505) INFO 07-12 17:03:27 [kv_cache_utils.py:2146] GPU KV cache size: 3,137,376 tokens
(EngineCore pid=2762505) INFO 07-12 17:03:27 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 76.60x
(Worker_TP1 pid=2762724) INFO 07-12 17:03:27 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP3 pid=2762726) (Worker_TP0 pid=2762723) (Worker_TP2 pid=2762725) INFO 07-12 17:03:27 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 17:03:27 [gpu_worker.py:648] Compile and warming up model for size 8192
INFO 07-12 17:03:27 [gpu_worker.py:648] Compile and warming up model for size 8192
(Worker_TP0 pid=2762723) INFO 07-12 17:03:27 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2762723) INFO 07-12 17:03:27 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP0 pid=2762723) 2026-07-12 17:03:27,723 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP2 pid=2762725) 2026-07-12 17:03:27,724 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP3 pid=2762726) 2026-07-12 17:03:27,724 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2762724) 2026-07-12 17:03:27,725 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2762723) 2026-07-12 17:03:27,773 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP3 pid=2762726) 2026-07-12 17:03:27,775 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP2 pid=2762725) 2026-07-12 17:03:27,777 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=2762724) 2026-07-12 17:03:27,778 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2762723)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/5 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 20%|██ | 1/5 [00:00<00:00, 8.02it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 40%|████ | 2/5 [00:00<00:00, 8.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 60%|██████ | 3/5 [00:00<00:00, 8.48it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 80%|████████ | 4/5 [00:00<00:00, 7.45it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:00<00:00, 6.71it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:00<00:00, 7.24it/s]
(Worker_TP0 pid=2762723)
Capturing CUDA graphs (decode, FULL): 0%| | 0/4 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 25%|██▌ | 1/4 [00:00<00:00, 8.74it/s]
Capturing CUDA graphs (decode, FULL): 50%|█████ | 2/4 [00:00<00:00, 9.11it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 4/4 [00:00<00:00, 9.50it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 4/4 [00:00<00:00, 9.38it/s]
(Worker_TP0 pid=2762723) INFO 07-12 17:03:29 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP3 pid=2762726) INFO 07-12 17:03:29 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2762724) INFO 07-12 17:03:29 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP2 pid=2762725) INFO 07-12 17:03:29 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2762723) INFO 07-12 17:03:29 [gpu_model_runner.py:6660] Graph capturing finished in 2 secs, took 0.06 GiB
(Worker_TP0 pid=2762723) INFO 07-12 17:03:29 [gpu_worker.py:667] CUDA graph pool memory: 0.06 GiB (actual), 0.08 GiB (estimated), difference: 0.01 GiB (21.9%).
(Worker_TP3 pid=2762726) INFO 07-12 17:03:29 [gpu_worker.py:667] CUDA graph pool memory: 0.06 GiB (actual), 0.08 GiB (estimated), difference: 0.01 GiB (21.9%).
(Worker_TP1 pid=2762724) INFO 07-12 17:03:29 [gpu_worker.py:667] CUDA graph pool memory: 0.06 GiB (actual), 0.08 GiB (estimated), difference: 0.01 GiB (21.9%).
(Worker_TP2 pid=2762725) INFO 07-12 17:03:29 [gpu_worker.py:667] CUDA graph pool memory: 0.06 GiB (actual), 0.08 GiB (estimated), difference: 0.01 GiB (21.9%).
(Worker_TP3 pid=2762726) INFO 07-12 17:03:29 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP1 pid=2762724) INFO 07-12 17:03:29 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP0 pid=2762723) INFO 07-12 17:03:29 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP2 pid=2762725) INFO 07-12 17:03:29 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2762505) INFO 07-12 17:03:30 [core.py:337] init engine (profile, create kv cache, warmup model) took 15.98 s (compilation: 9.98 s)
(EngineCore pid=2762505) INFO 07-12 17:03:31 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp4_mns8/opprof/opprof-v1-dp0-pid2762505-1783875811578126796.jsonl
(EngineCore pid=2762505) INFO 07-12 17:03:31 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2762505) INFO 07-12 17:03:31 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2762505) INFO 07-12 17:03:32 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2762128) INFO 07-12 17:03:32 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2762128) WARNING 07-12 17:03:32 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2762128) INFO 07-12 17:03:33 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2762128) INFO 07-12 17:03:33 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8707
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:37] Available routes are:
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2762128) INFO 07-12 17:03:33 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2762128) INFO: Started server process [2762128]
(APIServer pid=2762128) INFO: Waiting for application startup.
(APIServer pid=2762128) INFO: Application startup complete.
(APIServer pid=2762128) INFO: 127.0.0.1:35216 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2762723) WARNING 07-12 17:03:39 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2762128) INFO: 127.0.0.1:35244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2762723) WARNING 07-12 17:03:39 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2762128) INFO: 127.0.0.1:35246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:03:43 [loggers.py:273] Engine 000: Avg prompt throughput: 3980.4 tokens/s, Avg generation throughput: 174.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 10.3%
(APIServer pid=2762128) INFO: 127.0.0.1:33874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:03:53 [loggers.py:273] Engine 000: Avg prompt throughput: 746.5 tokens/s, Avg generation throughput: 219.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 50.8%
(APIServer pid=2762128) INFO: 127.0.0.1:40722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:04:03 [loggers.py:273] Engine 000: Avg prompt throughput: 8887.1 tokens/s, Avg generation throughput: 552.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 33.5%
(APIServer pid=2762128) INFO: 127.0.0.1:40644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:04:13 [loggers.py:273] Engine 000: Avg prompt throughput: 16161.2 tokens/s, Avg generation throughput: 609.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 24.0%
(APIServer pid=2762128) INFO: 127.0.0.1:45454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:04:23 [loggers.py:273] Engine 000: Avg prompt throughput: 13736.0 tokens/s, Avg generation throughput: 675.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 20.7%
(APIServer pid=2762128) INFO: 127.0.0.1:34960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:35230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:04:33 [loggers.py:273] Engine 000: Avg prompt throughput: 13839.3 tokens/s, Avg generation throughput: 617.6 tokens/s, Running: 8 reqs, Waiting: 2 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 19.2%
(APIServer pid=2762128) INFO: 127.0.0.1:52292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:04:43 [loggers.py:273] Engine 000: Avg prompt throughput: 14753.9 tokens/s, Avg generation throughput: 766.6 tokens/s, Running: 8 reqs, Waiting: 2 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 17.2%
(APIServer pid=2762128) INFO: 127.0.0.1:32846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:04:53 [loggers.py:273] Engine 000: Avg prompt throughput: 9315.0 tokens/s, Avg generation throughput: 500.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 16.4%
(APIServer pid=2762128) INFO: 127.0.0.1:60412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:05:03 [loggers.py:273] Engine 000: Avg prompt throughput: 17.5 tokens/s, Avg generation throughput: 301.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 22.6%
(APIServer pid=2762128) INFO: 127.0.0.1:42898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:43204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:05:13 [loggers.py:273] Engine 000: Avg prompt throughput: 41.3 tokens/s, Avg generation throughput: 656.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 30.3%
(APIServer pid=2762128) INFO: 127.0.0.1:53242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:05:23 [loggers.py:273] Engine 000: Avg prompt throughput: 42.7 tokens/s, Avg generation throughput: 631.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 40.6%
(APIServer pid=2762128) INFO: 127.0.0.1:56718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:05:33 [loggers.py:273] Engine 000: Avg prompt throughput: 37.2 tokens/s, Avg generation throughput: 573.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 45.6%
(APIServer pid=2762128) INFO: 127.0.0.1:46986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:46990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:05:43 [loggers.py:273] Engine 000: Avg prompt throughput: 57.0 tokens/s, Avg generation throughput: 750.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 51.8%
(APIServer pid=2762128) INFO: 127.0.0.1:32876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:32994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:05:53 [loggers.py:273] Engine 000: Avg prompt throughput: 44.4 tokens/s, Avg generation throughput: 697.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 55.3%
(APIServer pid=2762128) INFO: 127.0.0.1:55366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:06:03 [loggers.py:273] Engine 000: Avg prompt throughput: 20.3 tokens/s, Avg generation throughput: 331.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 56.9%
(APIServer pid=2762128) INFO: 127.0.0.1:47702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:06:13 [loggers.py:273] Engine 000: Avg prompt throughput: 33.2 tokens/s, Avg generation throughput: 532.9 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 59.1%
(APIServer pid=2762128) INFO: 127.0.0.1:55704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:06:23 [loggers.py:273] Engine 000: Avg prompt throughput: 37.6 tokens/s, Avg generation throughput: 619.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 62.1%
(APIServer pid=2762128) INFO: 127.0.0.1:44986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:06:33 [loggers.py:273] Engine 000: Avg prompt throughput: 46.4 tokens/s, Avg generation throughput: 702.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 65.3%
(APIServer pid=2762128) INFO: 127.0.0.1:55986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:56184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:06:43 [loggers.py:273] Engine 000: Avg prompt throughput: 36.0 tokens/s, Avg generation throughput: 513.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 67.0%
(APIServer pid=2762128) INFO: 127.0.0.1:34528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:06:53 [loggers.py:273] Engine 000: Avg prompt throughput: 55.8 tokens/s, Avg generation throughput: 756.4 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 69.2%
(APIServer pid=2762128) INFO: 127.0.0.1:36822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:36998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:53788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:07:03 [loggers.py:273] Engine 000: Avg prompt throughput: 45.5 tokens/s, Avg generation throughput: 689.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 70.8%
(APIServer pid=2762128) INFO: 127.0.0.1:51640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:07:13 [loggers.py:273] Engine 000: Avg prompt throughput: 367.5 tokens/s, Avg generation throughput: 229.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 71.2%
(APIServer pid=2762128) INFO: 127.0.0.1:51756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:07:23 [loggers.py:273] Engine 000: Avg prompt throughput: 283.4 tokens/s, Avg generation throughput: 600.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 72.2%
(APIServer pid=2762128) INFO: 127.0.0.1:55110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:33994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:07:33 [loggers.py:273] Engine 000: Avg prompt throughput: 350.4 tokens/s, Avg generation throughput: 591.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 74.9%
(APIServer pid=2762128) INFO: 127.0.0.1:34106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:07:43 [loggers.py:273] Engine 000: Avg prompt throughput: 43.9 tokens/s, Avg generation throughput: 663.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 78.8%
(APIServer pid=2762128) INFO: 127.0.0.1:51974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:51992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:52262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:07:53 [loggers.py:273] Engine 000: Avg prompt throughput: 486.3 tokens/s, Avg generation throughput: 633.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 83.9%
(APIServer pid=2762128) INFO: 127.0.0.1:40862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:40990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:41302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:08:03 [loggers.py:273] Engine 000: Avg prompt throughput: 739.6 tokens/s, Avg generation throughput: 857.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 89.8%
(APIServer pid=2762128) INFO: 127.0.0.1:34384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:34746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:08:13 [loggers.py:273] Engine 000: Avg prompt throughput: 459.0 tokens/s, Avg generation throughput: 520.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 93.0%
(APIServer pid=2762128) INFO: 127.0.0.1:47064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:08:23 [loggers.py:273] Engine 000: Avg prompt throughput: 18.4 tokens/s, Avg generation throughput: 304.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 94.8%
(APIServer pid=2762128) INFO: 127.0.0.1:50260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:50610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:08:33 [loggers.py:273] Engine 000: Avg prompt throughput: 40.9 tokens/s, Avg generation throughput: 646.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 98.9%
(APIServer pid=2762128) INFO: 127.0.0.1:47838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:08:43 [loggers.py:273] Engine 000: Avg prompt throughput: 39.6 tokens/s, Avg generation throughput: 649.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 98.9%
(APIServer pid=2762128) INFO: 127.0.0.1:55072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:08:53 [loggers.py:273] Engine 000: Avg prompt throughput: 42.0 tokens/s, Avg generation throughput: 577.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 98.9%
(APIServer pid=2762128) INFO: 127.0.0.1:60674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:60948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:54994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:09:03 [loggers.py:273] Engine 000: Avg prompt throughput: 56.0 tokens/s, Avg generation throughput: 768.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 98.9%
(APIServer pid=2762128) INFO: 127.0.0.1:55154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:55590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:09:13 [loggers.py:273] Engine 000: Avg prompt throughput: 49.2 tokens/s, Avg generation throughput: 752.9 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 98.9%
(APIServer pid=2762128) INFO: 127.0.0.1:45580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:45832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:09:23 [loggers.py:273] Engine 000: Avg prompt throughput: 21.7 tokens/s, Avg generation throughput: 360.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 98.9%
(APIServer pid=2762128) INFO: 127.0.0.1:37910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:37994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:38338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:09:33 [loggers.py:273] Engine 000: Avg prompt throughput: 9737.6 tokens/s, Avg generation throughput: 865.1 tokens/s, Running: 8 reqs, Waiting: 11 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 95.7%
(APIServer pid=2762128) INFO: 127.0.0.1:47974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:47992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO: 127.0.0.1:42380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2762128) INFO 07-12 17:09:43 [loggers.py:273] Engine 000: Avg prompt throughput: 7415.5 tokens/s, Avg generation throughput: 993.8 tokens/s, Running: 8 reqs, Waiting: 36 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 93.2%
(APIServer pid=2762128) INFO 07-12 17:09:49 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2762128) INFO 07-12 17:09:49 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2762128) INFO 07-12 17:09:49 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2762128) INFO 07-12 17:09:49 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2762505) INFO 07-12 17:09:49 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2762505) INFO 07-12 17:09:49 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2762505) INFO 07-12 17:09:49 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2762505) INFO 07-12 17:09:49 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(Worker_TP0 pid=2762723) INFO 07-12 17:09:49 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(EngineCore pid=2762505) INFO 07-12 17:09:49 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=4
(APIServer pid=2762128) INFO: Shutting down
(APIServer pid=2762128) INFO: Waiting for application shutdown.
(APIServer pid=2762128) INFO: Application shutdown complete.
(APIServer pid=2762128) INFO: Finished server process [2762128]
(EngineCore pid=2762505) INFO 07-12 17:09:53 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2762128) INFO 07-12 17:09:55 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2762128) INFO 07-12 17:09:55 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2762128) INFO 07-12 17:09:55 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink