gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a3b25f4a92c88d338a3437e91fdec8fc88ea8a9c
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp2_mns32/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

1505 lines
155 KiB
Plaintext
Raw Blame History

(APIServer pid=2730322) INFO 07-12 16:31:23 [api_utils.py:339]
(APIServer pid=2730322) INFO 07-12 16:31:23 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2730322) INFO 07-12 16:31:23 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2730322) INFO 07-12 16:31:23 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2730322) INFO 07-12 16:31:23 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2730322) INFO 07-12 16:31:23 [api_utils.py:339]
(APIServer pid=2730322) INFO 07-12 16:31:23 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8702, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 2, 'max_num_batched_tokens': 8192, 'max_num_seqs': 32, 'shutdown_timeout': 120}
(APIServer pid=2730322) INFO 07-12 16:31:23 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2730322) INFO 07-12 16:31:23 [model.py:1725] Using max model len 40960
(APIServer pid=2730322) INFO 07-12 16:31:23 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2730322) INFO 07-12 16:31:23 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2730322) INFO 07-12 16:31:23 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2730322) INFO 07-12 16:31:25 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2730662) INFO 07-12 16:31:35 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2730662) WARNING 07-12 16:31:35 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2730662) INFO 07-12 16:31:35 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2
(Worker pid=2730817) INFO 07-12 16:31:45 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:54663 backend=nccl
(Worker pid=2730818) INFO 07-12 16:31:45 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:54663 backend=nccl
(Worker pid=2730817) INFO 07-12 16:31:46 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2730817) INFO 07-12 16:31:47 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2730817) INFO 07-12 16:31:47 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2730817) INFO 07-12 16:31:47 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2730817) INFO 07-12 16:31:48 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2730817) INFO 07-12 16:31:48 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2730817) INFO 07-12 16:31:48 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2730817) INFO 07-12 16:31:48 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2730817) INFO 07-12 16:31:48 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2730817) INFO 07-12 16:31:49 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1286.57 GiB.
(Worker_TP0 pid=2730817) INFO 07-12 16:31:49 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:09, 1.66it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:09, 1.53it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:08, 1.49it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:02<00:08, 1.45it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:03<00:07, 1.42it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:04<00:07, 1.40it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:04<00:06, 1.39it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:05<00:05, 1.41it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:06<00:04, 1.41it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:06<00:04, 1.44it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:07<00:03, 1.46it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:08<00:02, 1.48it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:08<00:02, 1.49it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:09<00:01, 1.48it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:10<00:00, 1.45it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.86it/s]
(Worker_TP0 pid=2730817)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.52it/s]
(Worker_TP0 pid=2730817)
(Worker_TP0 pid=2730817) INFO 07-12 16:31:59 [default_loader.py:430] Loading weights took 10.54 seconds
(Worker_TP0 pid=2730817) INFO 07-12 16:31:59 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2730817) INFO 07-12 16:32:00 [gpu_model_runner.py:5259] Model loading took 28.46 GiB memory and 10.981014 seconds
(Worker_TP0 pid=2730817) INFO 07-12 16:32:03 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/d2f6c70834/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2730817) INFO 07-12 16:32:03 [backends.py:1148] Dynamo bytecode transform time: 3.45 s
(Worker_TP0 pid=2730817) INFO 07-12 16:32:03 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2730817) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2730817) return func(*args, **kwargs)
[rank0]:[W712 16:32:04.203632029 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2730817) INFO 07-12 16:32:04 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP1 pid=2730818) INFO 07-12 16:32:07 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/597ec81617fac86c912b3de6d61fe206016c47a08580acb57fd5cb1fd717f71f/rank_1_0/model
(Worker_TP0 pid=2730817) INFO 07-12 16:32:07 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 2.414 s
(Worker_TP0 pid=2730817) INFO 07-12 16:32:07 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/597ec81617fac86c912b3de6d61fe206016c47a08580acb57fd5cb1fd717f71f/rank_0_0/model
(Worker_TP0 pid=2730817) INFO 07-12 16:32:07 [monitor.py:53] torch.compile took 7.30 s in total
(Worker_TP0 pid=2730817) INFO 07-12 16:32:07 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2730817) INFO 07-12 16:32:08 [monitor.py:81] Initial profiling/warmup run took 0.25 s
(Worker_TP1 pid=2730818) INFO 07-12 16:32:08 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=11 (largest=64), FULL=7 (largest=32)
(Worker_TP0 pid=2730817) INFO 07-12 16:32:08 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=11 (largest=64), FULL=7 (largest=32)
(Worker_TP0 pid=2730817) INFO 07-12 16:32:09 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2730818) INFO 07-12 16:32:09 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2730817) INFO 07-12 16:32:10 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.11 GiB total
(Worker_TP1 pid=2730818) INFO 07-12 16:32:10 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.11 GiB total
(Worker_TP0 pid=2730817) INFO 07-12 16:32:10 [gpu_worker.py:508] Available KV cache memory: 57.35 GiB
(Worker_TP0 pid=2730817) INFO 07-12 16:32:10 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9188 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9212. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP1 pid=2730818) INFO 07-12 16:32:10 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9188 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9212. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2730662) INFO 07-12 16:32:10 [kv_cache_utils.py:2146] GPU KV cache size: 1,252,928 tokens
(EngineCore pid=2730662) INFO 07-12 16:32:10 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 30.59x
(Worker_TP0 pid=2730817) INFO 07-12 16:32:10 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2730817) INFO 07-12 16:32:10 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP0 pid=2730817) 2026-07-12 16:32:10,962 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1 pid=2730818) 2026-07-12 16:32:10,963 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2730817) 2026-07-12 16:32:11,021 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=2730818) 2026-07-12 16:32:11,022 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2730817)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 9%|▉ | 1/11 [00:00<00:01, 8.25it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 18%|█▊ | 2/11 [00:00<00:01, 8.57it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 27%|██▋ | 3/11 [00:00<00:00, 8.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 36%|███▋ | 4/11 [00:00<00:00, 8.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 45%|████▌ | 5/11 [00:00<00:00, 8.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 55%|█████▍ | 6/11 [00:00<00:00, 8.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 64%|██████▎ | 7/11 [00:00<00:00, 8.32it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 73%|███████▎ | 8/11 [00:00<00:00, 7.39it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 82%|████████▏ | 9/11 [00:01<00:00, 6.89it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 91%|█████████ | 10/11 [00:01<00:00, 6.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:01<00:00, 6.34it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:01<00:00, 7.32it/s]
(Worker_TP0 pid=2730817)
Capturing CUDA graphs (decode, FULL): 0%| | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 14%|█▍ | 1/7 [00:00<00:00, 8.97it/s]
Capturing CUDA graphs (decode, FULL): 29%|██▊ | 2/7 [00:00<00:00, 9.33it/s]
Capturing CUDA graphs (decode, FULL): 43%|████▎ | 3/7 [00:00<00:00, 9.27it/s]
Capturing CUDA graphs (decode, FULL): 57%|█████▋ | 4/7 [00:00<00:00, 8.51it/s]
Capturing CUDA graphs (decode, FULL): 71%|███████▏ | 5/7 [00:00<00:00, 8.27it/s]
Capturing CUDA graphs (decode, FULL): 86%|████████▌ | 6/7 [00:00<00:00, 8.33it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:00<00:00, 8.74it/s]
(Worker_TP0 pid=2730817) INFO 07-12 16:32:13 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2730818) INFO 07-12 16:32:13 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2730817) INFO 07-12 16:32:14 [gpu_model_runner.py:6660] Graph capturing finished in 3 secs, took 0.13 GiB
(Worker_TP0 pid=2730817) INFO 07-12 16:32:14 [gpu_worker.py:667] CUDA graph pool memory: 0.13 GiB (actual), 0.11 GiB (estimated), difference: 0.02 GiB (14.9%).
(Worker_TP1 pid=2730818) INFO 07-12 16:32:14 [gpu_worker.py:667] CUDA graph pool memory: 0.13 GiB (actual), 0.11 GiB (estimated), difference: 0.02 GiB (14.9%).
(Worker_TP1 pid=2730818) INFO 07-12 16:32:14 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP0 pid=2730817) INFO 07-12 16:32:14 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2730662) INFO 07-12 16:32:14 [core.py:337] init engine (profile, create kv cache, warmup model) took 14.19 s (compilation: 7.30 s)
(EngineCore pid=2730662) INFO 07-12 16:32:15 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp2_mns32/opprof/opprof-v1-dp0-pid2730662-1783873935616837232.jsonl
(EngineCore pid=2730662) INFO 07-12 16:32:15 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2730662) INFO 07-12 16:32:15 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2730662) INFO 07-12 16:32:16 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2730322) INFO 07-12 16:32:16 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2730322) WARNING 07-12 16:32:16 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2730322) INFO 07-12 16:32:17 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2730322) INFO 07-12 16:32:17 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8702
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:37] Available routes are:
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2730322) INFO 07-12 16:32:17 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2730322) INFO: Started server process [2730322]
(APIServer pid=2730322) INFO: Waiting for application startup.
(APIServer pid=2730322) INFO: Application startup complete.
(APIServer pid=2730322) INFO: 127.0.0.1:38554 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2730817) WARNING 07-12 16:32:24 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2730322) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2730817) WARNING 07-12 16:32:24 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2730322) INFO: 127.0.0.1:49240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:32:27 [loggers.py:273] Engine 000: Avg prompt throughput: 3778.3 tokens/s, Avg generation throughput: 168.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 6.5%
(APIServer pid=2730322) INFO: 127.0.0.1:57642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:32:37 [loggers.py:273] Engine 000: Avg prompt throughput: 1644.8 tokens/s, Avg generation throughput: 190.3 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 42.4%
(APIServer pid=2730322) INFO: 127.0.0.1:57848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:32:47 [loggers.py:273] Engine 000: Avg prompt throughput: 15310.2 tokens/s, Avg generation throughput: 452.7 tokens/s, Running: 32 reqs, Waiting: 12 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 22.7%
(APIServer pid=2730322) INFO: 127.0.0.1:35708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:54650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:32:57 [loggers.py:273] Engine 000: Avg prompt throughput: 17855.4 tokens/s, Avg generation throughput: 849.0 tokens/s, Running: 25 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.4%, Prefix cache hit rate: 16.4%
(APIServer pid=2730322) INFO: 127.0.0.1:44792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:33:07 [loggers.py:273] Engine 000: Avg prompt throughput: 19.3 tokens/s, Avg generation throughput: 423.3 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 25.7%
(APIServer pid=2730322) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:33:17 [loggers.py:273] Engine 000: Avg prompt throughput: 61.0 tokens/s, Avg generation throughput: 828.9 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 46.7%
(APIServer pid=2730322) INFO: 127.0.0.1:45280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:45362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:33:27 [loggers.py:273] Engine 000: Avg prompt throughput: 3641.6 tokens/s, Avg generation throughput: 779.7 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 53.8%
(APIServer pid=2730322) INFO: 127.0.0.1:35384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:35540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:33:37 [loggers.py:273] Engine 000: Avg prompt throughput: 17464.0 tokens/s, Avg generation throughput: 721.1 tokens/s, Running: 32 reqs, Waiting: 2 reqs, GPU KV cache usage: 7.0%, Prefix cache hit rate: 46.6%
(APIServer pid=2730322) INFO: 127.0.0.1:40946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:60992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:33:47 [loggers.py:273] Engine 000: Avg prompt throughput: 17058.9 tokens/s, Avg generation throughput: 760.9 tokens/s, Running: 32 reqs, Waiting: 1 reqs, GPU KV cache usage: 7.5%, Prefix cache hit rate: 41.4%
(APIServer pid=2730322) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:32994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:33002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:33:57 [loggers.py:273] Engine 000: Avg prompt throughput: 14332.9 tokens/s, Avg generation throughput: 796.3 tokens/s, Running: 32 reqs, Waiting: 8 reqs, GPU KV cache usage: 7.3%, Prefix cache hit rate: 38.3%
(APIServer pid=2730322) INFO: 127.0.0.1:56706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:34:07 [loggers.py:273] Engine 000: Avg prompt throughput: 20273.1 tokens/s, Avg generation throughput: 660.3 tokens/s, Running: 32 reqs, Waiting: 6 reqs, GPU KV cache usage: 10.5%, Prefix cache hit rate: 34.2%
(APIServer pid=2730322) INFO: 127.0.0.1:41174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:34:17 [loggers.py:273] Engine 000: Avg prompt throughput: 2296.7 tokens/s, Avg generation throughput: 353.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 34.8%
(APIServer pid=2730322) INFO: 127.0.0.1:41232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:34:27 [loggers.py:273] Engine 000: Avg prompt throughput: 58.6 tokens/s, Avg generation throughput: 754.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 40.6%
(APIServer pid=2730322) INFO: 127.0.0.1:57440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:34:37 [loggers.py:273] Engine 000: Avg prompt throughput: 63.8 tokens/s, Avg generation throughput: 881.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 46.8%
(APIServer pid=2730322) INFO: 127.0.0.1:56576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:56666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:34:47 [loggers.py:273] Engine 000: Avg prompt throughput: 61.3 tokens/s, Avg generation throughput: 831.6 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 51.6%
(APIServer pid=2730322) INFO: 127.0.0.1:42126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:34:57 [loggers.py:273] Engine 000: Avg prompt throughput: 47.3 tokens/s, Avg generation throughput: 804.4 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 55.0%
(APIServer pid=2730322) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:49128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:35:07 [loggers.py:273] Engine 000: Avg prompt throughput: 54.1 tokens/s, Avg generation throughput: 845.7 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 57.8%
(APIServer pid=2730322) INFO: 127.0.0.1:39382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:39508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:35:17 [loggers.py:273] Engine 000: Avg prompt throughput: 48.3 tokens/s, Avg generation throughput: 834.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 61.4%
(APIServer pid=2730322) INFO: 127.0.0.1:40818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:35:27 [loggers.py:273] Engine 000: Avg prompt throughput: 23.4 tokens/s, Avg generation throughput: 270.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 62.5%
(APIServer pid=2730322) INFO: 127.0.0.1:40982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:40996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:41158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:35:37 [loggers.py:273] Engine 000: Avg prompt throughput: 63.1 tokens/s, Avg generation throughput: 862.5 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 65.3%
(APIServer pid=2730322) INFO: 127.0.0.1:38024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:38218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:35:47 [loggers.py:273] Engine 000: Avg prompt throughput: 60.9 tokens/s, Avg generation throughput: 797.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 70.4%
(APIServer pid=2730322) INFO: 127.0.0.1:42920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:42994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:43052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:35:57 [loggers.py:273] Engine 000: Avg prompt throughput: 58.1 tokens/s, Avg generation throughput: 856.2 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 76.1%
(APIServer pid=2730322) INFO: 127.0.0.1:53826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:53966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:36:07 [loggers.py:273] Engine 000: Avg prompt throughput: 49.1 tokens/s, Avg generation throughput: 795.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 76.1%
(APIServer pid=2730322) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:36:17 [loggers.py:273] Engine 000: Avg prompt throughput: 56.2 tokens/s, Avg generation throughput: 848.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 76.1%
(APIServer pid=2730322) INFO: 127.0.0.1:47938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:47992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:48092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO: 127.0.0.1:51224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2730322) INFO 07-12 16:36:25 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2730322) INFO 07-12 16:36:25 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2730322) INFO 07-12 16:36:25 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2730322) INFO 07-12 16:36:25 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2730662) INFO 07-12 16:36:25 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2730662) INFO 07-12 16:36:25 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2730662) INFO 07-12 16:36:25 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2730662) INFO 07-12 16:36:25 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(Worker_TP0 pid=2730817) INFO 07-12 16:36:25 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(EngineCore pid=2730662) INFO 07-12 16:36:25 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2
(APIServer pid=2730322) INFO: Shutting down
(APIServer pid=2730322) INFO: Waiting for application shutdown.
(APIServer pid=2730322) INFO: Application shutdown complete.
(APIServer pid=2730322) INFO: Finished server process [2730322]
(EngineCore pid=2730662) INFO 07-12 16:36:29 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2730322) INFO 07-12 16:36:30 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2730322) INFO 07-12 16:36:30 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2730322) INFO 07-12 16:36:30 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink