gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2261818994a2e3a7960fc2d6c30187a953333ac4
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp2_mns64/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

1270 lines
135 KiB
Plaintext
Raw Blame History

(APIServer pid=2735795) INFO 07-12 16:36:43 [api_utils.py:339]
(APIServer pid=2735795) INFO 07-12 16:36:43 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2735795) INFO 07-12 16:36:43 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2735795) INFO 07-12 16:36:43 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2735795) INFO 07-12 16:36:43 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2735795) INFO 07-12 16:36:43 [api_utils.py:339]
(APIServer pid=2735795) INFO 07-12 16:36:43 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8703, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 2, 'max_num_batched_tokens': 8192, 'max_num_seqs': 64, 'shutdown_timeout': 120}
(APIServer pid=2735795) INFO 07-12 16:36:43 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2735795) INFO 07-12 16:36:43 [model.py:1725] Using max model len 40960
(APIServer pid=2735795) INFO 07-12 16:36:43 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2735795) INFO 07-12 16:36:43 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2735795) INFO 07-12 16:36:43 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2735795) INFO 07-12 16:36:45 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2736153) INFO 07-12 16:36:54 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2736153) WARNING 07-12 16:36:54 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2736153) INFO 07-12 16:36:54 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2
(Worker pid=2736308) INFO 07-12 16:37:04 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:51703 backend=nccl
(Worker pid=2736309) INFO 07-12 16:37:04 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:51703 backend=nccl
(Worker pid=2736308) INFO 07-12 16:37:05 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2736308) INFO 07-12 16:37:06 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2736308) INFO 07-12 16:37:07 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2736308) INFO 07-12 16:37:07 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2736308) INFO 07-12 16:37:07 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2736308) INFO 07-12 16:37:07 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2736308) INFO 07-12 16:37:08 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2736308) INFO 07-12 16:37:08 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2736308) INFO 07-12 16:37:08 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2736308) INFO 07-12 16:37:08 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1286.55 GiB.
(Worker_TP0 pid=2736308) INFO 07-12 16:37:08 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:09, 1.57it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:09, 1.49it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:08, 1.50it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:02<00:08, 1.45it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:03<00:07, 1.43it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:04<00:07, 1.40it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:04<00:06, 1.40it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:05<00:05, 1.43it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:06<00:04, 1.44it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:06<00:04, 1.46it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:07<00:03, 1.46it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:08<00:02, 1.48it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:08<00:02, 1.48it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:09<00:01, 1.47it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:10<00:00, 1.47it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.90it/s]
(Worker_TP0 pid=2736308)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.53it/s]
(Worker_TP0 pid=2736308)
(Worker_TP0 pid=2736308) INFO 07-12 16:37:19 [default_loader.py:430] Loading weights took 10.46 seconds
(Worker_TP0 pid=2736308) INFO 07-12 16:37:19 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2736308) INFO 07-12 16:37:19 [gpu_model_runner.py:5259] Model loading took 28.46 GiB memory and 10.930648 seconds
(Worker_TP0 pid=2736308) INFO 07-12 16:37:23 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/7a9372c548/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2736308) INFO 07-12 16:37:23 [backends.py:1148] Dynamo bytecode transform time: 3.36 s
(Worker_TP0 pid=2736308) INFO 07-12 16:37:23 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2736308) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2736308) return func(*args, **kwargs)
[rank0]:[W712 16:37:23.673795145 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2736308) INFO 07-12 16:37:24 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP0 pid=2736308) INFO 07-12 16:37:27 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 2.356 s
(Worker_TP0 pid=2736308) INFO 07-12 16:37:27 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/7c6e0ddc6dcbaf895fc62cc90d8a628d2dfd8c5b38e2d7747d4762b6a26b182b/rank_0_0/model
(Worker_TP0 pid=2736308) INFO 07-12 16:37:27 [monitor.py:53] torch.compile took 7.25 s in total
(Worker_TP1 pid=2736309) INFO 07-12 16:37:27 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/7c6e0ddc6dcbaf895fc62cc90d8a628d2dfd8c5b38e2d7747d4762b6a26b182b/rank_1_0/model
(Worker_TP0 pid=2736308) INFO 07-12 16:37:27 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2736308) INFO 07-12 16:37:27 [monitor.py:81] Initial profiling/warmup run took 0.30 s
(Worker_TP1 pid=2736309) INFO 07-12 16:37:28 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=19 (largest=128), FULL=11 (largest=64)
(Worker_TP0 pid=2736308) INFO 07-12 16:37:28 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=19 (largest=128), FULL=11 (largest=64)
(Worker_TP0 pid=2736308) INFO 07-12 16:37:29 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2736309) INFO 07-12 16:37:29 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2736308) INFO 07-12 16:37:29 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.17 GiB total
(Worker_TP1 pid=2736309) INFO 07-12 16:37:29 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.17 GiB total
(Worker_TP0 pid=2736308) INFO 07-12 16:37:30 [gpu_worker.py:508] Available KV cache memory: 57.27 GiB
(Worker_TP0 pid=2736308) INFO 07-12 16:37:30 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9182 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9218. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP1 pid=2736309) INFO 07-12 16:37:30 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9182 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9218. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2736153) INFO 07-12 16:37:30 [kv_cache_utils.py:2146] GPU KV cache size: 1,251,040 tokens
(EngineCore pid=2736153) INFO 07-12 16:37:30 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 30.54x
(Worker_TP0 pid=2736308) INFO 07-12 16:37:30 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2736308) INFO 07-12 16:37:30 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP1 pid=2736309) 2026-07-12 16:37:30,378 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2736308) 2026-07-12 16:37:30,379 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2736308) 2026-07-12 16:37:30,438 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=2736309) 2026-07-12 16:37:30,440 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2736308)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 5%|▌ | 1/19 [00:00<00:02, 8.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:02, 8.32it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 3/19 [00:00<00:01, 8.27it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 21%|██ | 4/19 [00:00<00:01, 8.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:00<00:01, 8.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 32%|███▏ | 6/19 [00:00<00:01, 8.18it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 7/19 [00:00<00:01, 8.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:00<00:01, 8.22it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 9/19 [00:01<00:01, 8.28it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 10/19 [00:01<00:01, 8.20it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:01<00:00, 8.15it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:01<00:00, 7.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 68%|██████▊ | 13/19 [00:01<00:00, 7.48it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:01<00:00, 7.00it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 79%|███████▉ | 15/19 [00:01<00:00, 7.25it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 16/19 [00:02<00:00, 7.43it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:02<00:00, 6.97it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:02<00:00, 6.65it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:02<00:00, 6.40it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:02<00:00, 7.44it/s]
(Worker_TP0 pid=2736308)
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 9%|▉ | 1/11 [00:00<00:01, 8.47it/s]
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:00<00:01, 8.98it/s]
Capturing CUDA graphs (decode, FULL): 27%|██▋ | 3/11 [00:00<00:00, 8.68it/s]
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:00<00:00, 9.00it/s]
Capturing CUDA graphs (decode, FULL): 45%|████▌ | 5/11 [00:00<00:00, 8.94it/s]
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:00<00:00, 8.79it/s]
Capturing CUDA graphs (decode, FULL): 64%|██████▎ | 7/11 [00:00<00:00, 8.65it/s]
Capturing CUDA graphs (decode, FULL): 73%|███████▎ | 8/11 [00:00<00:00, 8.62it/s]
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 9/11 [00:01<00:00, 8.69it/s]
Capturing CUDA graphs (decode, FULL): 91%|█████████ | 10/11 [00:01<00:00, 8.98it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:01<00:00, 9.16it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:01<00:00, 8.89it/s]
(Worker_TP0 pid=2736308) INFO 07-12 16:37:34 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2736309) INFO 07-12 16:37:34 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2736309) INFO 07-12 16:37:34 [gpu_worker.py:667] CUDA graph pool memory: 0.2 GiB (actual), 0.17 GiB (estimated), difference: 0.03 GiB (14.9%).
(Worker_TP0 pid=2736308) INFO 07-12 16:37:34 [gpu_model_runner.py:6660] Graph capturing finished in 5 secs, took 0.20 GiB
(Worker_TP0 pid=2736308) INFO 07-12 16:37:34 [gpu_worker.py:667] CUDA graph pool memory: 0.2 GiB (actual), 0.17 GiB (estimated), difference: 0.03 GiB (14.9%).
(Worker_TP1 pid=2736309) INFO 07-12 16:37:34 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP0 pid=2736308) INFO 07-12 16:37:35 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2736153) INFO 07-12 16:37:35 [core.py:337] init engine (profile, create kv cache, warmup model) took 15.61 s (compilation: 7.28 s)
(EngineCore pid=2736153) INFO 07-12 16:37:36 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp2_mns64/opprof/opprof-v1-dp0-pid2736153-1783874256706634444.jsonl
(EngineCore pid=2736153) INFO 07-12 16:37:36 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2736153) INFO 07-12 16:37:36 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2736153) INFO 07-12 16:37:37 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2735795) INFO 07-12 16:37:37 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2735795) WARNING 07-12 16:37:38 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2735795) INFO 07-12 16:37:38 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2735795) INFO 07-12 16:37:38 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8703
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:37] Available routes are:
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2735795) INFO 07-12 16:37:38 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2735795) INFO: Started server process [2735795]
(APIServer pid=2735795) INFO: Waiting for application startup.
(APIServer pid=2735795) INFO: Application startup complete.
(APIServer pid=2735795) INFO: 127.0.0.1:41546 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2736308) WARNING 07-12 16:37:44 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2735795) INFO: 127.0.0.1:38264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2736308) WARNING 07-12 16:37:44 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2735795) INFO: 127.0.0.1:38292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:37:48 [loggers.py:273] Engine 000: Avg prompt throughput: 3796.9 tokens/s, Avg generation throughput: 173.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 6.5%
(APIServer pid=2735795) INFO: 127.0.0.1:59916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:59930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:59932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:59936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:59948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:59962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:59974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:59976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:59984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:59992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:37:58 [loggers.py:273] Engine 000: Avg prompt throughput: 3189.9 tokens/s, Avg generation throughput: 189.9 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 38.4%
(APIServer pid=2735795) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:38:08 [loggers.py:273] Engine 000: Avg prompt throughput: 18372.2 tokens/s, Avg generation throughput: 586.2 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.2%, Prefix cache hit rate: 20.6%
(APIServer pid=2735795) INFO: 127.0.0.1:57538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:57224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:38:18 [loggers.py:273] Engine 000: Avg prompt throughput: 12865.3 tokens/s, Avg generation throughput: 913.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 16.4%
(APIServer pid=2735795) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:38:28 [loggers.py:273] Engine 000: Avg prompt throughput: 33.9 tokens/s, Avg generation throughput: 373.8 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 32.1%
(APIServer pid=2735795) INFO: 127.0.0.1:38440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:38:38 [loggers.py:273] Engine 000: Avg prompt throughput: 57.0 tokens/s, Avg generation throughput: 864.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 49.6%
(APIServer pid=2735795) INFO: 127.0.0.1:47366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:38:48 [loggers.py:273] Engine 000: Avg prompt throughput: 7912.1 tokens/s, Avg generation throughput: 701.3 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 51.7%
(APIServer pid=2735795) INFO: 127.0.0.1:50626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:38:58 [loggers.py:273] Engine 000: Avg prompt throughput: 15999.5 tokens/s, Avg generation throughput: 877.4 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 45.7%
(APIServer pid=2735795) INFO: 127.0.0.1:38122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:39:08 [loggers.py:273] Engine 000: Avg prompt throughput: 17268.9 tokens/s, Avg generation throughput: 699.1 tokens/s, Running: 34 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 40.8%
(APIServer pid=2735795) INFO: 127.0.0.1:49696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:49806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:39:18 [loggers.py:273] Engine 000: Avg prompt throughput: 19084.0 tokens/s, Avg generation throughput: 683.3 tokens/s, Running: 41 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.6%, Prefix cache hit rate: 36.4%
(APIServer pid=2735795) INFO: 127.0.0.1:43804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:39:28 [loggers.py:273] Engine 000: Avg prompt throughput: 13700.7 tokens/s, Avg generation throughput: 805.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 34.0%
(APIServer pid=2735795) INFO: 127.0.0.1:54724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:39:38 [loggers.py:273] Engine 000: Avg prompt throughput: 27.9 tokens/s, Avg generation throughput: 313.9 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 37.0%
(APIServer pid=2735795) INFO: 127.0.0.1:54944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:54990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:55000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:55008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:55016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:55032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:39:48 [loggers.py:273] Engine 000: Avg prompt throughput: 45.7 tokens/s, Avg generation throughput: 682.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 42.1%
(APIServer pid=2735795) INFO: 127.0.0.1:56422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:39:58 [loggers.py:273] Engine 000: Avg prompt throughput: 39.6 tokens/s, Avg generation throughput: 527.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 45.5%
(APIServer pid=2735795) INFO: 127.0.0.1:44428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:44460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:32994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:40:08 [loggers.py:273] Engine 000: Avg prompt throughput: 36.8 tokens/s, Avg generation throughput: 575.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 48.8%
(APIServer pid=2735795) INFO: 127.0.0.1:33220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:33240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:40:18 [loggers.py:273] Engine 000: Avg prompt throughput: 39.4 tokens/s, Avg generation throughput: 639.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 52.3%
(APIServer pid=2735795) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:37578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:40:28 [loggers.py:273] Engine 000: Avg prompt throughput: 35.3 tokens/s, Avg generation throughput: 543.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 55.1%
(APIServer pid=2735795) INFO: 127.0.0.1:51134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:51194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:40:38 [loggers.py:273] Engine 000: Avg prompt throughput: 12.6 tokens/s, Avg generation throughput: 251.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 56.3%
(APIServer pid=2735795) INFO: 127.0.0.1:39742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:40:48 [loggers.py:273] Engine 000: Avg prompt throughput: 39.7 tokens/s, Avg generation throughput: 493.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 58.2%
(APIServer pid=2735795) INFO: 127.0.0.1:39084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:38990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:40:58 [loggers.py:273] Engine 000: Avg prompt throughput: 46.6 tokens/s, Avg generation throughput: 664.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 60.4%
(APIServer pid=2735795) INFO: 127.0.0.1:39016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:39048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:41:08 [loggers.py:273] Engine 000: Avg prompt throughput: 39.6 tokens/s, Avg generation throughput: 486.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 62.0%
(APIServer pid=2735795) INFO: 127.0.0.1:56888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:56966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:41:18 [loggers.py:273] Engine 000: Avg prompt throughput: 29.8 tokens/s, Avg generation throughput: 561.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 63.7%
(APIServer pid=2735795) INFO: 127.0.0.1:40952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:40996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:41022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:41:28 [loggers.py:273] Engine 000: Avg prompt throughput: 44.8 tokens/s, Avg generation throughput: 675.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 66.6%
(APIServer pid=2735795) INFO: 127.0.0.1:43536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:43596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:41:38 [loggers.py:273] Engine 000: Avg prompt throughput: 33.6 tokens/s, Avg generation throughput: 531.7 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 70.6%
(APIServer pid=2735795) INFO: 127.0.0.1:46642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO: 127.0.0.1:46686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2735795) INFO 07-12 16:41:41 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2735795) INFO 07-12 16:41:41 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2735795) INFO 07-12 16:41:41 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2735795) INFO 07-12 16:41:41 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2736153) INFO 07-12 16:41:41 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2736153) INFO 07-12 16:41:41 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2736153) INFO 07-12 16:41:41 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2736153) INFO 07-12 16:41:41 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(Worker_TP0 pid=2736308) INFO 07-12 16:41:41 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(EngineCore pid=2736153) INFO 07-12 16:41:41 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2
(APIServer pid=2735795) INFO: Shutting down
(APIServer pid=2735795) INFO: Waiting for application shutdown.
(APIServer pid=2735795) INFO: Application shutdown complete.
(APIServer pid=2735795) INFO: Finished server process [2735795]
(EngineCore pid=2736153) INFO 07-12 16:41:44 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2735795) INFO 07-12 16:41:45 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2735795) INFO 07-12 16:41:45 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2735795) INFO 07-12 16:41:45 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink