gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a730b368d6ec1760f165e502bc6cbb6fd0d3ccf4
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp2_mns8/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

1087 lines
115 KiB
Plaintext
Raw Blame History

(APIServer pid=2749973) INFO 07-12 16:50:20 [api_utils.py:339]
(APIServer pid=2749973) INFO 07-12 16:50:20 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2749973) INFO 07-12 16:50:20 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2749973) INFO 07-12 16:50:20 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2749973) INFO 07-12 16:50:20 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2749973) INFO 07-12 16:50:20 [api_utils.py:339]
(APIServer pid=2749973) INFO 07-12 16:50:20 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8705, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 2, 'max_num_batched_tokens': 8192, 'max_num_seqs': 8, 'shutdown_timeout': 120}
(APIServer pid=2749973) INFO 07-12 16:50:20 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2749973) INFO 07-12 16:50:20 [model.py:1725] Using max model len 40960
(APIServer pid=2749973) INFO 07-12 16:50:20 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2749973) INFO 07-12 16:50:20 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2749973) INFO 07-12 16:50:20 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2749973) INFO 07-12 16:50:22 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2750298) INFO 07-12 16:50:31 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2750298) WARNING 07-12 16:50:31 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2750298) INFO 07-12 16:50:31 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2
(Worker pid=2750550) INFO 07-12 16:50:41 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:45665 backend=nccl
(Worker pid=2750551) INFO 07-12 16:50:41 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:45665 backend=nccl
(Worker pid=2750550) INFO 07-12 16:50:42 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2750550) INFO 07-12 16:50:43 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2750550) INFO 07-12 16:50:44 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2750550) INFO 07-12 16:50:44 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2750550) INFO 07-12 16:50:44 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2750550) INFO 07-12 16:50:44 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2750550) INFO 07-12 16:50:45 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2750550) INFO 07-12 16:50:45 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2750550) INFO 07-12 16:50:45 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2750550) INFO 07-12 16:50:45 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1286.53 GiB.
(Worker_TP0 pid=2750550) INFO 07-12 16:50:45 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:08, 1.69it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:08, 1.56it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:01<00:08, 1.51it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:02<00:08, 1.47it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:03<00:07, 1.45it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:04<00:07, 1.43it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:04<00:06, 1.41it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:05<00:05, 1.42it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:06<00:04, 1.45it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:06<00:04, 1.48it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:07<00:03, 1.49it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:08<00:02, 1.50it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:08<00:02, 1.50it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:09<00:01, 1.50it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:10<00:00, 1.51it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.95it/s]
(Worker_TP0 pid=2750550)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.56it/s]
(Worker_TP0 pid=2750550)
(Worker_TP0 pid=2750550) INFO 07-12 16:50:55 [default_loader.py:430] Loading weights took 10.29 seconds
(Worker_TP0 pid=2750550) INFO 07-12 16:50:55 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2750550) INFO 07-12 16:50:56 [gpu_model_runner.py:5259] Model loading took 28.46 GiB memory and 10.716717 seconds
(Worker_TP0 pid=2750550) INFO 07-12 16:51:00 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/0c37cd99d7/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2750550) INFO 07-12 16:51:00 [backends.py:1148] Dynamo bytecode transform time: 3.36 s
(Worker_TP0 pid=2750550) INFO 07-12 16:51:00 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2750550) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2750550) return func(*args, **kwargs)
[rank0]:[W712 16:51:00.424757606 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2750550) INFO 07-12 16:51:00 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP1 pid=2750551) INFO 07-12 16:51:04 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/7db77e4784ad115c73ccf5b7472a39190ea4c85a91f415675a26d40f2c108d60/rank_1_0/model
(Worker_TP0 pid=2750550) INFO 07-12 16:51:04 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 2.769 s
(Worker_TP0 pid=2750550) INFO 07-12 16:51:04 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/7db77e4784ad115c73ccf5b7472a39190ea4c85a91f415675a26d40f2c108d60/rank_0_0/model
(Worker_TP0 pid=2750550) INFO 07-12 16:51:04 [monitor.py:53] torch.compile took 7.60 s in total
(Worker_TP0 pid=2750550) INFO 07-12 16:51:04 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2750550) INFO 07-12 16:51:04 [monitor.py:81] Initial profiling/warmup run took 0.25 s
(Worker_TP0 pid=2750550) INFO 07-12 16:51:11 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(Worker_TP1 pid=2750551) INFO 07-12 16:51:11 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(Worker_TP0 pid=2750550) INFO 07-12 16:51:12 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2750551) INFO 07-12 16:51:12 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2750550) INFO 07-12 16:51:12 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.07 GiB total
(Worker_TP1 pid=2750551) INFO 07-12 16:51:12 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.07 GiB total
(Worker_TP0 pid=2750550) INFO 07-12 16:51:13 [gpu_worker.py:508] Available KV cache memory: 57.4 GiB
(Worker_TP0 pid=2750550) INFO 07-12 16:51:13 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9193 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9207. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP1 pid=2750551) INFO 07-12 16:51:13 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9193 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9207. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2750298) INFO 07-12 16:51:13 [kv_cache_utils.py:2146] GPU KV cache size: 1,253,824 tokens
(EngineCore pid=2750298) INFO 07-12 16:51:13 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 30.61x
(Worker_TP0 pid=2750550) INFO 07-12 16:51:13 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2750550) INFO 07-12 16:51:13 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP1 pid=2750551) 2026-07-12 16:51:13,410 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2750550) 2026-07-12 16:51:13,411 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2750550) 2026-07-12 16:51:13,463 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=2750551) 2026-07-12 16:51:13,470 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2750550)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/5 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 20%|██ | 1/5 [00:00<00:00, 6.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 40%|████ | 2/5 [00:00<00:00, 7.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 60%|██████ | 3/5 [00:00<00:00, 8.12it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 80%|████████ | 4/5 [00:00<00:00, 7.52it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:00<00:00, 7.13it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:00<00:00, 7.33it/s]
(Worker_TP0 pid=2750550)
Capturing CUDA graphs (decode, FULL): 0%| | 0/4 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 25%|██▌ | 1/4 [00:00<00:00, 8.92it/s]
Capturing CUDA graphs (decode, FULL): 50%|█████ | 2/4 [00:00<00:00, 9.42it/s]
Capturing CUDA graphs (decode, FULL): 75%|███████▌ | 3/4 [00:00<00:00, 9.46it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 4/4 [00:00<00:00, 9.60it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 4/4 [00:00<00:00, 9.49it/s]
(Worker_TP0 pid=2750550) INFO 07-12 16:51:14 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2750551) INFO 07-12 16:51:14 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2750550) INFO 07-12 16:51:15 [gpu_model_runner.py:6660] Graph capturing finished in 2 secs, took 0.08 GiB
(Worker_TP0 pid=2750550) INFO 07-12 16:51:15 [gpu_worker.py:667] CUDA graph pool memory: 0.08 GiB (actual), 0.07 GiB (estimated), difference: 0.01 GiB (14.3%).
(Worker_TP0 pid=2750550) INFO 07-12 16:51:15 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP1 pid=2750551) INFO 07-12 16:51:15 [gpu_worker.py:667] CUDA graph pool memory: 0.08 GiB (actual), 0.07 GiB (estimated), difference: 0.01 GiB (14.3%).
(Worker_TP1 pid=2750551) INFO 07-12 16:51:15 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2750298) INFO 07-12 16:51:15 [core.py:337] init engine (profile, create kv cache, warmup model) took 19.13 s (compilation: 7.60 s)
(EngineCore pid=2750298) INFO 07-12 16:51:16 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp2_mns8/opprof/opprof-v1-dp0-pid2750298-1783875076865595283.jsonl
(EngineCore pid=2750298) INFO 07-12 16:51:17 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2750298) INFO 07-12 16:51:17 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2750298) INFO 07-12 16:51:17 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2749973) INFO 07-12 16:51:17 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2749973) WARNING 07-12 16:51:18 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2749973) INFO 07-12 16:51:18 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2749973) INFO 07-12 16:51:18 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8705
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:37] Available routes are:
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2749973) INFO 07-12 16:51:18 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2749973) INFO: Started server process [2749973]
(APIServer pid=2749973) INFO: Waiting for application startup.
(APIServer pid=2749973) INFO: Application startup complete.
(APIServer pid=2749973) INFO: 127.0.0.1:47654 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2750550) WARNING 07-12 16:51:25 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2749973) INFO: 127.0.0.1:46842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2750550) WARNING 07-12 16:51:25 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2749973) INFO: 127.0.0.1:46870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:46994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:51:28 [loggers.py:273] Engine 000: Avg prompt throughput: 2859.3 tokens/s, Avg generation throughput: 103.2 tokens/s, Running: 8 reqs, Waiting: 2 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 5.2%
(APIServer pid=2749973) INFO: 127.0.0.1:52088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:51:38 [loggers.py:273] Engine 000: Avg prompt throughput: 1831.9 tokens/s, Avg generation throughput: 211.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 39.9%
(APIServer pid=2749973) INFO: 127.0.0.1:52184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:51:48 [loggers.py:273] Engine 000: Avg prompt throughput: 9089.8 tokens/s, Avg generation throughput: 470.4 tokens/s, Running: 8 reqs, Waiting: 10 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 31.2%
(APIServer pid=2749973) INFO: 127.0.0.1:49918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:49956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:51:58 [loggers.py:273] Engine 000: Avg prompt throughput: 11205.4 tokens/s, Avg generation throughput: 575.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 22.9%
(APIServer pid=2749973) INFO: 127.0.0.1:59728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:52:08 [loggers.py:273] Engine 000: Avg prompt throughput: 14.7 tokens/s, Avg generation throughput: 200.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 30.5%
(APIServer pid=2749973) INFO: 127.0.0.1:59818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:52:18 [loggers.py:273] Engine 000: Avg prompt throughput: 46.0 tokens/s, Avg generation throughput: 627.0 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 50.6%
(APIServer pid=2749973) INFO: 127.0.0.1:33052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:52:28 [loggers.py:273] Engine 000: Avg prompt throughput: 2482.9 tokens/s, Avg generation throughput: 549.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 55.8%
(APIServer pid=2749973) INFO: 127.0.0.1:37218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:37272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:52:38 [loggers.py:273] Engine 000: Avg prompt throughput: 12246.8 tokens/s, Avg generation throughput: 482.2 tokens/s, Running: 8 reqs, Waiting: 6 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 47.7%
(APIServer pid=2749973) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:34980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:34990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:34994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:52:48 [loggers.py:273] Engine 000: Avg prompt throughput: 12517.4 tokens/s, Avg generation throughput: 554.9 tokens/s, Running: 8 reqs, Waiting: 8 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 41.9%
(APIServer pid=2749973) INFO: 127.0.0.1:35370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:44998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:52:58 [loggers.py:273] Engine 000: Avg prompt throughput: 11552.3 tokens/s, Avg generation throughput: 587.3 tokens/s, Running: 8 reqs, Waiting: 7 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 38.0%
(APIServer pid=2749973) INFO: 127.0.0.1:45114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:53:08 [loggers.py:273] Engine 000: Avg prompt throughput: 8720.7 tokens/s, Avg generation throughput: 356.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 35.6%
(APIServer pid=2749973) INFO: 127.0.0.1:38894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:53:18 [loggers.py:273] Engine 000: Avg prompt throughput: 32.3 tokens/s, Avg generation throughput: 356.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 41.0%
(APIServer pid=2749973) INFO: 127.0.0.1:39142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:39200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:53:28 [loggers.py:273] Engine 000: Avg prompt throughput: 45.1 tokens/s, Avg generation throughput: 678.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 46.9%
(APIServer pid=2749973) INFO: 127.0.0.1:45874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:45910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:53:38 [loggers.py:273] Engine 000: Avg prompt throughput: 37.3 tokens/s, Avg generation throughput: 503.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 51.2%
(APIServer pid=2749973) INFO: 127.0.0.1:38906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:38934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:53:48 [loggers.py:273] Engine 000: Avg prompt throughput: 33.4 tokens/s, Avg generation throughput: 561.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 55.2%
(APIServer pid=2749973) INFO: 127.0.0.1:32810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:32830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:35986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:53:58 [loggers.py:273] Engine 000: Avg prompt throughput: 38.9 tokens/s, Avg generation throughput: 630.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 59.0%
(APIServer pid=2749973) INFO: 127.0.0.1:36348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:36400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:54:08 [loggers.py:273] Engine 000: Avg prompt throughput: 32.5 tokens/s, Avg generation throughput: 488.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 61.8%
(APIServer pid=2749973) INFO: 127.0.0.1:55402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:55468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:54:18 [loggers.py:273] Engine 000: Avg prompt throughput: 5433.0 tokens/s, Avg generation throughput: 224.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 60.3%
(APIServer pid=2749973) INFO: 127.0.0.1:47322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:54:28 [loggers.py:273] Engine 000: Avg prompt throughput: 41.1 tokens/s, Avg generation throughput: 497.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 62.5%
(APIServer pid=2749973) INFO: 127.0.0.1:33598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:33658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:52996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:54:38 [loggers.py:273] Engine 000: Avg prompt throughput: 44.0 tokens/s, Avg generation throughput: 655.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 64.9%
(APIServer pid=2749973) INFO: 127.0.0.1:53300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:53326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:59996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:54:48 [loggers.py:273] Engine 000: Avg prompt throughput: 39.7 tokens/s, Avg generation throughput: 485.3 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 66.8%
(APIServer pid=2749973) INFO: 127.0.0.1:60088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:54:58 [loggers.py:273] Engine 000: Avg prompt throughput: 28.2 tokens/s, Avg generation throughput: 534.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 68.5%
(APIServer pid=2749973) INFO: 127.0.0.1:60842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:60902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:55:08 [loggers.py:273] Engine 000: Avg prompt throughput: 41.5 tokens/s, Avg generation throughput: 662.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 70.2%
(APIServer pid=2749973) INFO: 127.0.0.1:40720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:40770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:55:18 [loggers.py:273] Engine 000: Avg prompt throughput: 33.0 tokens/s, Avg generation throughput: 496.7 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 71.9%
(APIServer pid=2749973) INFO: 127.0.0.1:47320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO: 127.0.0.1:47352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2749973) INFO 07-12 16:55:21 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2749973) INFO 07-12 16:55:21 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2749973) INFO 07-12 16:55:21 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2749973) INFO 07-12 16:55:21 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2750298) INFO 07-12 16:55:21 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2750298) INFO 07-12 16:55:21 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2750298) INFO 07-12 16:55:21 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2750298) INFO 07-12 16:55:21 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(Worker_TP0 pid=2750550) INFO 07-12 16:55:21 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(EngineCore pid=2750298) INFO 07-12 16:55:21 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2
(APIServer pid=2749973) INFO: Shutting down
(APIServer pid=2749973) INFO: Waiting for application shutdown.
(APIServer pid=2749973) INFO: Application shutdown complete.
(APIServer pid=2749973) INFO: Finished server process [2749973]
(EngineCore pid=2750298) INFO 07-12 16:55:24 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2749973) INFO 07-12 16:55:26 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2749973) INFO 07-12 16:55:26 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2749973) INFO 07-12 16:55:26 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink