gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
34e1f4c144641c2b2fcc308a2b776de09f747a24
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp2_mns16/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

1572 lines
162 KiB
Plaintext
Raw Blame History

(APIServer pid=2755321) INFO 07-12 16:55:39 [api_utils.py:339]
(APIServer pid=2755321) INFO 07-12 16:55:39 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2755321) INFO 07-12 16:55:39 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2755321) INFO 07-12 16:55:39 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2755321) INFO 07-12 16:55:39 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2755321) INFO 07-12 16:55:39 [api_utils.py:339]
(APIServer pid=2755321) INFO 07-12 16:55:39 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8706, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'tensor_parallel_size': 2, 'max_num_batched_tokens': 8192, 'max_num_seqs': 16, 'shutdown_timeout': 120}
(APIServer pid=2755321) INFO 07-12 16:55:39 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2755321) INFO 07-12 16:55:39 [model.py:1725] Using max model len 40960
(APIServer pid=2755321) INFO 07-12 16:55:39 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2755321) INFO 07-12 16:55:39 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2755321) INFO 07-12 16:55:39 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2755321) INFO 07-12 16:55:41 [compilation.py:310] Enabled custom fusions: allreduce_rms
(EngineCore pid=2755628) INFO 07-12 16:55:50 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2755628) WARNING 07-12 16:55:50 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=2755628) INFO 07-12 16:55:50 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2
(Worker pid=2755779) INFO 07-12 16:56:00 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:41575 backend=nccl
(Worker pid=2755780) INFO 07-12 16:56:00 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:41575 backend=nccl
(Worker pid=2755779) INFO 07-12 16:56:01 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=2755779) INFO 07-12 16:56:02 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2755779) INFO 07-12 16:56:03 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=2755779) INFO 07-12 16:56:03 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=2755779) INFO 07-12 16:56:03 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=2755779) INFO 07-12 16:56:03 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(Worker_TP0 pid=2755779) INFO 07-12 16:56:04 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=2755779) INFO 07-12 16:56:04 [flash_attn.py:670] Using FlashAttention version 3
(Worker_TP0 pid=2755779) INFO 07-12 16:56:04 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(Worker_TP0 pid=2755779) INFO 07-12 16:56:04 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1286.52 GiB.
(Worker_TP0 pid=2755779) INFO 07-12 16:56:04 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:09, 1.59it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:09, 1.48it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:02<00:08, 1.48it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:02<00:08, 1.46it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:03<00:07, 1.48it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:04<00:07, 1.29it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:04<00:06, 1.36it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:05<00:05, 1.43it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:06<00:04, 1.47it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:06<00:03, 1.50it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:07<00:03, 1.53it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:08<00:02, 1.54it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:08<00:01, 1.54it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:09<00:01, 1.53it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:10<00:00, 1.53it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.98it/s]
(Worker_TP0 pid=2755779)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:10<00:00, 1.56it/s]
(Worker_TP0 pid=2755779)
(Worker_TP0 pid=2755779) INFO 07-12 16:56:15 [default_loader.py:430] Loading weights took 10.29 seconds
(Worker_TP0 pid=2755779) INFO 07-12 16:56:15 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=2755779) INFO 07-12 16:56:15 [gpu_model_runner.py:5259] Model loading took 28.46 GiB memory and 10.780291 seconds
(Worker_TP0 pid=2755779) INFO 07-12 16:56:19 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/71caca9a09/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=2755779) INFO 07-12 16:56:19 [backends.py:1148] Dynamo bytecode transform time: 3.20 s
(Worker_TP0 pid=2755779) INFO 07-12 16:56:19 [flashinfer_all_reduce.py:112] Auto-selected flashinfer allreduce backend: trtllm
(Worker_TP0 pid=2755779) /tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
(Worker_TP0 pid=2755779) return func(*args, **kwargs)
[rank0]:[W712 16:56:19.510648819 ProcessGroupNCCL.cpp:5188] Guessing device ID based on global rank. This can cause a hang if rank to GPU mapping is heterogeneous. You can specify device_id in init_process_group()
(Worker_TP0 pid=2755779) INFO 07-12 16:56:19 [flashinfer_all_reduce.py:152] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
(Worker_TP1 pid=2755780) INFO 07-12 16:56:23 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/b1870a24d902681201272cd6a2355f94635aa2439d53d1b45755c3823de597b6/rank_1_0/model
(Worker_TP0 pid=2755779) INFO 07-12 16:56:23 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 2.410 s
(Worker_TP0 pid=2755779) INFO 07-12 16:56:23 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/b1870a24d902681201272cd6a2355f94635aa2439d53d1b45755c3823de597b6/rank_0_0/model
(Worker_TP0 pid=2755779) INFO 07-12 16:56:23 [monitor.py:53] torch.compile took 7.09 s in total
(Worker_TP0 pid=2755779) INFO 07-12 16:56:23 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=384,device_name=NVIDIA_H20.json for MoE layer.
(Worker_TP0 pid=2755779) INFO 07-12 16:56:23 [monitor.py:81] Initial profiling/warmup run took 0.25 s
(Worker_TP1 pid=2755780) INFO 07-12 16:56:23 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP0 pid=2755779) INFO 07-12 16:56:24 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(Worker_TP1 pid=2755780) INFO 07-12 16:56:24 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2755779) INFO 07-12 16:56:24 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP1 pid=2755780) INFO 07-12 16:56:25 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.08 GiB total
(Worker_TP0 pid=2755779) INFO 07-12 16:56:25 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.08 GiB total
(Worker_TP1 pid=2755780) INFO 07-12 16:56:25 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9191 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9209. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(Worker_TP0 pid=2755779) INFO 07-12 16:56:25 [gpu_worker.py:508] Available KV cache memory: 57.38 GiB
(Worker_TP0 pid=2755779) INFO 07-12 16:56:25 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9191 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9209. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2755628) INFO 07-12 16:56:25 [kv_cache_utils.py:2146] GPU KV cache size: 1,253,520 tokens
(EngineCore pid=2755628) INFO 07-12 16:56:25 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 30.60x
(Worker_TP0 pid=2755779) INFO 07-12 16:56:26 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=2755779) INFO 07-12 16:56:26 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP1 pid=2755780) 2026-07-12 16:56:26,088 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2755779) 2026-07-12 16:56:26,089 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0 pid=2755779) 2026-07-12 16:56:26,145 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1 pid=2755780) 2026-07-12 16:56:26,145 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0 pid=2755779)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 14%|█▍ | 1/7 [00:00<00:00, 9.38it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 29%|██▊ | 2/7 [00:00<00:00, 9.45it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 43%|████▎ | 3/7 [00:00<00:00, 9.47it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 57%|█████▋ | 4/7 [00:00<00:00, 9.45it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 71%|███████▏ | 5/7 [00:00<00:00, 7.96it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 86%|████████▌ | 6/7 [00:00<00:00, 7.37it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:00<00:00, 7.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:00<00:00, 7.83it/s]
(Worker_TP0 pid=2755779)
Capturing CUDA graphs (decode, FULL): 0%| | 0/5 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 40%|████ | 2/5 [00:00<00:00, 10.11it/s]
Capturing CUDA graphs (decode, FULL): 80%|████████ | 4/5 [00:00<00:00, 10.30it/s](Worker_TP1 pid=2755780) INFO 07-12 16:56:27 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
Capturing CUDA graphs (decode, FULL): 100%|██████████| 5/5 [00:00<00:00, 10.24it/s]
(Worker_TP0 pid=2755779) INFO 07-12 16:56:27 [custom_all_reduce.py:213] Registering 0 cuda graph addresses
(Worker_TP0 pid=2755779) (Worker_TP1 pid=2755780) INFO 07-12 16:56:28 [gpu_model_runner.py:6660] Graph capturing finished in 2 secs, took 0.10 GiB
INFO 07-12 16:56:28 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.08 GiB (estimated), difference: 0.01 GiB (14.0%).
(Worker_TP0 pid=2755779) INFO 07-12 16:56:28 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.08 GiB (estimated), difference: 0.01 GiB (14.0%).
(Worker_TP0 pid=2755779) (Worker_TP1 pid=2755780) INFO 07-12 16:56:28 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
INFO 07-12 16:56:28 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2755628) INFO 07-12 16:56:28 [core.py:337] init engine (profile, create kv cache, warmup model) took 12.85 s (compilation: 7.15 s)
(EngineCore pid=2755628) INFO 07-12 16:56:29 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp2_mns16/opprof/opprof-v1-dp0-pid2755628-1783875389707263062.jsonl
(EngineCore pid=2755628) INFO 07-12 16:56:29 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2755628) INFO 07-12 16:56:29 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2755628) INFO 07-12 16:56:30 [compilation.py:310] Enabled custom fusions: allreduce_rms
(APIServer pid=2755321) INFO 07-12 16:56:30 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2755321) WARNING 07-12 16:56:30 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2755321) INFO 07-12 16:56:31 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2755321) INFO 07-12 16:56:31 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8706
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:37] Available routes are:
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2755321) INFO 07-12 16:56:31 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2755321) INFO: Started server process [2755321]
(APIServer pid=2755321) INFO: Waiting for application startup.
(APIServer pid=2755321) INFO: Application startup complete.
(APIServer pid=2755321) INFO: 127.0.0.1:35476 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2755779) WARNING 07-12 16:56:38 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2755321) INFO: 127.0.0.1:35496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(Worker_TP0 pid=2755779) WARNING 07-12 16:56:38 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2755321) INFO: 127.0.0.1:35518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:56:41 [loggers.py:273] Engine 000: Avg prompt throughput: 3489.5 tokens/s, Avg generation throughput: 122.1 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 4.7%
(APIServer pid=2755321) INFO: 127.0.0.1:37080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:56:51 [loggers.py:273] Engine 000: Avg prompt throughput: 1107.4 tokens/s, Avg generation throughput: 183.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 39.9%
(APIServer pid=2755321) INFO: 127.0.0.1:44176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:57:01 [loggers.py:273] Engine 000: Avg prompt throughput: 10281.3 tokens/s, Avg generation throughput: 513.3 tokens/s, Running: 16 reqs, Waiting: 1 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 29.3%
(APIServer pid=2755321) INFO: 127.0.0.1:49802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:49998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:57:11 [loggers.py:273] Engine 000: Avg prompt throughput: 12469.0 tokens/s, Avg generation throughput: 621.3 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 22.0%
(APIServer pid=2755321) INFO: 127.0.0.1:35194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:57:21 [loggers.py:273] Engine 000: Avg prompt throughput: 12256.5 tokens/s, Avg generation throughput: 528.0 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 18.6%
(APIServer pid=2755321) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:57:31 [loggers.py:273] Engine 000: Avg prompt throughput: 12509.7 tokens/s, Avg generation throughput: 541.7 tokens/s, Running: 16 reqs, Waiting: 3 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 16.5%
(APIServer pid=2755321) INFO: 127.0.0.1:45832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:57:41 [loggers.py:273] Engine 000: Avg prompt throughput: 13587.1 tokens/s, Avg generation throughput: 648.2 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 15.9%
(APIServer pid=2755321) INFO: 127.0.0.1:38020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:38324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:57:51 [loggers.py:273] Engine 000: Avg prompt throughput: 12116.6 tokens/s, Avg generation throughput: 519.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 14.5%
(APIServer pid=2755321) INFO: 127.0.0.1:48276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:48382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:58:01 [loggers.py:273] Engine 000: Avg prompt throughput: 18.9 tokens/s, Avg generation throughput: 226.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 19.7%
(APIServer pid=2755321) INFO: 127.0.0.1:34994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:58:11 [loggers.py:273] Engine 000: Avg prompt throughput: 47.3 tokens/s, Avg generation throughput: 660.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 30.2%
(APIServer pid=2755321) INFO: 127.0.0.1:32822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:56900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:56906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:56918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:56930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:56944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:56952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:58:21 [loggers.py:273] Engine 000: Avg prompt throughput: 39.2 tokens/s, Avg generation throughput: 521.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 37.3%
(APIServer pid=2755321) INFO: 127.0.0.1:56966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:56978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:56988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:56992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:57288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:58:31 [loggers.py:273] Engine 000: Avg prompt throughput: 37.2 tokens/s, Avg generation throughput: 579.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 43.6%
(APIServer pid=2755321) INFO: 127.0.0.1:55496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:55902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:58:41 [loggers.py:273] Engine 000: Avg prompt throughput: 34.7 tokens/s, Avg generation throughput: 600.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 48.8%
(APIServer pid=2755321) INFO: 127.0.0.1:42890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:58:51 [loggers.py:273] Engine 000: Avg prompt throughput: 38.7 tokens/s, Avg generation throughput: 575.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 53.3%
(APIServer pid=2755321) INFO: 127.0.0.1:58882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:59:01 [loggers.py:273] Engine 000: Avg prompt throughput: 18.6 tokens/s, Avg generation throughput: 330.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 55.8%
(APIServer pid=2755321) INFO: 127.0.0.1:32966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:32992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:59:11 [loggers.py:273] Engine 000: Avg prompt throughput: 35.6 tokens/s, Avg generation throughput: 413.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 58.4%
(APIServer pid=2755321) INFO: 127.0.0.1:59978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:59992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:60388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:59:21 [loggers.py:273] Engine 000: Avg prompt throughput: 45.0 tokens/s, Avg generation throughput: 678.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 61.1%
(APIServer pid=2755321) INFO: 127.0.0.1:42358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:42602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:59:31 [loggers.py:273] Engine 000: Avg prompt throughput: 38.1 tokens/s, Avg generation throughput: 486.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 63.4%
(APIServer pid=2755321) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:59:41 [loggers.py:273] Engine 000: Avg prompt throughput: 29.5 tokens/s, Avg generation throughput: 540.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 65.5%
(APIServer pid=2755321) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:58794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 16:59:51 [loggers.py:273] Engine 000: Avg prompt throughput: 40.3 tokens/s, Avg generation throughput: 644.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 67.6%
(APIServer pid=2755321) INFO: 127.0.0.1:45494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:00:01 [loggers.py:273] Engine 000: Avg prompt throughput: 34.1 tokens/s, Avg generation throughput: 489.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 69.4%
(APIServer pid=2755321) INFO: 127.0.0.1:46090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:46200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:00:11 [loggers.py:273] Engine 000: Avg prompt throughput: 13.7 tokens/s, Avg generation throughput: 216.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 70.2%
(APIServer pid=2755321) INFO: 127.0.0.1:50956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:50998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:00:21 [loggers.py:273] Engine 000: Avg prompt throughput: 43.0 tokens/s, Avg generation throughput: 574.3 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 71.5%
(APIServer pid=2755321) INFO: 127.0.0.1:51638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:00:31 [loggers.py:273] Engine 000: Avg prompt throughput: 815.1 tokens/s, Avg generation throughput: 615.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 72.6%
(APIServer pid=2755321) INFO: 127.0.0.1:51842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:52184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:00:41 [loggers.py:273] Engine 000: Avg prompt throughput: 43.2 tokens/s, Avg generation throughput: 548.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 74.0%
(APIServer pid=2755321) INFO: 127.0.0.1:35606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:35942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:00:51 [loggers.py:273] Engine 000: Avg prompt throughput: 29.2 tokens/s, Avg generation throughput: 561.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 76.1%
(APIServer pid=2755321) INFO: 127.0.0.1:34080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:01:01 [loggers.py:273] Engine 000: Avg prompt throughput: 48.4 tokens/s, Avg generation throughput: 653.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 79.0%
(APIServer pid=2755321) INFO: 127.0.0.1:33726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:33986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:34032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:01:11 [loggers.py:273] Engine 000: Avg prompt throughput: 782.0 tokens/s, Avg generation throughput: 553.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 82.3%
(APIServer pid=2755321) INFO: 127.0.0.1:43626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:01:21 [loggers.py:273] Engine 000: Avg prompt throughput: 16.3 tokens/s, Avg generation throughput: 175.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 83.8%
(APIServer pid=2755321) INFO: 127.0.0.1:39656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:40000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:40014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:40020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:01:31 [loggers.py:273] Engine 000: Avg prompt throughput: 45.0 tokens/s, Avg generation throughput: 647.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 88.1%
(APIServer pid=2755321) INFO: 127.0.0.1:37546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:37910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:01:41 [loggers.py:273] Engine 000: Avg prompt throughput: 40.4 tokens/s, Avg generation throughput: 533.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 91.8%
(APIServer pid=2755321) INFO: 127.0.0.1:39204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:39498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:01:51 [loggers.py:273] Engine 000: Avg prompt throughput: 40.0 tokens/s, Avg generation throughput: 579.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 95.9%
(APIServer pid=2755321) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:45318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:02:01 [loggers.py:273] Engine 000: Avg prompt throughput: 34.2 tokens/s, Avg generation throughput: 587.7 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 99.2%
(APIServer pid=2755321) INFO: 127.0.0.1:43734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:43994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:44068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:02:11 [loggers.py:273] Engine 000: Avg prompt throughput: 39.3 tokens/s, Avg generation throughput: 613.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 99.2%
(APIServer pid=2755321) INFO: 127.0.0.1:51116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO: 127.0.0.1:51414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2755321) INFO 07-12 17:02:19 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2755321) INFO 07-12 17:02:19 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2755321) INFO 07-12 17:02:19 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2755321) INFO 07-12 17:02:19 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2755628) INFO 07-12 17:02:19 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2755628) INFO 07-12 17:02:19 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2755628) INFO 07-12 17:02:19 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2755628) INFO 07-12 17:02:19 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(Worker_TP0 pid=2755779) INFO 07-12 17:02:19 [multiproc_executor.py:790] Parent process exited, terminating worker queues
(EngineCore pid=2755628) INFO 07-12 17:02:19 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2
(APIServer pid=2755321) INFO: Shutting down
(APIServer pid=2755321) INFO: Waiting for application shutdown.
(APIServer pid=2755321) INFO: Application shutdown complete.
(APIServer pid=2755321) INFO: Finished server process [2755321]
(EngineCore pid=2755628) INFO 07-12 17:02:23 [multiproc_executor.py:433] [shutdown] Executor: all workers exited gracefully
(APIServer pid=2755321) INFO 07-12 17:02:24 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2755321) INFO 07-12 17:02:24 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2755321) INFO 07-12 17:02:24 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink