gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1db737e6414620d96375690d961d0b0c385d7041
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns8/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

1464 lines
154 KiB
Plaintext
Raw Blame History

(APIServer pid=2769802) INFO 07-12 17:10:09 [api_utils.py:339]
(APIServer pid=2769802) INFO 07-12 17:10:09 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2769802) INFO 07-12 17:10:09 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2769802) INFO 07-12 17:10:09 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2769802) INFO 07-12 17:10:09 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2769802) INFO 07-12 17:10:09 [api_utils.py:339]
(APIServer pid=2769802) INFO 07-12 17:10:09 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8708, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'max_num_batched_tokens': 8192, 'max_num_seqs': 8, 'shutdown_timeout': 120}
(APIServer pid=2769802) INFO 07-12 17:10:18 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2769802) INFO 07-12 17:10:18 [model.py:1725] Using max model len 40960
(APIServer pid=2769802) INFO 07-12 17:10:18 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2769802) INFO 07-12 17:10:18 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2769802) INFO 07-12 17:10:18 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2770271) INFO 07-12 17:10:28 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2770271) INFO 07-12 17:10:29 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:41283 backend=nccl
(EngineCore pid=2770271) INFO 07-12 17:10:29 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=2770271) INFO 07-12 17:10:30 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=2770271) INFO 07-12 17:10:30 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(EngineCore pid=2770271) INFO 07-12 17:10:31 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=2770271) INFO 07-12 17:10:31 [flash_attn.py:670] Using FlashAttention version 3
(EngineCore pid=2770271) INFO 07-12 17:10:31 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(EngineCore pid=2770271) INFO 07-12 17:10:31 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1288.54 GiB.
(EngineCore pid=2770271) INFO 07-12 17:10:31 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:13, 1.15it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:12, 1.11it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:02<00:11, 1.10it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:03<00:11, 1.08it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:04<00:10, 1.08it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:05<00:09, 1.06it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:06<00:08, 1.06it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:07<00:07, 1.08it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:08<00:06, 1.09it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:09<00:05, 1.09it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:10<00:04, 1.10it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:11<00:03, 1.10it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:11<00:02, 1.09it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:12<00:01, 1.08it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:13<00:00, 1.08it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:14<00:00, 1.38it/s]
(EngineCore pid=2770271)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:14<00:00, 1.14it/s]
(EngineCore pid=2770271)
(EngineCore pid=2770271) INFO 07-12 17:10:45 [default_loader.py:430] Loading weights took 14.10 seconds
(EngineCore pid=2770271) INFO 07-12 17:10:45 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=2770271) INFO 07-12 17:10:46 [gpu_model_runner.py:5259] Model loading took 56.88 GiB memory and 14.734821 seconds
(EngineCore pid=2770271) INFO 07-12 17:10:50 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/b95eb69a3d/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=2770271) INFO 07-12 17:10:50 [backends.py:1148] Dynamo bytecode transform time: 3.32 s
(EngineCore pid=2770271) INFO 07-12 17:10:52 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 2.279 s
(EngineCore pid=2770271) INFO 07-12 17:10:52 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/802e9eed10a34f87d59e9a1acfc3b883a63a5eaeb40312e4832ac27dfbf354e5/rank_0_0/model
(EngineCore pid=2770271) INFO 07-12 17:10:52 [monitor.py:53] torch.compile took 5.99 s in total
(EngineCore pid=2770271) INFO 07-12 17:10:52 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=NVIDIA_H20.json for MoE layer.
(EngineCore pid=2770271) INFO 07-12 17:10:53 [monitor.py:81] Initial profiling/warmup run took 0.25 s
(EngineCore pid=2770271) INFO 07-12 17:10:53 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(EngineCore pid=2770271) INFO 07-12 17:10:55 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.07 GiB total
(EngineCore pid=2770271) INFO 07-12 17:10:55 [gpu_worker.py:508] Available KV cache memory: 29.44 GiB
(EngineCore pid=2770271) INFO 07-12 17:10:55 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9193 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9207. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2770271) INFO 07-12 17:10:55 [kv_cache_utils.py:2146] GPU KV cache size: 321,600 tokens
(EngineCore pid=2770271) INFO 07-12 17:10:55 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 7.85x
(EngineCore pid=2770271) INFO 07-12 17:10:55 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=2770271) INFO 07-12 17:10:55 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=2770271) 2026-07-12 17:10:55,939 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(EngineCore pid=2770271) 2026-07-12 17:10:55,984 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(EngineCore pid=2770271)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/5 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 20%|██ | 1/5 [00:00<00:00, 9.19it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 40%|████ | 2/5 [00:00<00:00, 9.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 80%|████████ | 4/5 [00:00<00:00, 8.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:00<00:00, 8.02it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:00<00:00, 8.38it/s]
(EngineCore pid=2770271)
Capturing CUDA graphs (decode, FULL): 0%| | 0/4 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 50%|█████ | 2/4 [00:00<00:00, 11.08it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 4/4 [00:00<00:00, 11.41it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 4/4 [00:00<00:00, 11.35it/s]
(EngineCore pid=2770271) INFO 07-12 17:10:57 [gpu_model_runner.py:6660] Graph capturing finished in 2 secs, took 0.08 GiB
(EngineCore pid=2770271) INFO 07-12 17:10:57 [gpu_worker.py:667] CUDA graph pool memory: 0.08 GiB (actual), 0.07 GiB (estimated), difference: 0.01 GiB (12.2%).
(EngineCore pid=2770271) INFO 07-12 17:10:57 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2770271) INFO 07-12 17:10:57 [core.py:337] init engine (profile, create kv cache, warmup model) took 11.47 s (compilation: 5.99 s)
(EngineCore pid=2770271) INFO 07-12 17:10:58 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns8/opprof/opprof-v1-dp0-pid2770271-1783876258461072221.jsonl
(EngineCore pid=2770271) INFO 07-12 17:10:58 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2770271) INFO 07-12 17:10:58 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2769802) INFO 07-12 17:10:58 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2769802) WARNING 07-12 17:10:58 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2769802) INFO 07-12 17:10:59 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2769802) INFO 07-12 17:10:59 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8708
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:37] Available routes are:
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2769802) INFO 07-12 17:10:59 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2769802) INFO: Started server process [2769802]
(APIServer pid=2769802) INFO: Waiting for application startup.
(APIServer pid=2769802) INFO: Application startup complete.
(APIServer pid=2769802) INFO: 127.0.0.1:46570 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2770271) WARNING 07-12 17:11:05 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2769802) INFO: 127.0.0.1:36660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2770271) WARNING 07-12 17:11:05 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2769802) INFO: 127.0.0.1:36668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:11:09 [loggers.py:273] Engine 000: Avg prompt throughput: 2317.6 tokens/s, Avg generation throughput: 68.0 tokens/s, Running: 8 reqs, Waiting: 1 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 4.2%
(APIServer pid=2769802) INFO: 127.0.0.1:36730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:11:19 [loggers.py:273] Engine 000: Avg prompt throughput: 1782.3 tokens/s, Avg generation throughput: 130.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 5.0%
(APIServer pid=2769802) INFO: 127.0.0.1:57994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:11:29 [loggers.py:273] Engine 000: Avg prompt throughput: 252.6 tokens/s, Avg generation throughput: 240.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 47.2%
(APIServer pid=2769802) INFO: 127.0.0.1:39590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:11:39 [loggers.py:273] Engine 000: Avg prompt throughput: 6158.8 tokens/s, Avg generation throughput: 316.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 34.1%
(APIServer pid=2769802) INFO: 127.0.0.1:54462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:11:49 [loggers.py:273] Engine 000: Avg prompt throughput: 2063.2 tokens/s, Avg generation throughput: 121.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 30.9%
(APIServer pid=2769802) INFO: 127.0.0.1:54554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:11:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6430.2 tokens/s, Avg generation throughput: 270.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 25.5%
(APIServer pid=2769802) INFO: 127.0.0.1:46064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:12:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7624.4 tokens/s, Avg generation throughput: 353.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 21.7%
(APIServer pid=2769802) INFO: 127.0.0.1:37208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:12:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7247.2 tokens/s, Avg generation throughput: 257.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 18.7%
(APIServer pid=2769802) INFO: 127.0.0.1:52042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:12:29 [loggers.py:273] Engine 000: Avg prompt throughput: 2486.6 tokens/s, Avg generation throughput: 75.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 18.0%
(APIServer pid=2769802) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:12:39 [loggers.py:273] Engine 000: Avg prompt throughput: 4499.6 tokens/s, Avg generation throughput: 270.0 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 17.1%
(APIServer pid=2769802) INFO: 127.0.0.1:53564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:39022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:12:49 [loggers.py:273] Engine 000: Avg prompt throughput: 4808.1 tokens/s, Avg generation throughput: 298.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 16.4%
(APIServer pid=2769802) INFO: 127.0.0.1:59084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:12:59 [loggers.py:273] Engine 000: Avg prompt throughput: 4569.0 tokens/s, Avg generation throughput: 128.6 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 16.0%
(APIServer pid=2769802) INFO: 127.0.0.1:59206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:49516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:13:09 [loggers.py:273] Engine 000: Avg prompt throughput: 5494.8 tokens/s, Avg generation throughput: 275.8 tokens/s, Running: 8 reqs, Waiting: 2 reqs, GPU KV cache usage: 9.6%, Prefix cache hit rate: 15.3%
(APIServer pid=2769802) INFO: 127.0.0.1:49520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:13:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7696.8 tokens/s, Avg generation throughput: 337.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 14.5%
(APIServer pid=2769802) INFO: 127.0.0.1:41986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:13:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6654.9 tokens/s, Avg generation throughput: 280.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 13.8%
(APIServer pid=2769802) INFO: 127.0.0.1:47318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:13:39 [loggers.py:273] Engine 000: Avg prompt throughput: 1859.9 tokens/s, Avg generation throughput: 67.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 13.7%
(APIServer pid=2769802) INFO: 127.0.0.1:47390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:13:49 [loggers.py:273] Engine 000: Avg prompt throughput: 4874.8 tokens/s, Avg generation throughput: 280.5 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 5.8%, Prefix cache hit rate: 13.5%
(APIServer pid=2769802) INFO: 127.0.0.1:58384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:13:59 [loggers.py:273] Engine 000: Avg prompt throughput: 3508.7 tokens/s, Avg generation throughput: 227.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 13.5%
(APIServer pid=2769802) INFO: 127.0.0.1:56372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:14:09 [loggers.py:273] Engine 000: Avg prompt throughput: 4659.1 tokens/s, Avg generation throughput: 197.8 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 13.3%
(APIServer pid=2769802) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:14:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7160.0 tokens/s, Avg generation throughput: 245.2 tokens/s, Running: 8 reqs, Waiting: 6 reqs, GPU KV cache usage: 12.1%, Prefix cache hit rate: 12.9%
(APIServer pid=2769802) INFO: 127.0.0.1:59050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:14:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6056.8 tokens/s, Avg generation throughput: 325.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 12.6%
(APIServer pid=2769802) INFO: 127.0.0.1:34262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:14:39 [loggers.py:273] Engine 000: Avg prompt throughput: 5135.5 tokens/s, Avg generation throughput: 204.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 12.4%
(APIServer pid=2769802) INFO: 127.0.0.1:42476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:14:49 [loggers.py:273] Engine 000: Avg prompt throughput: 3442.5 tokens/s, Avg generation throughput: 153.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 12.3%
(APIServer pid=2769802) INFO: 127.0.0.1:42612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:14:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6044.6 tokens/s, Avg generation throughput: 318.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 12.3%
(APIServer pid=2769802) INFO: 127.0.0.1:36898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:15:09 [loggers.py:273] Engine 000: Avg prompt throughput: 2760.2 tokens/s, Avg generation throughput: 205.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 12.3%
(APIServer pid=2769802) INFO: 127.0.0.1:53096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:15:19 [loggers.py:273] Engine 000: Avg prompt throughput: 4610.5 tokens/s, Avg generation throughput: 253.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 12.3%
(APIServer pid=2769802) INFO: 127.0.0.1:53262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:15:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7306.4 tokens/s, Avg generation throughput: 275.9 tokens/s, Running: 8 reqs, Waiting: 1 reqs, GPU KV cache usage: 5.8%, Prefix cache hit rate: 12.1%
(APIServer pid=2769802) INFO: 127.0.0.1:55552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:15:39 [loggers.py:273] Engine 000: Avg prompt throughput: 8662.7 tokens/s, Avg generation throughput: 302.6 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.1%, Prefix cache hit rate: 11.7%
(APIServer pid=2769802) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:15:49 [loggers.py:273] Engine 000: Avg prompt throughput: 2354.0 tokens/s, Avg generation throughput: 153.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 11.7%
(APIServer pid=2769802) INFO: 127.0.0.1:50954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:15:59 [loggers.py:273] Engine 000: Avg prompt throughput: 4463.9 tokens/s, Avg generation throughput: 239.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 11.6%
(APIServer pid=2769802) INFO: 127.0.0.1:60800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:60808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:16:09 [loggers.py:273] Engine 000: Avg prompt throughput: 5549.2 tokens/s, Avg generation throughput: 325.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 11.5%
(APIServer pid=2769802) INFO: 127.0.0.1:43468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:16:19 [loggers.py:273] Engine 000: Avg prompt throughput: 2700.0 tokens/s, Avg generation throughput: 139.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 11.6%
(APIServer pid=2769802) INFO: 127.0.0.1:43564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:43576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:16:29 [loggers.py:273] Engine 000: Avg prompt throughput: 5916.0 tokens/s, Avg generation throughput: 269.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 11.6%
(APIServer pid=2769802) INFO: 127.0.0.1:52072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:52086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:16:39 [loggers.py:273] Engine 000: Avg prompt throughput: 8115.0 tokens/s, Avg generation throughput: 350.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 11.5%
(APIServer pid=2769802) INFO: 127.0.0.1:37428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:16:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7249.1 tokens/s, Avg generation throughput: 279.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 11.3%
(APIServer pid=2769802) INFO: 127.0.0.1:37214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:16:59 [loggers.py:273] Engine 000: Avg prompt throughput: 2772.3 tokens/s, Avg generation throughput: 66.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 11.2%
(APIServer pid=2769802) INFO: 127.0.0.1:53348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:53356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:17:09 [loggers.py:273] Engine 000: Avg prompt throughput: 4304.1 tokens/s, Avg generation throughput: 292.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 11.1%
(APIServer pid=2769802) INFO: 127.0.0.1:56836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:17:19 [loggers.py:273] Engine 000: Avg prompt throughput: 5203.6 tokens/s, Avg generation throughput: 314.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 11.2%
(APIServer pid=2769802) INFO: 127.0.0.1:57460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:57470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:56228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:17:29 [loggers.py:273] Engine 000: Avg prompt throughput: 4626.8 tokens/s, Avg generation throughput: 129.9 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 11.2%
(APIServer pid=2769802) INFO: 127.0.0.1:56238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:17:39 [loggers.py:273] Engine 000: Avg prompt throughput: 5472.3 tokens/s, Avg generation throughput: 288.0 tokens/s, Running: 8 reqs, Waiting: 3 reqs, GPU KV cache usage: 8.4%, Prefix cache hit rate: 11.2%
(APIServer pid=2769802) INFO: 127.0.0.1:37912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:32996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:33146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:17:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7595.6 tokens/s, Avg generation throughput: 368.1 tokens/s, Running: 8 reqs, Waiting: 2 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 11.1%
(APIServer pid=2769802) INFO: 127.0.0.1:33160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:48196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:17:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6878.9 tokens/s, Avg generation throughput: 288.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 10.9%
(APIServer pid=2769802) INFO: 127.0.0.1:46486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:18:09 [loggers.py:273] Engine 000: Avg prompt throughput: 3327.2 tokens/s, Avg generation throughput: 99.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 10.9%
(APIServer pid=2769802) INFO: 127.0.0.1:46578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:40932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:40948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:40950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:40952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:40966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:40972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:40986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:18:19 [loggers.py:273] Engine 000: Avg prompt throughput: 5107.8 tokens/s, Avg generation throughput: 290.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 10.9%
(APIServer pid=2769802) INFO: 127.0.0.1:41156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:41176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:18:29 [loggers.py:273] Engine 000: Avg prompt throughput: 3691.8 tokens/s, Avg generation throughput: 252.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 10.9%
(APIServer pid=2769802) INFO: 127.0.0.1:47388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:18:39 [loggers.py:273] Engine 000: Avg prompt throughput: 4858.2 tokens/s, Avg generation throughput: 211.2 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 10.9%
(APIServer pid=2769802) INFO: 127.0.0.1:47580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:18:49 [loggers.py:273] Engine 000: Avg prompt throughput: 6609.8 tokens/s, Avg generation throughput: 259.8 tokens/s, Running: 8 reqs, Waiting: 6 reqs, GPU KV cache usage: 12.2%, Prefix cache hit rate: 10.8%
(APIServer pid=2769802) INFO: 127.0.0.1:35408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:55808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:18:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6930.2 tokens/s, Avg generation throughput: 364.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 10.8%
(APIServer pid=2769802) INFO: 127.0.0.1:58288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:19:09 [loggers.py:273] Engine 000: Avg prompt throughput: 5135.2 tokens/s, Avg generation throughput: 213.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 10.7%
(APIServer pid=2769802) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:47660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:19:19 [loggers.py:273] Engine 000: Avg prompt throughput: 4480.6 tokens/s, Avg generation throughput: 171.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 10.6%
(APIServer pid=2769802) INFO: 127.0.0.1:47662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:19:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6546.0 tokens/s, Avg generation throughput: 324.3 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 10.7%
(APIServer pid=2769802) INFO: 127.0.0.1:38224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:38240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:46464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:19:39 [loggers.py:273] Engine 000: Avg prompt throughput: 3276.3 tokens/s, Avg generation throughput: 233.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 10.7%
(APIServer pid=2769802) INFO: 127.0.0.1:46480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:45746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:19:49 [loggers.py:273] Engine 000: Avg prompt throughput: 4895.9 tokens/s, Avg generation throughput: 257.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 10.7%
(APIServer pid=2769802) INFO: 127.0.0.1:45752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:50996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:19:59 [loggers.py:273] Engine 000: Avg prompt throughput: 8097.1 tokens/s, Avg generation throughput: 288.5 tokens/s, Running: 8 reqs, Waiting: 4 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 10.8%
(APIServer pid=2769802) INFO: 127.0.0.1:51200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:54270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:20:09 [loggers.py:273] Engine 000: Avg prompt throughput: 8497.5 tokens/s, Avg generation throughput: 320.6 tokens/s, Running: 8 reqs, Waiting: 3 reqs, GPU KV cache usage: 11.3%, Prefix cache hit rate: 9.2%
(APIServer pid=2769802) INFO: 127.0.0.1:54284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:51544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:20:19 [loggers.py:273] Engine 000: Avg prompt throughput: 4302.3 tokens/s, Avg generation throughput: 234.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 9.2%
(APIServer pid=2769802) INFO: 127.0.0.1:37414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:20:29 [loggers.py:273] Engine 000: Avg prompt throughput: 4518.6 tokens/s, Avg generation throughput: 191.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 9.1%
(APIServer pid=2769802) INFO: 127.0.0.1:37568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:37594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:20:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7035.5 tokens/s, Avg generation throughput: 365.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 9.2%
(APIServer pid=2769802) INFO: 127.0.0.1:58248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:58260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:20:49 [loggers.py:273] Engine 000: Avg prompt throughput: 3035.3 tokens/s, Avg generation throughput: 185.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 9.2%
(APIServer pid=2769802) INFO: 127.0.0.1:36514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:20:59 [loggers.py:273] Engine 000: Avg prompt throughput: 5391.5 tokens/s, Avg generation throughput: 257.6 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 9.3%
(APIServer pid=2769802) INFO: 127.0.0.1:36068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:21:09 [loggers.py:273] Engine 000: Avg prompt throughput: 8610.3 tokens/s, Avg generation throughput: 308.2 tokens/s, Running: 8 reqs, Waiting: 4 reqs, GPU KV cache usage: 5.8%, Prefix cache hit rate: 9.4%
(APIServer pid=2769802) INFO: 127.0.0.1:42722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:21:19 [loggers.py:273] Engine 000: Avg prompt throughput: 8750.1 tokens/s, Avg generation throughput: 323.9 tokens/s, Running: 8 reqs, Waiting: 3 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 9.2%
(APIServer pid=2769802) INFO: 127.0.0.1:34780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:34784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:42762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:21:29 [loggers.py:273] Engine 000: Avg prompt throughput: 2755.8 tokens/s, Avg generation throughput: 198.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 9.2%
(APIServer pid=2769802) INFO: 127.0.0.1:36162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:21:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7597.3 tokens/s, Avg generation throughput: 275.1 tokens/s, Running: 8 reqs, Waiting: 14 reqs, GPU KV cache usage: 8.0%, Prefix cache hit rate: 9.0%
(APIServer pid=2769802) INFO: 127.0.0.1:36472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:36502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO: 127.0.0.1:59904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2769802) INFO 07-12 17:21:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7305.5 tokens/s, Avg generation throughput: 419.9 tokens/s, Running: 8 reqs, Waiting: 17 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 9.2%
(APIServer pid=2769802) INFO 07-12 17:21:56 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2769802) INFO 07-12 17:21:56 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2769802) INFO 07-12 17:21:56 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2769802) INFO 07-12 17:21:56 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2770271) INFO 07-12 17:21:56 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2770271) INFO 07-12 17:21:56 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2770271) INFO 07-12 17:21:56 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2770271) INFO 07-12 17:21:56 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(APIServer pid=2769802) INFO: Shutting down
(APIServer pid=2769802) INFO: Waiting for application shutdown.
(APIServer pid=2769802) INFO: Application shutdown complete.
(APIServer pid=2769802) INFO: Finished server process [2769802]
(APIServer pid=2769802) INFO 07-12 17:21:58 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2769802) INFO 07-12 17:21:58 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2769802) INFO 07-12 17:21:58 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink