gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
34e1f4c144641c2b2fcc308a2b776de09f747a24
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns32/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

938 lines
102 KiB
Plaintext
Raw Blame History

(APIServer pid=2786173) INFO 07-12 17:27:18 [api_utils.py:339]
(APIServer pid=2786173) INFO 07-12 17:27:18 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2786173) INFO 07-12 17:27:18 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2786173) INFO 07-12 17:27:18 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2786173) INFO 07-12 17:27:18 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2786173) INFO 07-12 17:27:18 [api_utils.py:339]
(APIServer pid=2786173) INFO 07-12 17:27:18 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8710, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'max_num_batched_tokens': 8192, 'max_num_seqs': 32, 'shutdown_timeout': 120}
(APIServer pid=2786173) INFO 07-12 17:27:18 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2786173) INFO 07-12 17:27:18 [model.py:1725] Using max model len 40960
(APIServer pid=2786173) INFO 07-12 17:27:18 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2786173) INFO 07-12 17:27:18 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2786173) INFO 07-12 17:27:18 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2786356) INFO 07-12 17:27:28 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2786356) INFO 07-12 17:27:30 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:56647 backend=nccl
(EngineCore pid=2786356) INFO 07-12 17:27:30 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=2786356) INFO 07-12 17:27:31 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=2786356) INFO 07-12 17:27:31 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(EngineCore pid=2786356) INFO 07-12 17:27:31 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=2786356) INFO 07-12 17:27:31 [flash_attn.py:670] Using FlashAttention version 3
(EngineCore pid=2786356) INFO 07-12 17:27:31 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(EngineCore pid=2786356) INFO 07-12 17:27:32 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1288.53 GiB.
(EngineCore pid=2786356) INFO 07-12 17:27:32 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:12, 1.17it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:12, 1.12it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:02<00:13, 1.01s/it]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:03<00:11, 1.03it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:04<00:10, 1.06it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:05<00:09, 1.07it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:06<00:08, 1.08it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:07<00:07, 1.10it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:08<00:06, 1.12it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:09<00:05, 1.13it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:10<00:04, 1.14it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:10<00:03, 1.13it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:11<00:02, 1.13it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:12<00:01, 1.13it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:13<00:00, 1.12it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:13<00:00, 1.43it/s]
(EngineCore pid=2786356)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:13<00:00, 1.16it/s]
(EngineCore pid=2786356)
(EngineCore pid=2786356) INFO 07-12 17:27:46 [default_loader.py:430] Loading weights took 13.86 seconds
(EngineCore pid=2786356) INFO 07-12 17:27:46 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=2786356) INFO 07-12 17:27:46 [gpu_model_runner.py:5259] Model loading took 56.88 GiB memory and 14.484264 seconds
(EngineCore pid=2786356) INFO 07-12 17:27:49 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/30ef41b5f5/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=2786356) INFO 07-12 17:27:49 [backends.py:1148] Dynamo bytecode transform time: 3.07 s
(EngineCore pid=2786356) INFO 07-12 17:27:52 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 2.227 s
(EngineCore pid=2786356) INFO 07-12 17:27:52 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/738c624149a63d13eaf115eec4d2189ece948ac500e524d3e06e801d9915352d/rank_0_0/model
(EngineCore pid=2786356) INFO 07-12 17:27:52 [monitor.py:53] torch.compile took 5.68 s in total
(EngineCore pid=2786356) INFO 07-12 17:27:52 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=NVIDIA_H20.json for MoE layer.
(EngineCore pid=2786356) INFO 07-12 17:27:52 [monitor.py:81] Initial profiling/warmup run took 0.25 s
(EngineCore pid=2786356) INFO 07-12 17:27:53 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=11 (largest=64), FULL=7 (largest=32)
(EngineCore pid=2786356) INFO 07-12 17:27:55 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.11 GiB total
(EngineCore pid=2786356) INFO 07-12 17:27:55 [gpu_worker.py:508] Available KV cache memory: 29.4 GiB
(EngineCore pid=2786356) INFO 07-12 17:27:55 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9188 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9212. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2786356) INFO 07-12 17:27:55 [kv_cache_utils.py:2146] GPU KV cache size: 321,136 tokens
(EngineCore pid=2786356) INFO 07-12 17:27:55 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 7.84x
(EngineCore pid=2786356) INFO 07-12 17:27:55 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=2786356) INFO 07-12 17:27:55 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=2786356) 2026-07-12 17:27:55,549 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(EngineCore pid=2786356) 2026-07-12 17:27:55,596 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(EngineCore pid=2786356)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 18%|█▊ | 2/11 [00:00<00:00, 10.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 36%|███▋ | 4/11 [00:00<00:00, 10.50it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 55%|█████▍ | 6/11 [00:00<00:00, 10.45it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 73%|███████▎ | 8/11 [00:00<00:00, 8.62it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 82%|████████▏ | 9/11 [00:01<00:00, 8.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 91%|█████████ | 10/11 [00:01<00:00, 8.02it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:01<00:00, 7.68it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:01<00:00, 8.52it/s]
(EngineCore pid=2786356)
Capturing CUDA graphs (decode, FULL): 0%| | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 29%|██▊ | 2/7 [00:00<00:00, 11.62it/s]
Capturing CUDA graphs (decode, FULL): 57%|█████▋ | 4/7 [00:00<00:00, 11.67it/s]
Capturing CUDA graphs (decode, FULL): 86%|████████▌ | 6/7 [00:00<00:00, 11.85it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:00<00:00, 11.84it/s]
(EngineCore pid=2786356) INFO 07-12 17:27:58 [gpu_model_runner.py:6660] Graph capturing finished in 2 secs, took 0.13 GiB
(EngineCore pid=2786356) INFO 07-12 17:27:58 [gpu_worker.py:667] CUDA graph pool memory: 0.13 GiB (actual), 0.11 GiB (estimated), difference: 0.02 GiB (12.1%).
(EngineCore pid=2786356) INFO 07-12 17:27:58 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2786356) INFO 07-12 17:27:58 [core.py:337] init engine (profile, create kv cache, warmup model) took 11.67 s (compilation: 5.68 s)
(EngineCore pid=2786356) INFO 07-12 17:27:58 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns32/opprof/opprof-v1-dp0-pid2786356-1783877278912197439.jsonl
(EngineCore pid=2786356) INFO 07-12 17:27:58 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2786356) INFO 07-12 17:27:58 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2786173) INFO 07-12 17:27:58 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2786173) WARNING 07-12 17:27:59 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2786173) INFO 07-12 17:27:59 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2786173) INFO 07-12 17:27:59 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8710
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:37] Available routes are:
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2786173) INFO 07-12 17:27:59 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2786173) INFO: Started server process [2786173]
(APIServer pid=2786173) INFO: Waiting for application startup.
(APIServer pid=2786173) INFO: Application startup complete.
(APIServer pid=2786173) INFO: 127.0.0.1:38604 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:38614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2786356) WARNING 07-12 17:28:06 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2786173) INFO: 127.0.0.1:38618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:38630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:38634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:38638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2786356) WARNING 07-12 17:28:07 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2786173) INFO: 127.0.0.1:38650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:38660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:38662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:38664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:38680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:38682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:28:09 [loggers.py:273] Engine 000: Avg prompt throughput: 3320.8 tokens/s, Avg generation throughput: 65.0 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.9%, Prefix cache hit rate: 3.1%
(APIServer pid=2786173) INFO: 127.0.0.1:46870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:46874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:46888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:46890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:46892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:28:19 [loggers.py:273] Engine 000: Avg prompt throughput: 1678.4 tokens/s, Avg generation throughput: 133.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 4.2%
(APIServer pid=2786173) INFO: 127.0.0.1:40332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:28:29 [loggers.py:273] Engine 000: Avg prompt throughput: 741.6 tokens/s, Avg generation throughput: 274.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 45.9%
(APIServer pid=2786173) INFO: 127.0.0.1:40550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:28:39 [loggers.py:273] Engine 000: Avg prompt throughput: 6780.8 tokens/s, Avg generation throughput: 349.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 33.8%
(APIServer pid=2786173) INFO: 127.0.0.1:59632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:28:49 [loggers.py:273] Engine 000: Avg prompt throughput: 2089.4 tokens/s, Avg generation throughput: 131.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 31.3%
(APIServer pid=2786173) INFO: 127.0.0.1:59762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:28:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6406.3 tokens/s, Avg generation throughput: 279.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 26.5%
(APIServer pid=2786173) INFO: 127.0.0.1:41084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:29:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7668.2 tokens/s, Avg generation throughput: 381.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 23.1%
(APIServer pid=2786173) INFO: 127.0.0.1:43304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55652 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:29:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7347.0 tokens/s, Avg generation throughput: 290.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 20.3%
(APIServer pid=2786173) INFO: 127.0.0.1:55782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:55798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:29:29 [loggers.py:273] Engine 000: Avg prompt throughput: 2695.1 tokens/s, Avg generation throughput: 49.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.2%, Prefix cache hit rate: 19.1%
(APIServer pid=2786173) INFO: 127.0.0.1:53140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:29:39 [loggers.py:273] Engine 000: Avg prompt throughput: 5746.5 tokens/s, Avg generation throughput: 324.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 17.9%
(APIServer pid=2786173) INFO: 127.0.0.1:58582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58584 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:29:49 [loggers.py:273] Engine 000: Avg prompt throughput: 5414.2 tokens/s, Avg generation throughput: 327.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 17.4%
(APIServer pid=2786173) INFO: 127.0.0.1:33992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:29:59 [loggers.py:273] Engine 000: Avg prompt throughput: 4879.7 tokens/s, Avg generation throughput: 148.4 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 16.8%
(APIServer pid=2786173) INFO: 127.0.0.1:49832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:30:09 [loggers.py:273] Engine 000: Avg prompt throughput: 6255.2 tokens/s, Avg generation throughput: 283.8 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.7%, Prefix cache hit rate: 15.9%
(APIServer pid=2786173) INFO: 127.0.0.1:50026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:30:19 [loggers.py:273] Engine 000: Avg prompt throughput: 8412.7 tokens/s, Avg generation throughput: 386.2 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 15.2%
(APIServer pid=2786173) INFO: 127.0.0.1:42834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:58998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:30:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7168.7 tokens/s, Avg generation throughput: 332.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 14.6%
(APIServer pid=2786173) INFO: 127.0.0.1:59908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:59996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:30:39 [loggers.py:273] Engine 000: Avg prompt throughput: 2806.7 tokens/s, Avg generation throughput: 96.2 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.6%, Prefix cache hit rate: 14.3%
(APIServer pid=2786173) INFO: 127.0.0.1:59998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:30:49 [loggers.py:273] Engine 000: Avg prompt throughput: 6210.5 tokens/s, Avg generation throughput: 292.0 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 14.0%
(APIServer pid=2786173) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:42088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:30:59 [loggers.py:273] Engine 000: Avg prompt throughput: 4336.0 tokens/s, Avg generation throughput: 290.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 13.9%
(APIServer pid=2786173) INFO: 127.0.0.1:42096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:31:09 [loggers.py:273] Engine 000: Avg prompt throughput: 4970.2 tokens/s, Avg generation throughput: 207.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 13.8%
(APIServer pid=2786173) INFO: 127.0.0.1:36876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:36888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:31:19 [loggers.py:273] Engine 000: Avg prompt throughput: 8751.6 tokens/s, Avg generation throughput: 213.8 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.4%, Prefix cache hit rate: 13.3%
(APIServer pid=2786173) INFO: 127.0.0.1:60398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:60412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:31:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6532.7 tokens/s, Avg generation throughput: 447.7 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.0%, Prefix cache hit rate: 13.0%
(APIServer pid=2786173) INFO: 127.0.0.1:34646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:53120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:31:39 [loggers.py:273] Engine 000: Avg prompt throughput: 5848.1 tokens/s, Avg generation throughput: 257.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 12.8%
(APIServer pid=2786173) INFO: 127.0.0.1:57586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:31:49 [loggers.py:273] Engine 000: Avg prompt throughput: 4474.4 tokens/s, Avg generation throughput: 147.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 12.6%
(APIServer pid=2786173) INFO: 127.0.0.1:57714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:31:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6552.3 tokens/s, Avg generation throughput: 293.1 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.1%, Prefix cache hit rate: 12.5%
(APIServer pid=2786173) INFO: 127.0.0.1:49948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:49994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:50046 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:32:09 [loggers.py:273] Engine 000: Avg prompt throughput: 3238.5 tokens/s, Avg generation throughput: 268.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 12.5%
(APIServer pid=2786173) INFO: 127.0.0.1:50056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41822 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:41956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:32:19 [loggers.py:273] Engine 000: Avg prompt throughput: 4692.5 tokens/s, Avg generation throughput: 263.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 12.4%
(APIServer pid=2786173) INFO: 127.0.0.1:47724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:47978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:32:29 [loggers.py:273] Engine 000: Avg prompt throughput: 8338.4 tokens/s, Avg generation throughput: 299.1 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.6%, Prefix cache hit rate: 12.3%
(APIServer pid=2786173) INFO: 127.0.0.1:47990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:32:39 [loggers.py:273] Engine 000: Avg prompt throughput: 8498.7 tokens/s, Avg generation throughput: 312.5 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 11.9%
(APIServer pid=2786173) INFO: 127.0.0.1:40070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:40086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:34710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:32:49 [loggers.py:273] Engine 000: Avg prompt throughput: 4302.5 tokens/s, Avg generation throughput: 246.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 11.9%
(APIServer pid=2786173) INFO: 127.0.0.1:37076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:32:59 [loggers.py:273] Engine 000: Avg prompt throughput: 5531.4 tokens/s, Avg generation throughput: 197.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 11.7%
(APIServer pid=2786173) INFO: 127.0.0.1:43880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:43992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44060 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:44102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:33:09 [loggers.py:273] Engine 000: Avg prompt throughput: 6254.0 tokens/s, Avg generation throughput: 387.0 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 11.7%
(APIServer pid=2786173) INFO: 127.0.0.1:39180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:33:19 [loggers.py:273] Engine 000: Avg prompt throughput: 2883.5 tokens/s, Avg generation throughput: 168.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 11.7%
(APIServer pid=2786173) INFO: 127.0.0.1:39278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39154 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:33:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6387.2 tokens/s, Avg generation throughput: 257.9 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.8%, Prefix cache hit rate: 11.6%
(APIServer pid=2786173) INFO: 127.0.0.1:39226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:39236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:33:39 [loggers.py:273] Engine 000: Avg prompt throughput: 9079.1 tokens/s, Avg generation throughput: 346.6 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.4%, Prefix cache hit rate: 11.6%
(APIServer pid=2786173) INFO: 127.0.0.1:56494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:56502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:48610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:33:49 [loggers.py:273] Engine 000: Avg prompt throughput: 8292.8 tokens/s, Avg generation throughput: 338.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 11.3%
(APIServer pid=2786173) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:33:59 [loggers.py:273] Engine 000: Avg prompt throughput: 1669.9 tokens/s, Avg generation throughput: 133.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 11.3%
(APIServer pid=2786173) INFO: 127.0.0.1:37928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:37940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:34:09 [loggers.py:273] Engine 000: Avg prompt throughput: 11108.7 tokens/s, Avg generation throughput: 316.9 tokens/s, Running: 32 reqs, Waiting: 1 reqs, GPU KV cache usage: 28.8%, Prefix cache hit rate: 11.1%
(APIServer pid=2786173) INFO: 127.0.0.1:57648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:57666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:52176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:52190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:52192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO: 127.0.0.1:52206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2786173) INFO 07-12 17:34:14 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2786173) INFO 07-12 17:34:14 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2786173) INFO 07-12 17:34:14 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2786173) INFO 07-12 17:34:14 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2786356) INFO 07-12 17:34:14 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2786356) INFO 07-12 17:34:14 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2786356) INFO 07-12 17:34:14 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2786356) INFO 07-12 17:34:14 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(APIServer pid=2786173) INFO: Shutting down
(APIServer pid=2786173) INFO: Waiting for application shutdown.
(APIServer pid=2786173) INFO: Application shutdown complete.
(APIServer pid=2786173) INFO: Finished server process [2786173]
(APIServer pid=2786173) INFO 07-12 17:34:16 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2786173) INFO 07-12 17:34:16 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2786173) INFO 07-12 17:34:16 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink