gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
650f54b35ef300fb5c9c245163300996bee2ab8a
aituner/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns64/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

966 lines
106 KiB
Plaintext
Raw Blame History

(APIServer pid=2794784) INFO 07-12 17:36:28 [api_utils.py:339]
(APIServer pid=2794784) INFO 07-12 17:36:28 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2794784) INFO 07-12 17:36:28 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2794784) INFO 07-12 17:36:28 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2794784) INFO 07-12 17:36:28 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2794784) INFO 07-12 17:36:28 [api_utils.py:339]
(APIServer pid=2794784) INFO 07-12 17:36:28 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8711, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'max_num_batched_tokens': 8192, 'max_num_seqs': 64, 'shutdown_timeout': 120}
(APIServer pid=2794784) INFO 07-12 17:36:28 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2794784) INFO 07-12 17:36:28 [model.py:1725] Using max model len 40960
(APIServer pid=2794784) INFO 07-12 17:36:28 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2794784) INFO 07-12 17:36:28 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2794784) INFO 07-12 17:36:28 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2795014) INFO 07-12 17:36:37 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2795014) INFO 07-12 17:36:40 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:53983 backend=nccl
(EngineCore pid=2795014) INFO 07-12 17:36:40 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=2795014) INFO 07-12 17:36:41 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=2795014) INFO 07-12 17:36:41 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(EngineCore pid=2795014) INFO 07-12 17:36:41 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=2795014) INFO 07-12 17:36:41 [flash_attn.py:670] Using FlashAttention version 3
(EngineCore pid=2795014) INFO 07-12 17:36:41 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(EngineCore pid=2795014) INFO 07-12 17:36:41 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1288.62 GiB.
(EngineCore pid=2795014) INFO 07-12 17:36:41 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:00<00:12, 1.17it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:01<00:12, 1.13it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:02<00:11, 1.13it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:03<00:10, 1.11it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:04<00:09, 1.11it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:05<00:09, 1.09it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:06<00:08, 1.09it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:07<00:07, 1.11it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:08<00:06, 1.12it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:09<00:05, 1.10it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:09<00:04, 1.09it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:10<00:03, 1.11it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:11<00:02, 1.10it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:12<00:01, 1.10it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:13<00:00, 1.10it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:13<00:00, 1.40it/s]
(EngineCore pid=2795014)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:13<00:00, 1.16it/s]
(EngineCore pid=2795014)
(EngineCore pid=2795014) INFO 07-12 17:36:55 [default_loader.py:430] Loading weights took 13.83 seconds
(EngineCore pid=2795014) INFO 07-12 17:36:55 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=2795014) INFO 07-12 17:36:56 [gpu_model_runner.py:5259] Model loading took 56.88 GiB memory and 14.487128 seconds
(EngineCore pid=2795014) INFO 07-12 17:36:59 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/65b50dadd2/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=2795014) INFO 07-12 17:36:59 [backends.py:1148] Dynamo bytecode transform time: 3.25 s
(EngineCore pid=2795014) INFO 07-12 17:37:02 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 2.195 s
(EngineCore pid=2795014) INFO 07-12 17:37:02 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/ab53f49ec98f407fe24fecb037cb59739264f283939c70f41986d8369686d472/rank_0_0/model
(EngineCore pid=2795014) INFO 07-12 17:37:02 [monitor.py:53] torch.compile took 5.84 s in total
(EngineCore pid=2795014) INFO 07-12 17:37:02 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=NVIDIA_H20.json for MoE layer.
(EngineCore pid=2795014) INFO 07-12 17:37:02 [monitor.py:81] Initial profiling/warmup run took 0.21 s
(EngineCore pid=2795014) INFO 07-12 17:37:03 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=19 (largest=128), FULL=11 (largest=64)
(EngineCore pid=2795014) INFO 07-12 17:37:04 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.19 GiB total
(EngineCore pid=2795014) INFO 07-12 17:37:05 [gpu_worker.py:508] Available KV cache memory: 29.33 GiB
(EngineCore pid=2795014) INFO 07-12 17:37:05 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9180 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9220. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2795014) INFO 07-12 17:37:05 [kv_cache_utils.py:2146] GPU KV cache size: 320,304 tokens
(EngineCore pid=2795014) INFO 07-12 17:37:05 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 7.82x
(EngineCore pid=2795014) INFO 07-12 17:37:05 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=2795014) INFO 07-12 17:37:05 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=2795014) 2026-07-12 17:37:05,528 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(EngineCore pid=2795014) 2026-07-12 17:37:05,599 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(EngineCore pid=2795014)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 5%|▌ | 1/19 [00:00<00:01, 9.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 11%|█ | 2/19 [00:00<00:01, 9.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 3/19 [00:00<00:01, 9.58it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 21%|██ | 4/19 [00:00<00:01, 9.37it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:00<00:01, 9.40it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 32%|███▏ | 6/19 [00:00<00:01, 9.32it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 7/19 [00:00<00:01, 9.13it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:00<00:01, 8.96it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 9/19 [00:00<00:01, 9.13it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 10/19 [00:01<00:00, 9.25it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:01<00:00, 9.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:01<00:00, 8.24it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 68%|██████▊ | 13/19 [00:01<00:00, 8.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:01<00:00, 7.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 79%|███████▉ | 15/19 [00:01<00:00, 7.48it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 16/19 [00:01<00:00, 7.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:02<00:00, 7.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:02<00:00, 7.06it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:02<00:00, 6.90it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:02<00:00, 8.12it/s]
(EngineCore pid=2795014)
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:00<00:00, 10.54it/s]
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:00<00:00, 10.74it/s]
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:00<00:00, 10.88it/s]
Capturing CUDA graphs (decode, FULL): 73%|███████▎ | 8/11 [00:00<00:00, 10.76it/s]
Capturing CUDA graphs (decode, FULL): 91%|█████████ | 10/11 [00:00<00:00, 10.96it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:01<00:00, 10.93it/s]
(EngineCore pid=2795014) INFO 07-12 17:37:09 [gpu_model_runner.py:6660] Graph capturing finished in 4 secs, took 0.24 GiB
(EngineCore pid=2795014) INFO 07-12 17:37:09 [gpu_worker.py:667] CUDA graph pool memory: 0.24 GiB (actual), 0.19 GiB (estimated), difference: 0.05 GiB (19.8%).
(EngineCore pid=2795014) INFO 07-12 17:37:09 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2795014) INFO 07-12 17:37:09 [core.py:337] init engine (profile, create kv cache, warmup model) took 13.56 s (compilation: 5.84 s)
(EngineCore pid=2795014) INFO 07-12 17:37:10 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns64/opprof/opprof-v1-dp0-pid2795014-1783877830485564578.jsonl
(EngineCore pid=2795014) INFO 07-12 17:37:10 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2795014) INFO 07-12 17:37:10 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2794784) INFO 07-12 17:37:10 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2794784) WARNING 07-12 17:37:10 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2794784) INFO 07-12 17:37:11 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2794784) INFO 07-12 17:37:11 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8711
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:37] Available routes are:
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2794784) INFO 07-12 17:37:11 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2794784) INFO: Started server process [2794784]
(APIServer pid=2794784) INFO: Waiting for application startup.
(APIServer pid=2794784) INFO: Application startup complete.
(APIServer pid=2794784) INFO: 127.0.0.1:58048 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2795014) WARNING 07-12 17:37:18 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2794784) INFO: 127.0.0.1:58064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2795014) WARNING 07-12 17:37:18 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2794784) INFO: 127.0.0.1:58078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:37:21 [loggers.py:273] Engine 000: Avg prompt throughput: 2652.2 tokens/s, Avg generation throughput: 72.9 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.6%, Prefix cache hit rate: 3.3%
(APIServer pid=2794784) INFO: 127.0.0.1:48554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:37:31 [loggers.py:273] Engine 000: Avg prompt throughput: 2402.9 tokens/s, Avg generation throughput: 125.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 4.2%
(APIServer pid=2794784) INFO: 127.0.0.1:38796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:37:41 [loggers.py:273] Engine 000: Avg prompt throughput: 741.6 tokens/s, Avg generation throughput: 249.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 48.9%
(APIServer pid=2794784) INFO: 127.0.0.1:52104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:37:51 [loggers.py:273] Engine 000: Avg prompt throughput: 5886.8 tokens/s, Avg generation throughput: 361.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 34.9%
(APIServer pid=2794784) INFO: 127.0.0.1:58218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:38:01 [loggers.py:273] Engine 000: Avg prompt throughput: 2626.0 tokens/s, Avg generation throughput: 142.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 32.7%
(APIServer pid=2794784) INFO: 127.0.0.1:44458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:38:11 [loggers.py:273] Engine 000: Avg prompt throughput: 6273.9 tokens/s, Avg generation throughput: 257.9 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 27.0%
(APIServer pid=2794784) INFO: 127.0.0.1:49760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49844 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:38:21 [loggers.py:273] Engine 000: Avg prompt throughput: 8155.7 tokens/s, Avg generation throughput: 367.8 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.6%, Prefix cache hit rate: 23.1%
(APIServer pid=2794784) INFO: 127.0.0.1:60424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:38:31 [loggers.py:273] Engine 000: Avg prompt throughput: 7168.1 tokens/s, Avg generation throughput: 301.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 20.2%
(APIServer pid=2794784) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:38:41 [loggers.py:273] Engine 000: Avg prompt throughput: 2241.2 tokens/s, Avg generation throughput: 73.7 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 19.5%
(APIServer pid=2794784) INFO: 127.0.0.1:43752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:43926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:38:51 [loggers.py:273] Engine 000: Avg prompt throughput: 5967.3 tokens/s, Avg generation throughput: 305.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 18.1%
(APIServer pid=2794784) INFO: 127.0.0.1:39218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39346 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:39:01 [loggers.py:273] Engine 000: Avg prompt throughput: 5765.1 tokens/s, Avg generation throughput: 333.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 17.2%
(APIServer pid=2794784) INFO: 127.0.0.1:36766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:36874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:39:11 [loggers.py:273] Engine 000: Avg prompt throughput: 4274.7 tokens/s, Avg generation throughput: 133.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 16.8%
(APIServer pid=2794784) INFO: 127.0.0.1:56374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:56520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48522 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:39:21 [loggers.py:273] Engine 000: Avg prompt throughput: 6133.1 tokens/s, Avg generation throughput: 312.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.4%, Prefix cache hit rate: 16.1%
(APIServer pid=2794784) INFO: 127.0.0.1:48552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48716 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:48738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:39:31 [loggers.py:273] Engine 000: Avg prompt throughput: 8234.5 tokens/s, Avg generation throughput: 372.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.2%, Prefix cache hit rate: 15.3%
(APIServer pid=2794784) INFO: 127.0.0.1:58682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58854 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:58858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:39:41 [loggers.py:273] Engine 000: Avg prompt throughput: 8134.1 tokens/s, Avg generation throughput: 319.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 14.6%
(APIServer pid=2794784) INFO: 127.0.0.1:60852 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60866 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60868 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:39:51 [loggers.py:273] Engine 000: Avg prompt throughput: 2450.0 tokens/s, Avg generation throughput: 106.6 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 14.4%
(APIServer pid=2794784) INFO: 127.0.0.1:49686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:40:01 [loggers.py:273] Engine 000: Avg prompt throughput: 6478.6 tokens/s, Avg generation throughput: 257.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.0%, Prefix cache hit rate: 13.9%
(APIServer pid=2794784) INFO: 127.0.0.1:60644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:60816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:40:11 [loggers.py:273] Engine 000: Avg prompt throughput: 4424.8 tokens/s, Avg generation throughput: 339.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 13.9%
(APIServer pid=2794784) INFO: 127.0.0.1:55422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55450 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:40:21 [loggers.py:273] Engine 000: Avg prompt throughput: 4367.7 tokens/s, Avg generation throughput: 185.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 13.7%
(APIServer pid=2794784) INFO: 127.0.0.1:55130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53896 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53904 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:40:31 [loggers.py:273] Engine 000: Avg prompt throughput: 7697.0 tokens/s, Avg generation throughput: 225.4 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.7%, Prefix cache hit rate: 13.2%
(APIServer pid=2794784) INFO: 127.0.0.1:53928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53972 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:53984 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:40:41 [loggers.py:273] Engine 000: Avg prompt throughput: 7694.6 tokens/s, Avg generation throughput: 448.6 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 13.1%
(APIServer pid=2794784) INFO: 127.0.0.1:47580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47726 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:40:51 [loggers.py:273] Engine 000: Avg prompt throughput: 6341.9 tokens/s, Avg generation throughput: 267.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 12.8%
(APIServer pid=2794784) INFO: 127.0.0.1:59016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59042 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:41:01 [loggers.py:273] Engine 000: Avg prompt throughput: 4474.1 tokens/s, Avg generation throughput: 150.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 12.6%
(APIServer pid=2794784) INFO: 127.0.0.1:51020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51036 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:51230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:41:11 [loggers.py:273] Engine 000: Avg prompt throughput: 6553.7 tokens/s, Avg generation throughput: 298.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 12.5%
(APIServer pid=2794784) INFO: 127.0.0.1:39160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:41:21 [loggers.py:273] Engine 000: Avg prompt throughput: 3276.6 tokens/s, Avg generation throughput: 261.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 12.5%
(APIServer pid=2794784) INFO: 127.0.0.1:52404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:52586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:41:31 [loggers.py:273] Engine 000: Avg prompt throughput: 4826.0 tokens/s, Avg generation throughput: 261.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 12.5%
(APIServer pid=2794784) INFO: 127.0.0.1:47358 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:41:41 [loggers.py:273] Engine 000: Avg prompt throughput: 8166.5 tokens/s, Avg generation throughput: 296.0 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 12.3%
(APIServer pid=2794784) INFO: 127.0.0.1:45362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:45560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:41:51 [loggers.py:273] Engine 000: Avg prompt throughput: 8498.4 tokens/s, Avg generation throughput: 315.9 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.2%, Prefix cache hit rate: 11.9%
(APIServer pid=2794784) INFO: 127.0.0.1:38980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:38994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39020 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:39092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:42:01 [loggers.py:273] Engine 000: Avg prompt throughput: 4302.3 tokens/s, Avg generation throughput: 245.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 11.9%
(APIServer pid=2794784) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33394 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:42:11 [loggers.py:273] Engine 000: Avg prompt throughput: 4518.7 tokens/s, Avg generation throughput: 191.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 11.7%
(APIServer pid=2794784) INFO: 127.0.0.1:33422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33496 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33512 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:33664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:42:21 [loggers.py:273] Engine 000: Avg prompt throughput: 7035.5 tokens/s, Avg generation throughput: 369.9 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 11.7%
(APIServer pid=2794784) INFO: 127.0.0.1:57754 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:57834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:42:31 [loggers.py:273] Engine 000: Avg prompt throughput: 3034.9 tokens/s, Avg generation throughput: 181.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 11.7%
(APIServer pid=2794784) INFO: 127.0.0.1:59244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59360 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:59420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:42:41 [loggers.py:273] Engine 000: Avg prompt throughput: 5391.6 tokens/s, Avg generation throughput: 257.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 11.7%
(APIServer pid=2794784) INFO: 127.0.0.1:54760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54818 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54874 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54936 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54950 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:54998 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:55048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:42:51 [loggers.py:273] Engine 000: Avg prompt throughput: 10077.3 tokens/s, Avg generation throughput: 322.3 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 11.6%
(APIServer pid=2794784) INFO: 127.0.0.1:50352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50484 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:50540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:43:01 [loggers.py:273] Engine 000: Avg prompt throughput: 8366.7 tokens/s, Avg generation throughput: 335.8 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 11.3%
(APIServer pid=2794784) INFO: 127.0.0.1:44374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44396 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:44438 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:43:11 [loggers.py:273] Engine 000: Avg prompt throughput: 1671.8 tokens/s, Avg generation throughput: 172.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 11.3%
(APIServer pid=2794784) INFO: 127.0.0.1:35024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35056 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:35348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:43:21 [loggers.py:273] Engine 000: Avg prompt throughput: 11103.3 tokens/s, Avg generation throughput: 251.9 tokens/s, Running: 33 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 11.1%
(APIServer pid=2794784) INFO: 127.0.0.1:49222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49274 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49326 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49422 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49426 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49476 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO: 127.0.0.1:49510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2794784) INFO 07-12 17:43:31 [loggers.py:273] Engine 000: Avg prompt throughput: 10902.8 tokens/s, Avg generation throughput: 646.0 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.6%, Prefix cache hit rate: 11.0%
(APIServer pid=2794784) INFO 07-12 17:43:32 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2794784) INFO 07-12 17:43:32 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(APIServer pid=2794784) INFO 07-12 17:43:32 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2794784) INFO 07-12 17:43:32 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(EngineCore pid=2795014) INFO 07-12 17:43:32 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=2795014) INFO 07-12 17:43:32 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2795014) INFO 07-12 17:43:32 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2795014) INFO 07-12 17:43:32 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(APIServer pid=2794784) INFO: Shutting down
(APIServer pid=2794784) INFO: Waiting for application shutdown.
(APIServer pid=2794784) INFO: Application shutdown complete.
(APIServer pid=2794784) INFO: Finished server process [2794784]
(APIServer pid=2794784) INFO 07-12 17:43:35 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2794784) INFO 07-12 17:43:35 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2794784) INFO 07-12 17:43:35 [core_client.py:662] [shutdown] MPClient: complete
Reference in New Issue View Git Blame Copy Permalink