gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1db737e6414620d96375690d961d0b0c385d7041
aituner/runs/opprof-phase6/phase6/cells/tp1_mns64/server.log
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

438 lines
53 KiB
Plaintext
Raw Blame History

(APIServer pid=2638189) INFO 07-12 14:50:41 [api_utils.py:339]
(APIServer pid=2638189) INFO 07-12 14:50:41 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2638189) INFO 07-12 14:50:41 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2638189) INFO 07-12 14:50:41 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2638189) INFO 07-12 14:50:41 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2638189) INFO 07-12 14:50:41 [api_utils.py:339]
(APIServer pid=2638189) INFO 07-12 14:50:41 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8503, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'max_num_batched_tokens': 8192, 'max_num_seqs': 64, 'shutdown_timeout': 120}
(APIServer pid=2638189) INFO 07-12 14:50:52 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2638189) INFO 07-12 14:50:52 [model.py:1725] Using max model len 40960
(APIServer pid=2638189) INFO 07-12 14:50:52 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2638189) INFO 07-12 14:50:52 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2638189) INFO 07-12 14:50:52 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2638898) INFO 07-12 14:51:03 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2638898) INFO 07-12 14:51:06 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:39471 backend=nccl
(EngineCore pid=2638898) INFO 07-12 14:51:06 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=2638898) INFO 07-12 14:51:07 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=2638898) INFO 07-12 14:51:07 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(EngineCore pid=2638898) INFO 07-12 14:51:07 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=2638898) INFO 07-12 14:51:07 [flash_attn.py:670] Using FlashAttention version 3
(EngineCore pid=2638898) INFO 07-12 14:51:07 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(EngineCore pid=2638898) INFO 07-12 14:51:08 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1283.36 GiB.
(EngineCore pid=2638898) INFO 07-12 14:51:08 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:01<00:15, 1.06s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:02<00:14, 1.07s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:03<00:13, 1.06s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:04<00:13, 1.10s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:05<00:11, 1.08s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:06<00:10, 1.09s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:07<00:09, 1.11s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:08<00:08, 1.09s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:09<00:07, 1.08s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:10<00:06, 1.06s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:11<00:05, 1.06s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:12<00:04, 1.04s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:13<00:03, 1.06s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:15<00:02, 1.09s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:16<00:01, 1.09s/it]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:16<00:00, 1.17it/s]
(EngineCore pid=2638898)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:16<00:00, 1.03s/it]
(EngineCore pid=2638898)
(EngineCore pid=2638898) INFO 07-12 14:51:25 [default_loader.py:430] Loading weights took 16.52 seconds
(EngineCore pid=2638898) INFO 07-12 14:51:25 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=2638898) INFO 07-12 14:51:25 [gpu_model_runner.py:5259] Model loading took 56.88 GiB memory and 17.262335 seconds
(EngineCore pid=2638898) INFO 07-12 14:51:29 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/65b50dadd2/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=2638898) INFO 07-12 14:51:29 [backends.py:1148] Dynamo bytecode transform time: 3.30 s
(EngineCore pid=2638898) INFO 07-12 14:51:31 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 2.131 s
(EngineCore pid=2638898) INFO 07-12 14:51:31 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/ab53f49ec98f407fe24fecb037cb59739264f283939c70f41986d8369686d472/rank_0_0/model
(EngineCore pid=2638898) INFO 07-12 14:51:31 [monitor.py:53] torch.compile took 5.83 s in total
(EngineCore pid=2638898) INFO 07-12 14:51:31 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=NVIDIA_H20.json for MoE layer.
(EngineCore pid=2638898) INFO 07-12 14:51:32 [monitor.py:81] Initial profiling/warmup run took 0.21 s
(EngineCore pid=2638898) INFO 07-12 14:51:32 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=19 (largest=128), FULL=11 (largest=64)
(EngineCore pid=2638898) INFO 07-12 14:51:34 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.19 GiB total
(EngineCore pid=2638898) INFO 07-12 14:51:34 [gpu_worker.py:508] Available KV cache memory: 29.33 GiB
(EngineCore pid=2638898) INFO 07-12 14:51:34 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9180 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9220. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2638898) INFO 07-12 14:51:34 [kv_cache_utils.py:2146] GPU KV cache size: 320,304 tokens
(EngineCore pid=2638898) INFO 07-12 14:51:34 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 7.82x
(EngineCore pid=2638898) INFO 07-12 14:51:35 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=2638898) INFO 07-12 14:51:35 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=2638898) 2026-07-12 14:51:35,044 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(EngineCore pid=2638898) 2026-07-12 14:51:35,116 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(EngineCore pid=2638898)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/19 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 5%|▌ | 1/19 [00:00<00:01, 9.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 3/19 [00:00<00:01, 10.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 26%|██▋ | 5/19 [00:00<00:01, 10.01it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 7/19 [00:00<00:01, 9.67it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 42%|████▏ | 8/19 [00:00<00:01, 9.52it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 10/19 [00:01<00:00, 9.72it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 58%|█████▊ | 11/19 [00:01<00:00, 9.69it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 12/19 [00:01<00:00, 9.19it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 68%|██████▊ | 13/19 [00:01<00:00, 9.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 74%|███████▎ | 14/19 [00:01<00:00, 8.79it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 79%|███████▉ | 15/19 [00:01<00:00, 8.44it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 16/19 [00:01<00:00, 8.20it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 89%|████████▉ | 17/19 [00:01<00:00, 7.99it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 95%|█████████▍| 18/19 [00:02<00:00, 8.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:02<00:00, 7.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 19/19 [00:02<00:00, 8.89it/s]
(EngineCore pid=2638898)
Capturing CUDA graphs (decode, FULL): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 2/11 [00:00<00:00, 10.99it/s]
Capturing CUDA graphs (decode, FULL): 36%|███▋ | 4/11 [00:00<00:00, 11.14it/s]
Capturing CUDA graphs (decode, FULL): 55%|█████▍ | 6/11 [00:00<00:00, 11.17it/s]
Capturing CUDA graphs (decode, FULL): 73%|███████▎ | 8/11 [00:00<00:00, 11.23it/s]
Capturing CUDA graphs (decode, FULL): 91%|█████████ | 10/11 [00:00<00:00, 11.31it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 11/11 [00:00<00:00, 11.28it/s]
(EngineCore pid=2638898) INFO 07-12 14:51:38 [gpu_model_runner.py:6660] Graph capturing finished in 4 secs, took 0.24 GiB
(EngineCore pid=2638898) INFO 07-12 14:51:38 [gpu_worker.py:667] CUDA graph pool memory: 0.24 GiB (actual), 0.19 GiB (estimated), difference: 0.05 GiB (19.8%).
(EngineCore pid=2638898) INFO 07-12 14:51:38 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2638898) INFO 07-12 14:51:39 [core.py:337] init engine (profile, create kv cache, warmup model) took 13.60 s (compilation: 5.83 s)
(EngineCore pid=2638898) INFO 07-12 14:51:39 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns64/opprof/opprof-v1-dp0-pid2638898-1783867899798067669.jsonl
(EngineCore pid=2638898) INFO 07-12 14:51:39 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2638898) INFO 07-12 14:51:39 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2638189) INFO 07-12 14:51:39 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2638189) WARNING 07-12 14:51:40 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2638189) INFO 07-12 14:51:40 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2638189) INFO 07-12 14:51:40 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8503
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:37] Available routes are:
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2638189) INFO 07-12 14:51:40 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2638189) INFO: Started server process [2638189]
(APIServer pid=2638189) INFO: Waiting for application startup.
(APIServer pid=2638189) INFO: Application startup complete.
(APIServer pid=2638189) INFO: 127.0.0.1:46204 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2638898) WARNING 07-12 14:52:29 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2638189) INFO: 127.0.0.1:46214 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2638898) WARNING 07-12 14:52:29 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2638189) INFO: 127.0.0.1:46234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:52:30 [loggers.py:273] Engine 000: Avg prompt throughput: 1668.3 tokens/s, Avg generation throughput: 1.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.6%, Prefix cache hit rate: 4.0%
(APIServer pid=2638189) INFO: 127.0.0.1:54168 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:54256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:52:40 [loggers.py:273] Engine 000: Avg prompt throughput: 3630.8 tokens/s, Avg generation throughput: 203.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 4.2%
(APIServer pid=2638189) INFO: 127.0.0.1:39372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:39388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:39404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:39408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:39412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:39416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:39430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:39442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:39444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:39452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:52:50 [loggers.py:273] Engine 000: Avg prompt throughput: 11.3 tokens/s, Avg generation throughput: 117.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 42.7%
(APIServer pid=2638189) INFO: 127.0.0.1:46416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46444 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46518 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46532 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46554 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46604 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46632 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:53:00 [loggers.py:273] Engine 000: Avg prompt throughput: 3820.4 tokens/s, Avg generation throughput: 268.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 40.3%
(APIServer pid=2638189) INFO: 127.0.0.1:33660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:33836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:53:10 [loggers.py:273] Engine 000: Avg prompt throughput: 4364.6 tokens/s, Avg generation throughput: 291.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 33.3%
(APIServer pid=2638189) INFO: 127.0.0.1:51654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51684 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51766 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:51834 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:53:20 [loggers.py:273] Engine 000: Avg prompt throughput: 4686.8 tokens/s, Avg generation throughput: 194.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 28.9%
(APIServer pid=2638189) INFO: 127.0.0.1:55448 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55546 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55588 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55610 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55646 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:55658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:53:30 [loggers.py:273] Engine 000: Avg prompt throughput: 7377.9 tokens/s, Avg generation throughput: 242.3 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.3%, Prefix cache hit rate: 24.2%
(APIServer pid=2638189) INFO: 127.0.0.1:37564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37622 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37696 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:53:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6039.0 tokens/s, Avg generation throughput: 407.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 21.3%
(APIServer pid=2638189) INFO: 127.0.0.1:52718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52736 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:52840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:53:50 [loggers.py:273] Engine 000: Avg prompt throughput: 5848.5 tokens/s, Avg generation throughput: 230.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 19.8%
(APIServer pid=2638189) INFO: 127.0.0.1:53304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53314 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53356 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53418 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:53446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:54:00 [loggers.py:273] Engine 000: Avg prompt throughput: 4481.1 tokens/s, Avg generation throughput: 175.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 18.4%
(APIServer pid=2638189) INFO: 127.0.0.1:42240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42292 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42348 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42362 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42380 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:42428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:54:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6653.1 tokens/s, Avg generation throughput: 325.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 17.6%
(APIServer pid=2638189) INFO: 127.0.0.1:43188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43264 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:43288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37148 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:54:20 [loggers.py:273] Engine 000: Avg prompt throughput: 3169.4 tokens/s, Avg generation throughput: 228.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 17.3%
(APIServer pid=2638189) INFO: 127.0.0.1:37150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37178 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37266 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37290 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37316 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:37330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:54:30 [loggers.py:273] Engine 000: Avg prompt throughput: 4896.1 tokens/s, Avg generation throughput: 259.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 16.9%
(APIServer pid=2638189) INFO: 127.0.0.1:59884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59980 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59990 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:59996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60086 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60090 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:60124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:54:40 [loggers.py:273] Engine 000: Avg prompt throughput: 8725.1 tokens/s, Avg generation throughput: 309.3 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 15.8%
(APIServer pid=2638189) INFO: 127.0.0.1:48398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48402 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48528 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48572 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48580 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:48598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO 07-12 14:54:50 [loggers.py:273] Engine 000: Avg prompt throughput: 8779.2 tokens/s, Avg generation throughput: 312.1 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.9%, Prefix cache hit rate: 14.8%
(APIServer pid=2638189) INFO: 127.0.0.1:46248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46256 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46296 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638189) INFO: 127.0.0.1:46330 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2638898) (APIServer pid=2638189) INFO 07-12 14:54:58 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
INFO 07-12 14:54:58 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=2638189) INFO 07-12 14:54:58 [launcher.py:116] [shutdown] API server: stopping engine client mode=drain timeout=120s
(EngineCore pid=2638898) INFO 07-12 14:54:58 [core.py:1333] [shutdown] EngineCore: start mode=drain timeout=120s
(EngineCore pid=2638898) INFO 07-12 14:54:58 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=2638898) INFO 07-12 14:54:58 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(APIServer pid=2638189) INFO 07-12 14:54:58 [core_client.py:655] [shutdown] MPClient: start timeout=120s
(APIServer pid=2638189) INFO 07-12 14:54:58 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(APIServer pid=2638189) INFO: Shutting down
(APIServer pid=2638189) INFO: Waiting for application shutdown.
(APIServer pid=2638189) INFO: Application shutdown complete.
(APIServer pid=2638189) INFO: Finished server process [2638189]
(APIServer pid=2638189) INFO 07-12 14:54:58 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=2638189) INFO 07-12 14:54:58 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=2638189) INFO 07-12 14:54:58 [core_client.py:662] [shutdown] MPClient: complete
/usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Reference in New Issue View Git Blame Copy Permalink