aituner/docs/opprof/phase0-recon-vllm-0.24.0.md

Phase 0 recon: vLLM 0.24.0 observability

Status: source recon and patch design complete; no instrumentation patch was implemented.

Campaign target: operator-level, pattern-conditioned profiling of Qwen3-30B-A3B serving on NVIDIA H20. This report is based on the pinned source clone only. No GPU was used, no package or model was installed, and no dash host was contacted.

Unless explicitly prefixed with v0.20.0:, every source path and line number below is relative to /home/gahow/phd/vllm-v0.24.0 at the pinned commit.

Outcome

vLLM 0.24.0 already contains most of the raw ingredients, but they are split across scheduler internals, aggregate metrics, worker traces, and an optional routed-expert return path. The smallest useful patch is therefore not a new profiler. It is one scheduler-owned per-step record that joins existing batch composition to the existing CUDAGraphStat; short torch-profiler windows and exact expert routes remain sampled Layer 2 facilities.

The proposed patch is detailed in patch-design.md. Estimated surface: five files and about 450 lines including tests, with about 260 production lines.

1. Source pin and compatibility record

The requested tag existed and the exact requested clone command completed successfully:

git clone --depth 1 --branch v0.24.0 https://github.com/vllm-project/vllm /home/gahow/phd/vllm-v0.24.0

Field	Pinned value
Repository	https://github.com/vllm-project/vllm
Local clone	/home/gahow/phd/vllm-v0.24.0
Tag	v0.24.0 (git describe --tags --exact-match)
Commit	ee0da84ab9e04ac7610e28580af62c365e898389
Clone completion	2026-07-11T08:26:16Z (2026-07-11 16:26:16 +08:00)
Checkout	Detached HEAD, clean
Top-level entries	45, including hidden entries and .git
Listing SHA-256	75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58

The listing digest is reproducible from entry names only, not file contents:

LC_ALL=C find . -mindepth 1 -maxdepth 1 -printf '%f\n' |
  LC_ALL=C sort | sha256sum

This definition includes .git, sorts bytewise under the C locale, and hashes the 45 LF-terminated root names. It is recorded to make the otherwise ambiguous phrase “sha256 of the top-level directory listing” precise.

Torch and CUDA requirements

• Build and NVIDIA runtime requirements pin torch==2.11.0 in pyproject.toml:1-13, requirements/build/cuda.txt:1-10, and requirements/cuda.txt:1-10; CMake repeats 2.11.0 as the supported CUDA and ROCm torch version at CMakeLists.txt:62-72.
• The NVIDIA requirements also pin torchvision==0.26.0, flashinfer-python==0.6.12, flashinfer-cubin==0.6.12, and CUDA-13 extras for CUTLASS DSL and Humming kernels (requirements/cuda.txt:6-29). Setup strips/substitutes those extras for CUDA 12 builds (setup.py:1066-1083).
• The source default is CUDA 13.0 (vllm/envs.py:85-87,555-563) and the Docker default is CUDA 13.0.2 (docker/Dockerfile:14-42). Wheel detection maps CUDA major 12 to cu129 and major 13 to cu130 (setup.py:528-568).
• The pinned installation document still describes the default precompiled binaries as CUDA 12.9 and lists CUDA 12.8 and 13.0 variants (docs/getting_started/installation/gpu.cuda.inc.md:4-19,24-54). This is a packaging/default distinction, not evidence that 13.0 is unsupported.
• NVIDIA compute capability 7.5+ is documented, and SM90 is in every relevant CMake CUDA architecture set (docs/getting_started/installation/gpu.cuda.inc.md:9-19; CMakeLists.txt:105-118). Optional FA3 and DeepGEMM builds need CUDA 12.3+, FlashMLA needs 12.9+, and the Hopper CUTLASS MoE build needs 12.3+ (setup.py:1111-1139; CMakeLists.txt:844-868).

Implication for dash0 later: H20/SM90 is a supported architecture, but the actual driver and installed CUDA were intentionally not queried in Phase 0. Phase 2 must choose a torch 2.11.0-compatible cu129 or cu130 artifact after checking the host driver. A wheel built against a different torch/CUDA build is not interchangeable; the source documentation explicitly warns about binary incompatibility (docs/getting_started/installation/gpu.cuda.inc.md:14-19).

2. Built-in observability inventory

2.1 Torch profiler integration

Configuration and control. v0.24.0 does not contain the legacy VLLM_TORCH_PROFILER_DIR symbol: a repository-wide exact-name search at the pinned commit returned zero matches. Profiling is now configured through ProfilerConfig: profiler is torch or cuda, and torch_profiler_dir holds the absolute/URI output location (vllm/config/profiler.py:33-46,125-145). Optional controls include stacks, FLOPs, gzip, CUDA-time tables, shapes, memory, frontend exclusion, delayed start, maximum iterations, and wait/warmup/active iteration scheduling (vllm/config/profiler.py:48-105).

The documented server form is a --profiler-config JSON object, for example {"profiler":"torch","torch_profiler_dir":"/abs/run/kineto"}, followed by POSTs to the two endpoints (docs/contributing/profiling.md:46-85).

When profiler config is present, the serving router exposes POST /start_profile and /stop_profile; without profiler config the router is not attached (vllm/entrypoints/serve/profile/api_router.py:21-45). The endpoints are therefore not an unauthenticated always-on feature independent of server configuration; they exist within the configured serving application and should be protected like the rest of the control plane.

Captured activity. Each GPU worker creates a torch profiler with both CPU and CUDA activities, using the configured shapes/memory/stack/FLOPs switches (vllm/v1/worker/gpu_worker.py:929-980 and vllm/profiler/wrapper.py:159-226). TensorBoard-compatible traces are written with an optional gzip handler. The wrapper also writes CPU/CUDA aggregate tables; only rank 0 prints the table to stdout (vllm/profiler/wrapper.py:251-287). The source documentation calls the profiler medium-overhead and warns that trace size and final flushing can be large (docs/contributing/profiling.md:1-38).

The V1 worker calls profiler.step() once per execution and wraps model execution in a record-function name containing context/generation request and token counts (vllm/v1/worker/gpu_worker.py:803-827,895-898). If frontend profiling is not ignored, AsyncLLM also creates a separate CPU-only trace (vllm/v1/engine/async_llm.py:178-200).

TP behavior. EngineCore.profile() delegates to the executor (vllm/v1/engine/core.py:662-663), whose profile RPC is a collective worker call (vllm/v1/executor/abstract.py:256-257). Multiprocessing broadcasts calls that do not declare a unique output rank, so the profile call reaches every TP worker (vllm/v1/executor/multiproc_executor.py:340-402). Each worker emits its own rank-qualified trace; the suffix includes DP/PP/TP/DCP/EP/global rank components (vllm/distributed/utils.py:664-695). For TP, kernel analysis must therefore preserve per-rank files, use the slowest rank for critical-path time, and optionally sum ranks to quantify total GPU work.

2.2 Iteration-level scheduler data and exported metrics

The V1 scheduler interface states that each schedule() output corresponds to one model forward and maps request IDs to scheduled tokens (vllm/v1/core/sched/interface.py:51-80). SchedulerOutput already carries new/cached request data, per-request and total scheduled tokens, common-prefix blocks, and the request IDs preempted in that step (vllm/v1/core/sched/output.py:180-219). The base scheduler constructs this object at vllm/v1/core/sched/scheduler.py:1012-1076, then advances request computed-token and chunk state at vllm/v1/core/sched/scheduler.py:1130-1155.

What is available today:

Requested datum	Internal per-step source	Externally exposed today
Batch size	len(SchedulerOutput.num_scheduled_tokens); the map is exact	No raw per-step batch-size Prometheus series. Optional iteration log reports context and generation request counts.
Scheduled prefill/decode tokens	compute_iteration_details() classifies new/cached context requests and sums scheduled tokens (vllm/v1/utils.py:766-813)	Optional log reports exact scheduled context/generation counts and elapsed time around the execution wait (vllm/v1/engine/core.py:433-472). Prometheus prompt/generation counters are cumulative output-side stats, not a raw scheduled-step stream.
Chunked-prefill splits	Request.is_prefill_chunk, num_computed_tokens, and prompt/output state exist (vllm/v1/request.py:130-175); extended chunks are classified as context	No chunk ordinal, first/middle/final split, or chunk-size series. These must be derived before _update_after_schedule() mutates state.
Preemptions	Exact step set in SchedulerOutput.preempted_req_ids	Stdout aggregates over its logging interval; vllm:num_preemptions is cumulative (vllm/v1/metrics/loggers.py:219-281,624-631).
Running/waiting/deferred queues	SchedulerStats.num_running_reqs, num_waiting_reqs, and num_skipped_waiting_reqs (vllm/v1/metrics/stats.py:170-183)	Latest-value gauges vllm:num_requests_running, vllm:num_requests_waiting, and reason-labeled capacity/deferred gauges (vllm/v1/metrics/loggers.py:453-496). Running queue size is not the same as scheduled batch size.
KV-cache usage	Scheduler reads kv_cache_manager.usage; the block pool computes a null-block-adjusted ratio in [0,1] (vllm/v1/core/kv_cache_manager.py:181-188; vllm/v1/core/block_pool.py:692-711)	Latest-value vllm:kv_cache_usage_perc; stdout prints the latest percent (vllm/v1/metrics/loggers.py:250-260,524-532). Raw total/free blocks are not exported per step.
Prefix-cache counters	PrefixCacheStats contains request/query/hit and separately preempted request/query/hit counters (vllm/v1/metrics/stats.py:114-143) for local and optional connector caches	Prometheus exposes cumulative token queries/hits for local and external caches (vllm/v1/metrics/loggers.py:547-565,1063-1101); stdout reports interval-derived hit rates. It does not expose the preempted sub-counters separately.
Total step tokens	Exact scheduled total is in SchedulerOutput	vllm:iteration_tokens_total is a histogram observed from output-side computed prompt plus generation tokens (vllm/v1/metrics/loggers.py:693-724,1148-1171), not a joinable raw step record.

IterationStats also has a wall timestamp, output-side prompt/generation data, preemptions, and finished-request latency samples (vllm/v1/metrics/stats.py:325-347). Scheduler.make_stats() produces latest queue/KV state, drains prefix stats, and passes through the step's CUDA-graph stat when an output is processed (vllm/v1/core/sched/scheduler.py:2228-2264). These are valuable existing plumbing, but async scheduling can schedule newer batches before an older output is processed. Consequently, those latest/drained values are not a safe per-step composition join without a schedule-time snapshot.

Bottom line: no new scheduler instrumentation is needed to discover batch and token composition, preemptions, queues, KV use, or prefix events. A patch is needed only to serialize their same-step association and the missing histograms/chunk split.

2.3 CUDA-graph observability

Yes, v0.24.0 can determine the runtime graph mode and capture bucket for each model step internally:

• CUDAGraphStat records unpadded tokens, padded tokens, padding, and runtime mode (vllm/compilation/cuda_graph.py:32-37).
• The GPU model runner asks the dispatcher for a concrete runtime mode and BatchDescriptor, then conditionally attaches the stat to its output (vllm/v1/worker/gpu_model_runner.py:3822-3934). It reaches the scheduler in ModelRunnerOutput and update_from_output() (vllm/v1/outputs.py:231-281; vllm/v1/core/sched/scheduler.py:1464-1477).
• The dispatcher is the source of truth: it pads to a captured descriptor, tries FULL, then PIECEWISE, and returns NONE when no graph key matches (vllm/v1/cudagraph_dispatcher.py:235-324; the design explanation is at docs/design/cuda_graphs.md:81-120). At serving time, FULL and PIECEWISE dispatch to a captured wrapper; NONE calls the runnable directly (vllm/compilation/cuda_graph.py:233-360).

Thus runtime_mode != NONE is a graph-path dispatch, and num_padded_tokens is the selected token bucket. PIECEWISE means graph-wrapped compiled pieces, not a full-step graph hit. Startup normally captures all dispatcher descriptors (vllm/v1/worker/gpu_model_runner.py:6595-6643), but CUDAGraphStat has no capture-versus-replay bit; a lazy first capture and a replay have the same stat. With built-in cudagraph_metrics, the stat is aggregated into a frequency table and printed at the logging interval (vllm/config/observability.py:56-58; vllm/compilation/cuda_graph.py:40-124); it is not exposed as a raw per-step Prometheus event. OpProf only needs to preserve the already-computed object on every enabled step.

Capture modes and sizes live in CompilationConfig: cudagraph_mode, cudagraph_capture_sizes, and max_cudagraph_capture_size are defined at vllm/config/compilation.py:587-688. The V1 default is FULL_AND_PIECEWISE. Final sizes are generated or validated in VllmConfig._set_cudagraph_sizes(); the default is 1/2/4, then steps of 8 below 256 and 16 above it, bounded by token/batch configuration, with user overrides written back to the final list (vllm/config/vllm.py:1635-1800).

2.4 MoE routing and Qwen3-30B-A3B backend

Qwen3 MoE creates a replicated gate and FusedMoE with the model's number of experts and top-k, and normally runs its router internally (vllm/model_executor/models/qwen3_moe.py:137-249). The causal-LM wrapper retains the list of MoE layers and their expert topology (vllm/model_executor/models/qwen3_moe.py:662-729). Router top-k IDs are available immediately after routing and before EPLB remapping (vllm/model_executor/layers/fused_moe/router/base_router.py:233-278).

There is an exact, built-in access path: --enable-return-routed-experts (vllm/config/model.py:209-215; vllm/engine/arg_utils.py:826-830). It installs per-layer callbacks, writes top-k IDs to preallocated GPU buffers, copies the step to pinned CPU memory, and returns it through ModelRunnerOutput (vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58-84; vllm/v1/worker/gpu_model_runner.py:3652-3666,4637-4683,7382-7437). This is reachable without modifying the fused kernels or saving router logits. Raw router-logit tensors are not plumbed to engine outputs; exposing those would be deeper and much higher-volume surgery than using the existing top-k ID callback.

It is not cheap enough for Layer 1. The per-worker transit buffer is a few MB; the scheduler's whole-slot buffer can reach multiple GB and performs a CPU fancy-index write each step (vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:223-305; the scheduler store is at vllm/v1/core/sched/scheduler.py:1501-1520). It is also incompatible with pipeline parallelism greater than one and with KV connectors (vllm/config/vllm.py:871-893). Exact per-layer expert load should therefore be collected in a separate sampled Layer-2 run and reduced offline. EPLB has load state, but enabling it adds communication and its logs are aggregate balancing summaries; it is not a free observability tap (vllm/distributed/eplb/eplb_state.py:62-171,478-575).

For the normal unquantized BF16/FP16 Qwen3-30B-A3B configuration on H20/SM90, the automatic unquantized oracle explicitly moves both FlashInfer backends behind Triton on Hopper because they are expected to be slower (vllm/model_executor/layers/fused_moe/oracle/unquantized.py:43-86). It selects the first supported backend and documents possible shape-specific fallbacks (vllm/model_executor/layers/fused_moe/oracle/unquantized.py:152-250). The primary backend is therefore Triton under those assumptions. A quantized checkpoint, LoRA, expert activation format, or explicit --moe-backend can change that conclusion and must be recorded with the campaign run.

2.5 V1 execution loop and natural hooks

The control path is:

EngineCore busy loop
  -> Scheduler.schedule()
  -> executor.execute_model(SchedulerOutput)
  -> GPUWorker -> GPUModelRunner.execute_model()
  -> Scheduler.update_from_output(SchedulerOutput, ModelRunnerOutput)

Evidence and hook implications:

• EngineCore.run_busy_loop() and _process_engine_step() drive steps at vllm/v1/engine/core.py:1259-1317.
• The normal step retains the exact scheduler output across the future and passes it back to update_from_output() (vllm/v1/engine/core.py:479-508). The batch-queue path does the same for multiple in-flight batches (vllm/v1/engine/core.py:519-632).
• The scheduler's stable interface is schedule/update at vllm/v1/core/sched/interface.py:51-107. The base implementation builds the output at vllm/v1/core/sched/scheduler.py:1012-1100, mutates request state immediately afterward at lines 1130-1155, and consumes model results at lines 1464-1803.
• Worker execution wraps gpu_model_runner.execute_model() at vllm/v1/worker/gpu_worker.py:895-898; the model runner's entry is vllm/v1/worker/gpu_model_runner.py:4055-4069.

The natural Layer-1 hooks are therefore in the base scheduler: snapshot after the SchedulerOutput is complete but before request state advances, then finalize with the matching model output. This covers the async subclass and avoids frontend/output aggregation. The only worker change is broadening the existing CUDA-graph-stat condition. Layer 2 uses the existing profile control path unchanged.

2.6 Relevant changes from 0.20.0

For comparison only, tag v0.20.0 was fetched into the allowed clone without changing the pinned working tree. It resolves to commit 88d34c6409e9fb3c7b8ca0c04756f061d2099eb1.

• Async scheduling and the batch-queue architecture already existed in 0.20.0; SchedulerConfig selected AsyncScheduler at v0.20.0:vllm/config/scheduler.py:146-176, and the core schedule/future/update paths were at v0.20.0:vllm/v1/engine/core.py:402-431,443-533. It is incorrect to treat async scheduling as new in 0.24.0.
• 0.24.0 adds DP prefill cadence/throttling and passes a prefill-throttle flag into schedule() (vllm/config/scheduler.py:140-161 and vllm/v1/engine/core.py:474-490). It also hardens zero-token/dummy-iteration handling (vllm/v1/engine/core.py:433-472) and changes when the batch queue fills.
• The 0.24 scheduler contains explicit scheduled/processed sequence fences and snapshots state that must survive later async schedules, including routed experts (vllm/v1/core/sched/scheduler.py:293-322,1093-1165). That is direct evidence that an OpProf hook must not read mutable request/queue state only when an older output returns.
• vllm/config/profiler.py, vllm/profiler/wrapper.py, and vllm/entrypoints/serve/profile/api_router.py are byte-identical between the two tags (git diff --quiet v0.20.0..v0.24.0 returned success). Profiler endpoint semantics did not move; the scheduler/core line numbers and safe composition hook did.

Practical conclusion: do not port a 0.20 line-number patch into EngineCore. Hook the 0.24 base scheduler's schedule/update pair so sync, async, and batch queue paths share one implementation.

3. Proposed dual-layer design

Layer 1 emits one schema-versioned JSONL record per scheduler step from the EngineCore/scheduler process. It contains step/timestamps; scheduled and decode batch counts; scheduled prefill/decode tokens; aggregate first/middle/final chunk information; fixed-bucket context and chunk-size histograms; exact step preemptions; schedule-time running/waiting/deferred queues; total/free/used KV blocks and ratio; local/external prefix deltas; and CUDA-graph mode/bucket. No request IDs or raw context-length lists are written. MoE expert load is null and explicitly marked Layer-2-only.

Records are encoded with the existing msgspec dependency (requirements/common.txt:35), placed into a bounded non-blocking queue, and drained by one background JSONL writer. It flushes every 1 MiB or one second and at clean shutdown, with explicit drop/gap counters and no per-step fsync. VLLM_OPPROF_DIR is the startup-only switch; unset is a true no-op. The unmeasured foreground estimate is 25-100 microseconds per step, so Phase 2 must enforce the less-than-3% overhead gate with paired off/on serving runs.

Layer 2 configures the already-built torch profiler and samples short windows, initially two warm-up plus eight active worker iterations. Set max_iterations=8; the wrapper stops the underlying profiler on the subsequent step after the counter exceeds the limit, as its existing test documents (vllm/profiler/wrapper.py:83-114; tests/v1/worker/test_gpu_profiler.py:77-98). The controller then calls /stop_profile to clear control state. Step-count/on-demand endpoint control is preferred to wall time. Layer 2 writes one trace per TP worker and joins to Layer 1 through the existing iteration annotation, order, token counts, and timestamps.

Under CUDA-graph replay, the profiler is configured for CUDA activity, but the Python callable is not re-executed: vLLM calls CUDAGraph.replay() (vllm/compilation/cuda_graph.py:233-360). CUDA activity may therefore be visible through Kineto/CUPTI, while Python/PyTorch scopes cannot be assumed to provide the original per-layer attribution. Layerwise NVTX is explicitly unsupported with graphs (vllm/config/observability.py:60-63). Phase 2 must check whether the pinned torch/CUPTI combination exposes individual replayed kernels or a coarser graph launch. A matched eager window can identify kernel families but must not replace graph-enabled production timing.

Layer-2 kernel time is grouped into attention, router/top-k, expert/dense GEMMs, normalization/activation, collectives, and sampling. Per-rank critical-path times calibrate a composition/graph-bucket-conditioned Layer-1 attribution; held-out profiler windows validate that mapping. Exact routed experts are enabled only in a separate sampled server run.

Patch surface

File	Approximate lines
vllm/envs.py	4
vllm/v1/opprof.py (new)	220
vllm/v1/core/sched/scheduler.py	32
vllm/v1/worker/gpu_model_runner.py	4
tests/v1/core/test_opprof.py (new)	190
Total	5 files / about 450 lines

Zero-patch wins: profiler config/endpoints/wrapper, SchedulerOutput, existing metrics plumbing, CUDA kernels, fused MoE/router code, Qwen3 model code, and frontend APIs. Supporting OpProf together with --disable-log-stats would add about 12 lines in vllm/v1/core/kv_cache_manager.py; the design recommends failing fast instead.

Main risks

• Histogram construction and JSON encoding are the likely foreground overhead; the 3% budget must be measured, not inferred.
• Async scheduling can misassociate latest queue/cache state with an older output; snapshot inside schedule() and pair by the exact output object.
• Layer 1 must be scheduler-owned to avoid TP duplicates; Layer 2 remains per-rank and needs offline aggregation.
• At 500 steps/s and about 1 KiB/record, an uncompressed stream is roughly 44 GB/day per EngineCore. Runs must be duration-bounded, with queue/drop counters and post-run compression.
• Kineto perturbs execution, and graph replay loses Python-scope attribution; it is a sampled calibration instrument, not an always-on latency source.

4. Open design decisions

1. Approve JSONL/msgspec, an 8192-record queue, context edges (128, 256, 512, 1K, 2K, 4K, 8K, 16K, 32K, 64K, 128K, +inf), and chunk-size edges (16, 32, 64, 128, 256, 512, 1K, 2K, +inf).
2. Approve exact MoE expert load as Layer-2-only. The alternative is a new GPU histogram path, with a larger patch and an overhead/graph-capture risk.
3. Confirm checkpoint precision/quantization and intended TP/EP/DP topology. The H20 Triton conclusion assumes unquantized BF16/FP16, auto backend, and no LoRA.
4. Choose the initial profiler window: recommended two warm-up plus eight active iterations, or a longer active window.
5. Decide whether the less-than-3% gate applies to point estimates or the upper 95% confidence bound for every primary metric.
6. Confirm that OpProf campaign runs may reject --disable-log-stats; support for it increases the patch to six files and about 462 lines.

Data sanity block

• Pinned root listing: n=45 names; lexicographic min=.buildkite, max=vllm; distinct=45; SHA-256=75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58.
• Source versions compared: n=2 tags; min=v0.20.0, max=v0.24.0; distinct=2; pinned working-tree version remains exactly v0.24.0.
• Torch pin observations: n=4 declarations (pyproject, build requirements, CUDA runtime requirements, CMake); min=max=2.11.0; distinct=1.
• Evidence-bearing source files inspected and cited: n=44 distinct v0.24.0 paths; min/max not applicable to categorical paths; v0.20.0 comparison touched n=5 distinct paths; min/max not applicable.
• Invariants checked: exact tag equals v0.24.0; HEAD equals the recorded commit; clone is clean; top-level names are unique; requested path was used; torch pins agree; CUDA-graph modes are in {NONE, PIECEWISE, FULL}; KV usage definition is bounded in [0,1]; scheduled-token total is defined as the sum of per-request scheduled tokens; graph bucket is not smaller than unpadded tokens; no remote/GPU/install/model-download action occurred.
• Benchmark/statistical conclusion check: no performance benchmark was run, so there are no per-configuration curves or measured overhead values on which to perform identical-value, monotonicity, or continuity checks. All overhead numbers in the design are labeled estimates.