aituner/docs/opprof/phase0-recon-vllm-0.24.0.md

# Phase 0 recon: vLLM 0.24.0 observability

Status: source recon and patch design complete; **no instrumentation patch was
implemented**.

Campaign target: operator-level, pattern-conditioned profiling of
Qwen3-30B-A3B serving on NVIDIA H20. This report is based on the pinned source
clone only. No GPU was used, no package or model was installed, and no dash
host was contacted.

Unless explicitly prefixed with `v0.20.0:`, every source path and line number
below is relative to `/home/gahow/phd/vllm-v0.24.0` at the pinned commit.

## Outcome

vLLM 0.24.0 already contains most of the raw ingredients, but they are split
across scheduler internals, aggregate metrics, worker traces, and an optional
routed-expert return path. The smallest useful patch is therefore not a new
profiler. It is one scheduler-owned per-step record that joins existing batch
composition to the existing `CUDAGraphStat`; short torch-profiler windows and
exact expert routes remain sampled Layer 2 facilities.

The proposed patch is detailed in
[`patch-design.md`](patch-design.md). Estimated surface: five files and about
450 lines including tests, with about 260 production lines.

## 1. Source pin and compatibility record

The requested tag existed and the exact requested clone command completed
successfully:

```text
git clone --depth 1 --branch v0.24.0 https://github.com/vllm-project/vllm /home/gahow/phd/vllm-v0.24.0
```

| Field | Pinned value |
|---|---|
| Repository | `https://github.com/vllm-project/vllm` |
| Local clone | `/home/gahow/phd/vllm-v0.24.0` |
| Tag | `v0.24.0` (`git describe --tags --exact-match`) |
| Commit | `ee0da84ab9e04ac7610e28580af62c365e898389` |
| Clone completion | `2026-07-11T08:26:16Z` (`2026-07-11 16:26:16 +08:00`) |
| Checkout | Detached HEAD, clean |
| Top-level entries | 45, including hidden entries and `.git` |
| Listing SHA-256 | `75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58` |

The listing digest is reproducible from entry names only, not file contents:

```text
LC_ALL=C find . -mindepth 1 -maxdepth 1 -printf '%f\n' |
  LC_ALL=C sort | sha256sum
```

This definition includes `.git`, sorts bytewise under the C locale, and hashes
the 45 LF-terminated root names. It is recorded to make the otherwise ambiguous
phrase “sha256 of the top-level directory listing” precise.

### Torch and CUDA requirements

- Build and NVIDIA runtime requirements pin `torch==2.11.0` in
  `pyproject.toml:1-13`, `requirements/build/cuda.txt:1-10`, and
  `requirements/cuda.txt:1-10`; CMake repeats 2.11.0 as the supported CUDA and
  ROCm torch version at `CMakeLists.txt:62-72`.
- The NVIDIA requirements also pin `torchvision==0.26.0`,
  `flashinfer-python==0.6.12`, `flashinfer-cubin==0.6.12`, and CUDA-13 extras
  for CUTLASS DSL and Humming kernels (`requirements/cuda.txt:6-29`). Setup
  strips/substitutes those extras for CUDA 12 builds
  (`setup.py:1066-1083`).
- The source default is CUDA 13.0 (`vllm/envs.py:85-87,555-563`) and the Docker
  default is CUDA 13.0.2 (`docker/Dockerfile:14-42`). Wheel detection maps CUDA
  major 12 to `cu129` and major 13 to `cu130` (`setup.py:528-568`).
- The pinned installation document still describes the default precompiled
  binaries as CUDA 12.9 and lists CUDA 12.8 and 13.0 variants
  (`docs/getting_started/installation/gpu.cuda.inc.md:4-19,24-54`). This is a
  packaging/default distinction, not evidence that 13.0 is unsupported.
- NVIDIA compute capability 7.5+ is documented, and SM90 is in every relevant
  CMake CUDA architecture set (`docs/getting_started/installation/gpu.cuda.inc.md:9-19`;
  `CMakeLists.txt:105-118`). Optional FA3 and DeepGEMM builds need CUDA 12.3+,
  FlashMLA needs 12.9+, and the Hopper CUTLASS MoE build needs 12.3+
  (`setup.py:1111-1139`; `CMakeLists.txt:844-868`).

Implication for dash0 later: H20/SM90 is a supported architecture, but the
actual driver and installed CUDA were intentionally not queried in Phase 0.
Phase 2 must choose a torch 2.11.0-compatible `cu129` or `cu130` artifact after
checking the host driver. A wheel built against a different torch/CUDA build is
not interchangeable; the source documentation explicitly warns about binary
incompatibility (`docs/getting_started/installation/gpu.cuda.inc.md:14-19`).

## 2. Built-in observability inventory

### 2.1 Torch profiler integration

**Configuration and control.** v0.24.0 does not contain the legacy
`VLLM_TORCH_PROFILER_DIR` symbol: a repository-wide exact-name search at the
pinned commit returned zero matches. Profiling is now configured through
`ProfilerConfig`: `profiler` is `torch` or `cuda`, and
`torch_profiler_dir` holds the absolute/URI output location
(`vllm/config/profiler.py:33-46,125-145`). Optional controls include stacks,
FLOPs, gzip, CUDA-time tables, shapes, memory, frontend exclusion, delayed
start, maximum iterations, and wait/warmup/active iteration scheduling
(`vllm/config/profiler.py:48-105`).

The documented server form is a `--profiler-config` JSON object, for example
`{"profiler":"torch","torch_profiler_dir":"/abs/run/kineto"}`, followed by
POSTs to the two endpoints (`docs/contributing/profiling.md:46-85`).

When profiler config is present, the serving router exposes POST
`/start_profile` and `/stop_profile`; without profiler config the router is not
attached (`vllm/entrypoints/serve/profile/api_router.py:21-45`). The endpoints
are therefore not an unauthenticated always-on feature independent of server
configuration; they exist within the configured serving application and should
be protected like the rest of the control plane.

**Captured activity.** Each GPU worker creates a torch profiler with both CPU
and CUDA activities, using the configured shapes/memory/stack/FLOPs switches
(`vllm/v1/worker/gpu_worker.py:929-980` and
`vllm/profiler/wrapper.py:159-226`). TensorBoard-compatible traces are written
with an optional gzip handler. The wrapper also writes CPU/CUDA aggregate
tables; only rank 0 prints the table to stdout
(`vllm/profiler/wrapper.py:251-287`). The source documentation calls the
profiler medium-overhead and warns that trace size and final flushing can be
large (`docs/contributing/profiling.md:1-38`).

The V1 worker calls `profiler.step()` once per execution and wraps model
execution in a record-function name containing context/generation request and
token counts (`vllm/v1/worker/gpu_worker.py:803-827,895-898`). If frontend
profiling is not ignored, `AsyncLLM` also creates a separate CPU-only trace
(`vllm/v1/engine/async_llm.py:178-200`).

**TP behavior.** `EngineCore.profile()` delegates to the executor
(`vllm/v1/engine/core.py:662-663`), whose profile RPC is a collective worker
call (`vllm/v1/executor/abstract.py:256-257`). Multiprocessing broadcasts calls
that do not declare a unique output rank, so the profile call reaches every TP
worker (`vllm/v1/executor/multiproc_executor.py:340-402`). Each worker emits its
own rank-qualified trace; the suffix includes DP/PP/TP/DCP/EP/global rank
components (`vllm/distributed/utils.py:664-695`). For TP, kernel analysis must
therefore preserve per-rank files, use the slowest rank for critical-path time,
and optionally sum ranks to quantify total GPU work.

### 2.2 Iteration-level scheduler data and exported metrics

The V1 scheduler interface states that each `schedule()` output corresponds to
one model forward and maps request IDs to scheduled tokens
(`vllm/v1/core/sched/interface.py:51-80`). `SchedulerOutput` already carries
new/cached request data, per-request and total scheduled tokens, common-prefix
blocks, and the request IDs preempted in that step
(`vllm/v1/core/sched/output.py:180-219`). The base scheduler constructs this
object at `vllm/v1/core/sched/scheduler.py:1012-1076`, then advances request
computed-token and chunk state at `vllm/v1/core/sched/scheduler.py:1130-1155`.

What is available today:

| Requested datum | Internal per-step source | Externally exposed today |
|---|---|---|
| Batch size | `len(SchedulerOutput.num_scheduled_tokens)`; the map is exact | No raw per-step batch-size Prometheus series. Optional iteration log reports context and generation request counts. |
| Scheduled prefill/decode tokens | `compute_iteration_details()` classifies new/cached context requests and sums scheduled tokens (`vllm/v1/utils.py:766-813`) | Optional log reports exact scheduled context/generation counts and elapsed time around the execution wait (`vllm/v1/engine/core.py:433-472`). Prometheus prompt/generation counters are cumulative output-side stats, not a raw scheduled-step stream. |
| Chunked-prefill splits | `Request.is_prefill_chunk`, `num_computed_tokens`, and prompt/output state exist (`vllm/v1/request.py:130-175`); extended chunks are classified as context | No chunk ordinal, first/middle/final split, or chunk-size series. These must be derived before `_update_after_schedule()` mutates state. |
| Preemptions | Exact step set in `SchedulerOutput.preempted_req_ids` | Stdout aggregates over its logging interval; `vllm:num_preemptions` is cumulative (`vllm/v1/metrics/loggers.py:219-281,624-631`). |
| Running/waiting/deferred queues | `SchedulerStats.num_running_reqs`, `num_waiting_reqs`, and `num_skipped_waiting_reqs` (`vllm/v1/metrics/stats.py:170-183`) | Latest-value gauges `vllm:num_requests_running`, `vllm:num_requests_waiting`, and reason-labeled capacity/deferred gauges (`vllm/v1/metrics/loggers.py:453-496`). Running queue size is not the same as scheduled batch size. |
| KV-cache usage | Scheduler reads `kv_cache_manager.usage`; the block pool computes a null-block-adjusted ratio in `[0,1]` (`vllm/v1/core/kv_cache_manager.py:181-188`; `vllm/v1/core/block_pool.py:692-711`) | Latest-value `vllm:kv_cache_usage_perc`; stdout prints the latest percent (`vllm/v1/metrics/loggers.py:250-260,524-532`). Raw total/free blocks are not exported per step. |
| Prefix-cache counters | `PrefixCacheStats` contains request/query/hit and separately preempted request/query/hit counters (`vllm/v1/metrics/stats.py:114-143`) for local and optional connector caches | Prometheus exposes cumulative token queries/hits for local and external caches (`vllm/v1/metrics/loggers.py:547-565,1063-1101`); stdout reports interval-derived hit rates. It does not expose the preempted sub-counters separately. |
| Total step tokens | Exact scheduled total is in `SchedulerOutput` | `vllm:iteration_tokens_total` is a histogram observed from output-side computed prompt plus generation tokens (`vllm/v1/metrics/loggers.py:693-724,1148-1171`), not a joinable raw step record. |

`IterationStats` also has a wall timestamp, output-side prompt/generation data,
preemptions, and finished-request latency samples
(`vllm/v1/metrics/stats.py:325-347`). `Scheduler.make_stats()` produces latest
queue/KV state, drains prefix stats, and passes through the step's CUDA-graph
stat when an output is processed
(`vllm/v1/core/sched/scheduler.py:2228-2264`). These are valuable existing
plumbing, but async scheduling can schedule newer batches before an older
output is processed. Consequently, those latest/drained values are not a safe
per-step composition join without a schedule-time snapshot.

Bottom line: no new scheduler instrumentation is needed to discover batch and
token composition, preemptions, queues, KV use, or prefix events. A patch is
needed only to serialize their **same-step association** and the missing
histograms/chunk split.

### 2.3 CUDA-graph observability

Yes, v0.24.0 can determine the runtime graph mode and capture bucket for each
model step internally:

- `CUDAGraphStat` records unpadded tokens, padded tokens, padding, and runtime
  mode (`vllm/compilation/cuda_graph.py:32-37`).
- The GPU model runner asks the dispatcher for a concrete runtime mode and
  `BatchDescriptor`, then conditionally attaches the stat to its output
  (`vllm/v1/worker/gpu_model_runner.py:3822-3934`). It reaches the scheduler in
  `ModelRunnerOutput` and `update_from_output()`
  (`vllm/v1/outputs.py:231-281`;
  `vllm/v1/core/sched/scheduler.py:1464-1477`).
- The dispatcher is the source of truth: it pads to a captured descriptor,
  tries `FULL`, then `PIECEWISE`, and returns `NONE` when no graph key matches
  (`vllm/v1/cudagraph_dispatcher.py:235-324`; the design explanation is at
  `docs/design/cuda_graphs.md:81-120`). At serving time, `FULL` and `PIECEWISE`
  dispatch to a captured wrapper; `NONE` calls the runnable directly
  (`vllm/compilation/cuda_graph.py:233-360`).

Thus `runtime_mode != NONE` is a graph-path dispatch, and `num_padded_tokens` is
the selected token bucket. `PIECEWISE` means graph-wrapped compiled pieces, not
a full-step graph hit. Startup normally captures all dispatcher descriptors
(`vllm/v1/worker/gpu_model_runner.py:6595-6643`), but `CUDAGraphStat` has no
capture-versus-replay bit; a lazy first capture and a replay have the same stat.
With built-in `cudagraph_metrics`, the stat is aggregated
into a frequency table and printed at the logging interval
(`vllm/config/observability.py:56-58`;
`vllm/compilation/cuda_graph.py:40-124`); it is not exposed as a raw per-step
Prometheus event. OpProf only needs to preserve the already-computed object on
every enabled step.

Capture modes and sizes live in `CompilationConfig`:
`cudagraph_mode`, `cudagraph_capture_sizes`, and
`max_cudagraph_capture_size` are defined at
`vllm/config/compilation.py:587-688`. The V1 default is
`FULL_AND_PIECEWISE`. Final sizes are generated or validated in
`VllmConfig._set_cudagraph_sizes()`; the default is 1/2/4, then steps of 8 below
256 and 16 above it, bounded by token/batch configuration, with user overrides
written back to the final list (`vllm/config/vllm.py:1635-1800`).

### 2.4 MoE routing and Qwen3-30B-A3B backend

Qwen3 MoE creates a replicated gate and `FusedMoE` with the model's number of
experts and top-k, and normally runs its router internally
(`vllm/model_executor/models/qwen3_moe.py:137-249`). The causal-LM wrapper
retains the list of MoE layers and their expert topology
(`vllm/model_executor/models/qwen3_moe.py:662-729`). Router top-k IDs are
available immediately after routing and before EPLB remapping
(`vllm/model_executor/layers/fused_moe/router/base_router.py:233-278`).

There is an exact, built-in access path: `--enable-return-routed-experts`
(`vllm/config/model.py:209-215`; `vllm/engine/arg_utils.py:826-830`). It installs
per-layer callbacks, writes top-k IDs to preallocated GPU buffers, copies the
step to pinned CPU memory, and returns it through `ModelRunnerOutput`
(`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58-84`;
`vllm/v1/worker/gpu_model_runner.py:3652-3666,4637-4683,7382-7437`). This is
reachable without modifying the fused kernels or saving router logits.
Raw router-logit tensors are not plumbed to engine outputs; exposing those
would be deeper and much higher-volume surgery than using the existing top-k
ID callback.

It is not cheap enough for Layer 1. The per-worker transit buffer is a few MB;
the scheduler's whole-slot buffer can reach multiple GB and performs a CPU
fancy-index write each step
(`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:223-305`;
the scheduler store is at `vllm/v1/core/sched/scheduler.py:1501-1520`). It is
also incompatible with pipeline parallelism greater than one and with KV
connectors (`vllm/config/vllm.py:871-893`). Exact per-layer expert load should
therefore be collected in a separate sampled Layer-2 run and reduced offline.
EPLB has load state, but enabling it adds communication and its logs are
aggregate balancing summaries; it is not a free observability tap
(`vllm/distributed/eplb/eplb_state.py:62-171,478-575`).

For the normal unquantized BF16/FP16 Qwen3-30B-A3B configuration on H20/SM90,
the automatic unquantized oracle explicitly moves both FlashInfer backends
behind Triton on Hopper because they are expected to be slower
(`vllm/model_executor/layers/fused_moe/oracle/unquantized.py:43-86`). It selects
the first supported backend and documents possible shape-specific fallbacks
(`vllm/model_executor/layers/fused_moe/oracle/unquantized.py:152-250`). The
primary backend is therefore **Triton** under those assumptions. A quantized
checkpoint, LoRA, expert activation format, or explicit `--moe-backend` can
change that conclusion and must be recorded with the campaign run.

### 2.5 V1 execution loop and natural hooks

The control path is:

```text
EngineCore busy loop
  -> Scheduler.schedule()
  -> executor.execute_model(SchedulerOutput)
  -> GPUWorker -> GPUModelRunner.execute_model()
  -> Scheduler.update_from_output(SchedulerOutput, ModelRunnerOutput)
```

Evidence and hook implications:

- `EngineCore.run_busy_loop()` and `_process_engine_step()` drive steps at
  `vllm/v1/engine/core.py:1259-1317`.
- The normal step retains the exact scheduler output across the future and
  passes it back to `update_from_output()`
  (`vllm/v1/engine/core.py:479-508`). The batch-queue path does the same for
  multiple in-flight batches (`vllm/v1/engine/core.py:519-632`).
- The scheduler's stable interface is schedule/update at
  `vllm/v1/core/sched/interface.py:51-107`. The base implementation builds the
  output at `vllm/v1/core/sched/scheduler.py:1012-1100`, mutates request state
  immediately afterward at lines 1130-1155, and consumes model results at
  lines 1464-1803.
- Worker execution wraps `gpu_model_runner.execute_model()` at
  `vllm/v1/worker/gpu_worker.py:895-898`; the model runner's entry is
  `vllm/v1/worker/gpu_model_runner.py:4055-4069`.

The natural Layer-1 hooks are therefore in the base scheduler: snapshot after
the `SchedulerOutput` is complete but before request state advances, then
finalize with the matching model output. This covers the async subclass and
avoids frontend/output aggregation. The only worker change is broadening the
existing CUDA-graph-stat condition. Layer 2 uses the existing profile control
path unchanged.

### 2.6 Relevant changes from 0.20.0

For comparison only, tag `v0.20.0` was fetched into the allowed clone without
changing the pinned working tree. It resolves to commit
`88d34c6409e9fb3c7b8ca0c04756f061d2099eb1`.

- Async scheduling and the batch-queue architecture already existed in 0.20.0;
  `SchedulerConfig` selected `AsyncScheduler` at
  `v0.20.0:vllm/config/scheduler.py:146-176`, and the core schedule/future/update
  paths were at `v0.20.0:vllm/v1/engine/core.py:402-431,443-533`. It is
  incorrect to treat async scheduling as new in 0.24.0.
- 0.24.0 adds DP prefill cadence/throttling and passes a prefill-throttle flag
  into `schedule()` (`vllm/config/scheduler.py:140-161` and
  `vllm/v1/engine/core.py:474-490`). It also hardens zero-token/dummy-iteration
  handling (`vllm/v1/engine/core.py:433-472`) and changes when the batch queue
  fills.
- The 0.24 scheduler contains explicit scheduled/processed sequence fences and
  snapshots state that must survive later async schedules, including routed
  experts (`vllm/v1/core/sched/scheduler.py:293-322,1093-1165`). That is direct
  evidence that an OpProf hook must not read mutable request/queue state only
  when an older output returns.
- `vllm/config/profiler.py`, `vllm/profiler/wrapper.py`, and
  `vllm/entrypoints/serve/profile/api_router.py` are byte-identical between the
  two tags (`git diff --quiet v0.20.0..v0.24.0` returned success). Profiler
  endpoint semantics did not move; the scheduler/core line numbers and safe
  composition hook did.

Practical conclusion: do not port a 0.20 line-number patch into EngineCore.
Hook the 0.24 base scheduler's schedule/update pair so sync, async, and batch
queue paths share one implementation.

## 3. Proposed dual-layer design

Layer 1 emits one schema-versioned JSONL record per scheduler step from the
EngineCore/scheduler process. It contains step/timestamps; scheduled and decode
batch counts; scheduled prefill/decode tokens; aggregate first/middle/final
chunk information; fixed-bucket context and chunk-size histograms; exact step
preemptions; schedule-time running/waiting/deferred queues; total/free/used KV
blocks and ratio; local/external prefix deltas; and CUDA-graph mode/bucket. No
request IDs or raw context-length lists are written. MoE expert load is null and
explicitly marked Layer-2-only.

Records are encoded with the existing `msgspec` dependency
(`requirements/common.txt:35`), placed into a
bounded non-blocking queue, and drained by one background JSONL writer. It
flushes every 1 MiB or one second and at clean shutdown, with explicit drop/gap
counters and no per-step `fsync`. `VLLM_OPPROF_DIR` is the startup-only switch;
unset is a true no-op. The unmeasured foreground estimate is 25-100
microseconds per step, so Phase 2 must enforce the less-than-3% overhead gate
with paired off/on serving runs.

Layer 2 configures the already-built torch profiler and samples short windows,
initially two warm-up plus eight active worker iterations. Set
`max_iterations=8`; the wrapper stops the underlying profiler on the subsequent
step after the counter exceeds the limit, as its existing test documents
(`vllm/profiler/wrapper.py:83-114`;
`tests/v1/worker/test_gpu_profiler.py:77-98`). The controller then calls
`/stop_profile` to clear control state. Step-count/on-demand endpoint control is
preferred to wall time. Layer 2 writes one trace per TP worker and joins to
Layer 1 through the existing iteration annotation, order, token counts, and
timestamps.

Under CUDA-graph replay, the profiler is configured for CUDA activity, but the
Python callable is not re-executed: vLLM calls `CUDAGraph.replay()`
(`vllm/compilation/cuda_graph.py:233-360`). CUDA activity may therefore be
visible through Kineto/CUPTI, while Python/PyTorch scopes cannot be assumed to
provide the original per-layer attribution. Layerwise NVTX is explicitly
unsupported with graphs (`vllm/config/observability.py:60-63`). Phase 2 must
check whether the pinned torch/CUPTI combination exposes individual replayed
kernels or a coarser graph launch. A matched eager window can identify kernel
families but must not replace graph-enabled production timing.

Layer-2 kernel time is grouped into attention, router/top-k, expert/dense GEMMs,
normalization/activation, collectives, and sampling. Per-rank critical-path
times calibrate a composition/graph-bucket-conditioned Layer-1 attribution;
held-out profiler windows validate that mapping. Exact routed experts are
enabled only in a separate sampled server run.

### Patch surface

| File | Approximate lines |
|---|---:|
| `vllm/envs.py` | 4 |
| `vllm/v1/opprof.py` (new) | 220 |
| `vllm/v1/core/sched/scheduler.py` | 32 |
| `vllm/v1/worker/gpu_model_runner.py` | 4 |
| `tests/v1/core/test_opprof.py` (new) | 190 |
| **Total** | **5 files / about 450 lines** |

Zero-patch wins: profiler config/endpoints/wrapper, `SchedulerOutput`, existing
metrics plumbing, CUDA kernels, fused MoE/router code, Qwen3 model code, and
frontend APIs. Supporting OpProf together with `--disable-log-stats` would add
about 12 lines in `vllm/v1/core/kv_cache_manager.py`; the design recommends
failing fast instead.

### Main risks

- Histogram construction and JSON encoding are the likely foreground overhead;
  the 3% budget must be measured, not inferred.
- Async scheduling can misassociate latest queue/cache state with an older
  output; snapshot inside `schedule()` and pair by the exact output object.
- Layer 1 must be scheduler-owned to avoid TP duplicates; Layer 2 remains
  per-rank and needs offline aggregation.
- At 500 steps/s and about 1 KiB/record, an uncompressed stream is roughly
  44 GB/day per EngineCore. Runs must be duration-bounded, with queue/drop
  counters and post-run compression.
- Kineto perturbs execution, and graph replay loses Python-scope attribution;
  it is a sampled calibration instrument, not an always-on latency source.

## 4. Open design decisions

1. Approve JSONL/`msgspec`, an 8192-record queue, context edges
   `(128, 256, 512, 1K, 2K, 4K, 8K, 16K, 32K, 64K, 128K, +inf)`, and chunk-size
   edges `(16, 32, 64, 128, 256, 512, 1K, 2K, +inf)`.
2. Approve exact MoE expert load as Layer-2-only. The alternative is a new GPU
   histogram path, with a larger patch and an overhead/graph-capture risk.
3. Confirm checkpoint precision/quantization and intended TP/EP/DP topology.
   The H20 Triton conclusion assumes unquantized BF16/FP16, auto backend, and no
   LoRA.
4. Choose the initial profiler window: recommended two warm-up plus eight
   active iterations, or a longer active window.
5. Decide whether the less-than-3% gate applies to point estimates or the upper
   95% confidence bound for every primary metric.
6. Confirm that OpProf campaign runs may reject `--disable-log-stats`; support
   for it increases the patch to six files and about 462 lines.

## Data sanity block

- **Pinned root listing:** n=45 names; lexicographic min=`.buildkite`,
  max=`vllm`; distinct=45; SHA-256=`75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58`.
- **Source versions compared:** n=2 tags; min=`v0.20.0`, max=`v0.24.0`;
  distinct=2; pinned working-tree version remains exactly `v0.24.0`.
- **Torch pin observations:** n=4 declarations
  (`pyproject`, build requirements, CUDA runtime requirements, CMake);
  min=max=`2.11.0`; distinct=1.
- **Evidence-bearing source files inspected and cited:** n=44
  distinct v0.24.0 paths; min/max not applicable to categorical paths;
  v0.20.0 comparison touched n=5 distinct paths; min/max not applicable.
- **Invariants checked:** exact tag equals `v0.24.0`; HEAD equals the recorded
  commit; clone is clean; top-level names are unique; requested path was used;
  torch pins agree; CUDA-graph modes are in `{NONE, PIECEWISE, FULL}`; KV usage
  definition is bounded in `[0,1]`; scheduled-token total is defined as the sum
  of per-request scheduled tokens; graph bucket is not smaller than unpadded
  tokens; no remote/GPU/install/model-download action occurred.
- **Benchmark/statistical conclusion check:** no performance benchmark was run,
  so there are no per-configuration curves or measured overhead values on which
  to perform identical-value, monotonicity, or continuity checks. All overhead
  numbers in the design are labeled estimates.