aituner/patches/vllm-0.24.0-opprof/README.md

vLLM 0.24.0 OpProf patch series

Goal

Apply the accepted OpProf Layer-1 instrumentation to exactly vLLM v0.24.0 at base commit ee0da84ab9e04ac7610e28580af62c365e898389. The series adds one scheduler-owned composition record per step without installing new runtime dependencies or changing GPU kernels.

Contents

• 0001-Add-lightweight-per-step-OpProf-telemetry.patch: adds the environment switch, import-light JSONL recorder/writer, scheduler hooks, and reuse of the existing CUDA-graph stat. Writer failures are exposed without blocking producers or shutdown, and request histograms are accumulated in-place.
• 0002-Add-standalone-OpProf-telemetry-tests.patch: adds CPU-only tests that load the recorder directly without importing or installing vLLM or torch, including ENOSPC, golden-record, and zero-token regressions.
• 0003-Log-the-OpProf-output-path-at-startup.patch: logs the resolved JSONL output path and covers it in the standalone shutdown test.
• 0004-Exclude-OpProf-output-path-from-compile-cache-key.patch: prevents the per-run telemetry destination from invalidating vLLM's torch.compile/AOT cache and adds an import-light regression test.
• 0005-Keep-compile-factor-regression-import-light.patch: isolates the new regression from torch in full vLLM test environments.
• 0006-Checkpoint-OpProf-accounting-across-hard-kills.patch: atomically checkpoints balanced writer counters beside each JSONL stream once per flush interval, with clean-close and hard-kill regressions.
• 0007-Recreate-scheduled-torch-profiler-between-windows.patch: discards a stopped scheduled torch-profiler wrapper so each subsequent official profile endpoint call receives a fresh 2+8 schedule and emits its own trace.
• apply.sh: verifies the exact base, refuses dirty/wrong revisions, applies all numbered patches with git am, and exits successfully only when the exact series is already applied directly on the required base.
• pytest-evidence.txt: exact isolated test command, dependency versions, and all-pass summary.

The source branch tip used to generate the patches is 23450fb21ac255b0cf710f4ee965ee694921975d (opprof).

Apply

Prerequisite: a clean checkout whose HEAD is the exact base commit.

./patches/vllm-0.24.0-opprof/apply.sh /path/to/vllm-v0.24.0

Running the command again is a no-op only when the five matching patch commits are rooted directly at the required base. A partially applied series, dirty tree, unrelated commit, or any other HEAD is rejected instead of being guessed around.

Enable and output

Set an absolute output directory before starting vLLM:

export VLLM_OPPROF_DIR=/absolute/path/to/run/opprof

Unset or empty disables the feature before recorder construction. Combining it with --disable-log-stats fails fast, as approved.

Each EngineCore/DP scheduler writes one file named approximately opprof-v1-dp0-pid1234-<start_ns>.jsonl. Records contain schema/engine/step and timestamps; scheduled prefill/decode composition; first/middle/final/unsplit prefill chunks; 12-bin context and 9-bin chunk-size histograms; preemptions; running/waiting/deferred queues; KV blocks/usage; local/external prefix deltas; CUDA-graph hit/mode/bucket/padding; explicit null Layer-1 MoE load; and drop-gap accounting. A clean close writes a final writer-count footer in the stream.

Every JSONL flush also atomically replaces <stream>.footer.json through a same-directory temporary file. The sidecar contains the encoded, written, and dropped counts through that durable flush, the last written step index, a wall-clock timestamp, the one-second flush interval, and whether it is final. Queue entries carry their submission ordinal and cumulative drops, so a periodic checkpoint always satisfies encoded = written + dropped without decoding records in the writer thread. On clean close the in-stream footer is authoritative and the final sidecar must agree with all three counters. If a hard kill prevents the in-stream footer, the latest sidecar is authoritative: the decoded data-line count must equal its written_records, its final data-line step must equal last_step_index, and its counters must balance. Data after that checkpoint may be lost, bounded by at most the configured one-second flush interval.

The bounded queue holds 8192 encoded records. Producers never wait for disk; full queues or a failed writer drop the new record and report the gap on the next successful record. A writer I/O failure is exposed through recorder state, logged once, and cannot make shutdown wait indefinitely. The writer flushes at 1 MiB, one second, or shutdown.

Test

Only pytest and msgspec are required. --confcutdir prevents vLLM's global test configuration from importing its full dependency stack.

cd /path/to/vllm-v0.24.0
uv run --no-project --with pytest --with msgspec \
  pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q

Expected summary:

18 passed in 1.09s

Caveats

• Layer 1 intentionally records no expert-load arrays. Exact routed experts remain a separate Layer-2 run.
• PIECEWISE means graph-wrapped compiled regions, not full-step graph replay.
• Phase 2 must measure the always-on overhead; acceptance requires the upper bound of the 95% confidence interval to remain below 3% for every primary serving metric.
• Primary campaign topology is TP1 on community BF16 Qwen3-30B-A3B, with TP2 and TP4 counterpoints. Record the selected MoE backend log every run.