aituner/patches/vllm-0.24.0-opprof/README.md

# vLLM 0.24.0 OpProf patch series

## Goal

Apply the accepted OpProf Layer-1 instrumentation to exactly vLLM `v0.24.0`
at base commit `ee0da84ab9e04ac7610e28580af62c365e898389`. The series adds one
scheduler-owned composition record per step without installing new runtime
dependencies or changing GPU kernels.

## Contents

- `0001-Add-lightweight-per-step-OpProf-telemetry.patch`: adds the environment
  switch, import-light JSONL recorder/writer, scheduler hooks, and reuse of the
  existing CUDA-graph stat. Writer failures are exposed without blocking
  producers or shutdown, and request histograms are accumulated in-place.
- `0002-Add-standalone-OpProf-telemetry-tests.patch`: adds CPU-only tests that
  load the recorder directly without importing or installing vLLM or torch,
  including ENOSPC, golden-record, and zero-token regressions.
- `0003-Log-the-OpProf-output-path-at-startup.patch`: logs the resolved JSONL
  output path and covers it in the standalone shutdown test.
- `0004-Exclude-OpProf-output-path-from-compile-cache-key.patch`: prevents the
  per-run telemetry destination from invalidating vLLM's torch.compile/AOT
  cache and adds an import-light regression test.
- `0005-Keep-compile-factor-regression-import-light.patch`: isolates the new
  regression from torch in full vLLM test environments.
- `0006-Checkpoint-OpProf-accounting-across-hard-kills.patch`: atomically
  checkpoints balanced writer counters beside each JSONL stream once per
  flush interval, with clean-close and hard-kill regressions.
- `0007-Recreate-scheduled-torch-profiler-between-windows.patch`: discards a
  stopped scheduled torch-profiler wrapper so each subsequent official profile
  endpoint call receives a fresh 2+8 schedule and emits its own trace.
- `apply.sh`: verifies the exact base, refuses dirty/wrong revisions, applies
  all numbered patches with `git am`, and exits successfully only when the
  exact series is already applied directly on the required base.
- `pytest-evidence.txt`: exact isolated test command, dependency versions, and
  all-pass summary.

The source branch tip used to generate the patches is
`23450fb21ac255b0cf710f4ee965ee694921975d` (`opprof`).

## Apply

Prerequisite: a clean checkout whose `HEAD` is the exact base commit.

```bash
./patches/vllm-0.24.0-opprof/apply.sh /path/to/vllm-v0.24.0
```

Running the command again is a no-op only when the five matching patch commits
are rooted directly at the required base. A partially applied series, dirty
tree, unrelated commit, or any other `HEAD` is rejected instead of being
guessed around.

## Enable and output

Set an absolute output directory before starting vLLM:

```bash
export VLLM_OPPROF_DIR=/absolute/path/to/run/opprof
```

Unset or empty disables the feature before recorder construction. Combining it
with `--disable-log-stats` fails fast, as approved.

Each EngineCore/DP scheduler writes one file named approximately
`opprof-v1-dp0-pid1234-<start_ns>.jsonl`. Records contain schema/engine/step and
timestamps; scheduled prefill/decode composition; first/middle/final/unsplit
prefill chunks; 12-bin context and 9-bin chunk-size histograms; preemptions;
running/waiting/deferred queues; KV blocks/usage; local/external prefix deltas;
CUDA-graph hit/mode/bucket/padding; explicit null Layer-1 MoE load; and drop-gap
accounting. A clean close writes a final writer-count footer in the stream.

Every JSONL flush also atomically replaces
`<stream>.footer.json` through a same-directory temporary file. The sidecar
contains the encoded, written, and dropped counts through that durable flush,
the last written step index, a wall-clock timestamp, the one-second flush
interval, and whether it is final. Queue entries carry their submission
ordinal and cumulative drops, so a periodic checkpoint always satisfies
`encoded = written + dropped` without decoding records in the writer thread.
On clean close the in-stream footer is authoritative and the final sidecar must
agree with all three counters. If a hard kill prevents the in-stream footer,
the latest sidecar is authoritative: the decoded data-line count must equal
its `written_records`, its final data-line step must equal
`last_step_index`, and its counters must balance. Data after that checkpoint
may be lost, bounded by at most the configured one-second flush interval.

The bounded queue holds 8192 encoded records. Producers never wait for disk;
full queues or a failed writer drop the new record and report the gap on the
next successful record. A writer I/O failure is exposed through recorder state,
logged once, and cannot make shutdown wait indefinitely. The writer flushes at
1 MiB, one second, or shutdown.

## Test

Only pytest and msgspec are required. `--confcutdir` prevents vLLM's global
test configuration from importing its full dependency stack.

```bash
cd /path/to/vllm-v0.24.0
uv run --no-project --with pytest --with msgspec \
  pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q
```

Expected summary:

```text
18 passed in 1.09s
```

## Caveats

- Layer 1 intentionally records no expert-load arrays. Exact routed experts
  remain a separate Layer-2 run.
- `PIECEWISE` means graph-wrapped compiled regions, not full-step graph replay.
- Phase 2 must measure the always-on overhead; acceptance requires the upper
  bound of the 95% confidence interval to remain below 3% for every primary
  serving metric.
- Primary campaign topology is TP1 on community BF16 Qwen3-30B-A3B, with TP2
  and TP4 counterpoints. Record the selected MoE backend log every run.