Add OpProf campaign: protocols, results, patches, run evidence (P0-P6)

Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

patches/vllm-0.24.0-opprof/README.md

@@ -0,0 +1,119 @@
+# vLLM 0.24.0 OpProf patch series
+
+## Goal
+
+Apply the accepted OpProf Layer-1 instrumentation to exactly vLLM `v0.24.0`
+at base commit `ee0da84ab9e04ac7610e28580af62c365e898389`. The series adds one
+scheduler-owned composition record per step without installing new runtime
+dependencies or changing GPU kernels.
+
+## Contents
+
+- `0001-Add-lightweight-per-step-OpProf-telemetry.patch`: adds the environment
+  switch, import-light JSONL recorder/writer, scheduler hooks, and reuse of the
+  existing CUDA-graph stat. Writer failures are exposed without blocking
+  producers or shutdown, and request histograms are accumulated in-place.
+- `0002-Add-standalone-OpProf-telemetry-tests.patch`: adds CPU-only tests that
+  load the recorder directly without importing or installing vLLM or torch,
+  including ENOSPC, golden-record, and zero-token regressions.
+- `0003-Log-the-OpProf-output-path-at-startup.patch`: logs the resolved JSONL
+  output path and covers it in the standalone shutdown test.
+- `0004-Exclude-OpProf-output-path-from-compile-cache-key.patch`: prevents the
+  per-run telemetry destination from invalidating vLLM's torch.compile/AOT
+  cache and adds an import-light regression test.
+- `0005-Keep-compile-factor-regression-import-light.patch`: isolates the new
+  regression from torch in full vLLM test environments.
+- `0006-Checkpoint-OpProf-accounting-across-hard-kills.patch`: atomically
+  checkpoints balanced writer counters beside each JSONL stream once per
+  flush interval, with clean-close and hard-kill regressions.
+- `0007-Recreate-scheduled-torch-profiler-between-windows.patch`: discards a
+  stopped scheduled torch-profiler wrapper so each subsequent official profile
+  endpoint call receives a fresh 2+8 schedule and emits its own trace.
+- `apply.sh`: verifies the exact base, refuses dirty/wrong revisions, applies
+  all numbered patches with `git am`, and exits successfully only when the
+  exact series is already applied directly on the required base.
+- `pytest-evidence.txt`: exact isolated test command, dependency versions, and
+  all-pass summary.
+
+The source branch tip used to generate the patches is
+`23450fb21ac255b0cf710f4ee965ee694921975d` (`opprof`).
+
+## Apply
+
+Prerequisite: a clean checkout whose `HEAD` is the exact base commit.
+
+```bash
+./patches/vllm-0.24.0-opprof/apply.sh /path/to/vllm-v0.24.0
+```
+
+Running the command again is a no-op only when the five matching patch commits
+are rooted directly at the required base. A partially applied series, dirty
+tree, unrelated commit, or any other `HEAD` is rejected instead of being
+guessed around.
+
+## Enable and output
+
+Set an absolute output directory before starting vLLM:
+
+```bash
+export VLLM_OPPROF_DIR=/absolute/path/to/run/opprof
+```
+
+Unset or empty disables the feature before recorder construction. Combining it
+with `--disable-log-stats` fails fast, as approved.
+
+Each EngineCore/DP scheduler writes one file named approximately
+`opprof-v1-dp0-pid1234-<start_ns>.jsonl`. Records contain schema/engine/step and
+timestamps; scheduled prefill/decode composition; first/middle/final/unsplit
+prefill chunks; 12-bin context and 9-bin chunk-size histograms; preemptions;
+running/waiting/deferred queues; KV blocks/usage; local/external prefix deltas;
+CUDA-graph hit/mode/bucket/padding; explicit null Layer-1 MoE load; and drop-gap
+accounting. A clean close writes a final writer-count footer in the stream.
+
+Every JSONL flush also atomically replaces
+`<stream>.footer.json` through a same-directory temporary file. The sidecar
+contains the encoded, written, and dropped counts through that durable flush,
+the last written step index, a wall-clock timestamp, the one-second flush
+interval, and whether it is final. Queue entries carry their submission
+ordinal and cumulative drops, so a periodic checkpoint always satisfies
+`encoded = written + dropped` without decoding records in the writer thread.
+On clean close the in-stream footer is authoritative and the final sidecar must
+agree with all three counters. If a hard kill prevents the in-stream footer,
+the latest sidecar is authoritative: the decoded data-line count must equal
+its `written_records`, its final data-line step must equal
+`last_step_index`, and its counters must balance. Data after that checkpoint
+may be lost, bounded by at most the configured one-second flush interval.
+
+The bounded queue holds 8192 encoded records. Producers never wait for disk;
+full queues or a failed writer drop the new record and report the gap on the
+next successful record. A writer I/O failure is exposed through recorder state,
+logged once, and cannot make shutdown wait indefinitely. The writer flushes at
+1 MiB, one second, or shutdown.
+
+## Test
+
+Only pytest and msgspec are required. `--confcutdir` prevents vLLM's global
+test configuration from importing its full dependency stack.
+
+```bash
+cd /path/to/vllm-v0.24.0
+uv run --no-project --with pytest --with msgspec \
+  pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q
+```
+
+Expected summary:
+
+```text
+18 passed in 1.09s
+```
+
+## Caveats
+
+- Layer 1 intentionally records no expert-load arrays. Exact routed experts
+  remain a separate Layer-2 run.
+- `PIECEWISE` means graph-wrapped compiled regions, not full-step graph replay.
+- Phase 2 must measure the always-on overhead; acceptance requires the upper
+  bound of the 95% confidence interval to remain below 3% for every primary
+  serving metric.
+- Primary campaign topology is TP1 on community BF16 Qwen3-30B-A3B, with TP2
+  and TP4 counterpoints. Record the selected MoE backend log every run.