Add OpProf campaign: protocols, results, patches, run evidence (P0-P6)

Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/opprof/phase4-optimization-plan.md

@@ -0,0 +1,306 @@
+# OpProf Phase 4 measured optimization plan
+
+Status: **PROPOSAL FROM ACCEPTED PHASE-3 DATA; NO UNMEASURED GAIN CLAIMS**.
+
+Date: 2026-07-12. This plan uses only the accepted Phase-3 40/52-run,
+20/24-cell dataset and the single optional Phase-4 capture-size validation.
+Phase-3 protocol and results are frozen. Bounds below are ceilings in the
+units actually measured; padding or raggedness percentages are not relabeled
+as end-to-end throughput gains.
+
+## Pinned implementation context
+
+- Model/hardware: Qwen3-30B-A3B BF16, one H20, TP1 primary.
+- vLLM source: accepted OpProf tip
+  `23450fb21ac255b0cf710f4ee965ee694921975d` on v0.24.0.
+- vLLM 0.24.0 exposes `--cudagraph-capture-sizes`,
+  `--max-cudagraph-capture-size`, `--max-num-seqs`, and
+  `--max-num-batched-tokens` (`vllm/engine/arg_utils.py:1390-1467`).
+- An explicit capture-size list replaces the inferred list. The default is
+  `[1,2,4]`, multiples of eight below 256, then multiples of sixteen through
+  the maximum, normally 512 (`vllm/config/vllm.py:1669-1792`).
+- Chunked prefill remains enabled. vLLM schedules decode first, then fills the
+  remaining MBT budget with prefill and chunks an over-budget prefill
+  (`docs/configuration/optimization.md:45-59`).
+
+The ranking is by measured opportunity, readiness, and downside together—not
+by a normalized composite score. Items that share the same raggedness bound
+are explicitly non-additive.
+
+## Ranked optimization list
+
+| Rank | Tier | Target and affected regime | Measured bound | Owner | Decision |
+|---:|---|---|---|---|---|
+| 1 | Config now / scheduler backlog | Preserve prefix affinity and prefix caching for P08-like shared-prefix traffic | P08 vs matched P07: **+82.14% saturation req/s**, 80% prefix-hit ratio, **62.29% fewer prefill tokens** | Serving/config owner now; cache-aware dispatch upstream | Deploy behind a workload classifier; do not apply to no-sharing traffic |
+| 2 | Scheduler | Length-aware cohorting/admission for ragged P10/P09/P06 | At most **44.79 pp** R64 contrast and **44.69%** measured efficiency gap on P10; 39.62 pp/8.32% on P09; 35.44 pp/22.85% on P06 | Upstream vLLM scheduler | Highest structural backlog item; preserve fairness and arrival semantics |
+| 3 | Config now | Add exact small CUDAGraph sizes for P09/P10-like moderate decode batches | Distributional bound **4.98 pp P09 / 5.26 pp P10** padding; P09 validation achieved **4.980 pp** | Serving/config owner | Mechanism confirmed; canary only because p95 was +3.01% in one pair |
+| 4 | Config now | Route P06/P10-like pools to MNS=64 | Saturation throughput **+3.37% P06 / +3.70% P10** versus C00 | Serving/config owner | Pattern-specific trial only; same setting was −24.27% on P01 |
+| 5 | Kernel backlog | Ragged-aware MoE GEMM and attention over the measured shape stream | Shares the rank-2 ceiling; no independent additive gain. Descriptive MoE share is 54.04–75.00% on P10/P09/P06 moderate | Engine/kernel colleagues | Optimize exact weighted shapes below; require serving confirmation |
+| 6 | Config guardrail | Keep MBT=8192 for short/high-throughput and long-prefill classes; do not globally set 2048 | Avoided saturation regressions up to **11.64% P03**, 6.53% P01, 5.82% P10 | Serving/config owner | Encode as a policy guardrail, not a positive optimization claim |
+
+## Tier A — configuration-level actions deployable today
+
+### A1. Prefix-affine routing with prefix caching
+
+**Measured regime.** P07 and P08 have the same 1,280-token prompt length,
+512-token output, burst-of-eight arrival, C00 config, and seed. P08 alone uses
+eight 1,024-token shared prefixes plus a unique 256-token suffix. With prefix
+caching already enabled:
+
+| Metric, saturation | P07 no sharing | P08 high sharing | Delta |
+|---|---:|---:|---:|
+| Prefix-query hit ratio | 0.00 | 0.80 | +0.80 |
+| Clean prefill tokens | 1,528,968 | 576,512 | **−62.29%** |
+| Completed throughput | 5.1083 req/s | 9.3042 req/s | **+82.14%** |
+
+**Root mechanism.** The matched cell changes only controlled prefix sharing;
+the observed hit and prefill-token changes directly identify cache reuse. The
+82.14% throughput delta is a point observation and the maximum evidence-backed
+gain for this exact regime, not a fleet-wide forecast.
+
+**Action now.** Keep `--enable-prefix-caching`; hash an application-known
+stable prefix or conversation identity to a replica so related requests do
+not destroy affinity through round-robin load balancing. Apply only when the
+online prefix-query hit ratio resembles P08, not P07.
+
+**Verification.** Interleave affinity ON/OFF on the same replicas and fixed
+request stream. Primary gates are prefix-hit ratio, prefill-token reduction,
+completed req/s, TTFT p95, per-replica queue imbalance, and KV occupancy. A
+throughput gain with queue/fairness or KV-capacity regression does not pass.
+
+### A2. Measured CUDAGraph capture sizes
+
+**Measured regime.** P09 moderate has decode-batch p50/p95/max 4/16/25. P10
+moderate has 1/4/7. Default captures skip sizes 3, 5, 6, 7, and 9, so those
+batches pad upward. Replaying the Phase-3 hit distribution predicts that
+adding exactly `{3,5,6,7,9}` to the complete default list can remove:
+
+- **4.9776 percentage points** of P09's 8.5421% graph-hit padding; and
+- **5.2579 points** of P10's 5.5652% padding.
+
+The list must be `default ∪ {3,5,6,7,9}`; passing only five sizes would replace
+and discard the rest of the default list.
+
+**Closed-loop result.** One P09 moderate ON/OFF pair used fresh servers, the
+same fixed seed and accepted saturation-rate source, 60 excluded warm-up
+seconds, and 240 clean seconds per arm. No Layer-2 profiler ran.
+
+| Metric | ON: exact sizes | OFF: default | ON relative to OFF |
+|---|---:|---:|---:|
+| Graph-hit padding | **3.5659%** | 8.5456% | **−4.9798 pp; −58.27%** |
+| Useful tokens/model-step ms | 4.56222 | 4.55405 | +0.179% |
+| Completed throughput | 4.9583 req/s | 4.9333 req/s | +0.507% |
+| Mean E2E latency | 1.6123 s | 1.6812 s | −4.10% |
+| p95 E2E latency | 3.9930 s | 3.8763 s | **+3.01%** |
+| Clean failures | 0 | 0 | equal |
+
+The observed padding reduction differs from the Phase-3 bound by only 0.0022
+percentage points, confirming the mechanism. It does **not** establish a broad
+performance win: token efficiency and throughput moved less than 1%, p95 moved
+the wrong way, and there is one ordered pair with no CI.
+
+Operational cost also matters: ON captured 56 FULL and 56 PIECEWISE sizes
+versus 51/51 OFF, estimated graph memory increased 0.64→0.68 GiB, and server
+ownership was 76.9 seconds longer. Deploy only as a pattern-specific canary;
+require interleaved replication with a p95 non-regression gate before rollout.
+
+### A3. Pattern-specific MNS pools
+
+The complete saturation comparisons show that `--max-num-seqs 64` is an
+interaction, not a globally better default:
+
+| Pattern | C10 MNS=64 vs C00 | Interpretation |
+|---|---:|---|
+| P01 short/short | **−24.27% req/s** | Reject for dense short traffic |
+| P03 long/short | −1.41% | No measured benefit |
+| P06 bimodal/long burst | **+3.37%** | Candidate pattern pool |
+| P10 real long-context | **+3.70%** | Candidate pattern pool |
+
+The evidence identifies the config×pattern interaction but not a lower-level
+cause. Do not attribute it to a particular kernel or queue effect without a
+new bisection. Verification is five interleaved saturation pairs per intended
+class plus TTFT, queue depth, preemption, KV usage, and exact-work checks.
+
+### A4. MBT policy guardrail
+
+`--max-num-batched-tokens 2048` versus the default 8192 changed saturation
+throughput by −6.53% P01, −11.64% P03, +0.35% P06, and −5.82% P10. The combined
+MNS64/MBT2048 setting was −30.22% on P01. Phase 3 therefore supports retaining
+MBT8192 for these classes and rejects a global MBT2048 rollout. The bound is
+an avoided regression, not new speedup.
+
+## Tier B — upstream scheduler changes
+
+### B1. Length-aware cohorting without starvation
+
+**Measured mechanism.** R64 is the rectangular padding fraction of the exact
+arrival-order prompt stream. It is 0.6923 for P10, 0.7648 for P09, and 0.5988
+for P06. Their passing control contrasts are:
+
+| Irregular pattern | Control | R64 excess | Useful-token efficiency loss |
+|---|---|---:|---:|
+| P10 | P03 | **44.79 pp** | **44.69%** |
+| P10 | P04 | **44.79 pp** | **14.26%** |
+| P09 | P01 | **39.62 pp** | **8.32%** |
+| P06 | P02 | **23.01 pp** | **11.61%** |
+| P06 | P04 | **35.44 pp** | **22.85%** |
+
+These are upper bounds on waste a length-aware path could avoid. R64 is not
+observed GPU time, and efficiency association is not causal. The scheduler
+change should maintain several ready queues by remaining prompt/context band,
+select a less-ragged cohort subject to the existing decode-first token budget,
+and impose a finite age/fairness bound. It must not rewrite request arrivals or
+drop long requests.
+
+**Verification.** Add a runtime per-step raggedness counter rather than using
+manifest R64 as a surrogate. Compare fixed-arrival ON/OFF runs for useful
+tokens/model-step ms, TTFT/E2E p95, queue age, starvation count, preemption,
+KV occupancy, and the full length histogram. The gain cannot exceed the
+corresponding R64/efficiency bounds above, and it is non-additive with a
+ragged-aware kernel.
+
+### B2. Automatic cache-aware dispatch
+
+The upstream form of A1 is a scheduler/replica dispatcher that chooses a live
+prefix-cache owner while respecting load. Its collaboration contract is the
+measured P07/P08 tuple: 1,024 shared + 256 unique prompt tokens, eight prefix
+IDs, burst size eight, output 512, target hit ratio 0.80. The load-balancing
+penalty and lost cache hits must be reported together; a synthetic cache hit
+increase without end-to-end balance is insufficient.
+
+### B3. Histogram-driven capture-list generation
+
+Static A2 proves that Layer-1 can choose useful sizes. An upstream controller
+could select a bounded number of exact sizes from padding contribution
+`count(size) * (next_bucket-size)`, while retaining the default list and a
+memory/startup budget. P09's top five `{3,5,6,7,9}` recovered 4.98 points at a
+0.04-GiB graph-memory and 76.9-second server-lifetime cost in this run. The
+selector must freeze its list before measurement and never continually tune on
+the scored window.
+
+## Tier C — kernel-level backlog and collaboration interface
+
+H1a is inconclusive, so Phase 3 does not prove a universal top operator. The
+available moderate windows are still useful shape inputs: descriptive MoE-GEMM
+shares are 57.52% P06, 75.00% P09, and 54.04% P10; attention shares are 31.60%,
+16.19%, and 30.61%. Only P04's operator windows pass inference gates, where
+attention is 47.88% and MoE GEMM 40.64%. Kernel work must therefore claim
+shape-local improvement, not a resolved global bottleneck.
+
+### Exact shape stream for kernel engineers
+
+`P`, `D`, and `N` below are per-step prefill tokens, decode tokens, and
+scheduled requests. Counts are from clean C00-moderate Layer-1 records.
+Context mix is the fraction of scheduled request-context observations in
+`<=1024 / 1025–8192 / 8193–32768 / >32768` bins.
+
+| Pattern | Model steps: decode / mixed / prefill | N p50 / p95 / max | Dominant exact `(P,D,N): count` | Context mix | Chunk signal |
+|---|---|---|---|---|---|
+| P01 | 5,760: 114 / 5,646 / 0 | 69 / 74 / 77 | `(0,66,66):18`, `(0,67,67):18`; mixed P p50=334, D p50=68 | 100.00 / 0 / 0 / 0% | 6,235 unsplit; sizes 129–512 dominate |
+| P04 | 12,455: 12,291 / 141 / 23 | 8 / 8 / 16 | `(0,8,8):12,129`, `(8191,1,3):23` | 0.03 / 94.62 / 5.35 / 0% | 238/319 chunks >2,048; first/final 122/123 |
+| P06 | 17,464: 17,310 / 152 / 2 | 8 / 16 / 16 | `(0,8,8):13,103`, `(0,16,16):4,098` | 55.11 / 42.93 / 1.96 / 0% | 174/422 >2,048; 136 in 257–512 |
+| P09 | 12,837: 11,783 / 1,054 / 0 | 4 / 16 / 25 | `(0,3,3):3,397`, `(0,2,2):1,549`, `(0,4,4):1,526`, `(0,5,5):1,050` | 51.70 / 48.26 / 0.04 / 0% | 228/1,193 >2,048; 345 in 1,025–2,048 |
+| P10 | 18,130: 17,941 / 92 / 97 | 1 / 4 / 7 | `(0,1,1):13,572`, `(0,2,2):2,675`, `(0,3,3):792`, `(0,4,4):524`, `(8192,0,1):38` | 12.25 / 39.94 / 47.76 / 0.05% | 138/191 >2,048; first/middle/final/unsplit 49/29/49/64 |
+
+The exported kernel-benchmark interface should be a prompt-free weighted table
+with `(P,D,N,context_bin,chunk_class,chunk_size_bin,runtime_mode,count)` plus
+step duration and useful tokens. Use the frozen histogram edges already emitted
+by Layer 1: context `128..131072` powers of two and chunk `16..2048` powers of
+two. Preserve the joint tuples; independent marginal sampling would erase the
+mixed-batch structure.
+
+### Kernel targets and acceptance
+
+1. **Ragged MoE GEMM:** accept variable token counts without padding every
+   expert/layer tile to the largest sequence. Weight microbenchmarks by the P06,
+   P09, and P10 tuples above. The ceiling is the same R64/efficiency opportunity
+   as B1, not an additional gain.
+2. **Attention:** retain P04 `(D,N)=(8,8)` as the valid long rectangular control
+   and test P10's mostly 1–4 decode batches plus 8,192-token chunks. Report
+   useful-token time, workspace, and graph compatibility.
+3. **Serving confirmation:** kernel time must improve on the exact weighted
+   stream, then pass a fixed-arrival serving A/B for throughput, TTFT/p95,
+   memory, and correctness. A rectangular-only kernel win does not close the
+   Phase-3 finding.
+
+`moe_expert_load` was unavailable in Phase 3. No expert-imbalance mechanism or
+gain is claimed; expert-specific packing requires a new low-overhead route
+histogram before implementation.
+
+## Honest limits and Phase-5 measurement requirement
+
+- H1a remains inconclusive: only P04 had two representative/recovered operator
+  windows. Eight other completed moderate patterns failed window validity even
+  though kernel classifiability was 97.05–99.64%.
+- A Phase 5 operator study needs longer, time-stratified samples that reproduce
+  clean scheduled-token, prefill-fraction, decode-batch, and graph-mode
+  distributions, plus a lower-perturbation per-op timer. It must demonstrate
+  overhead before using shares for optimization; the Phase-2 Kineto active
+  window perturbed throughput by 51.3%.
+- Confirmation runs are absent. The MNS and MBT config effects are single-run
+  point estimates and require replication before production decisions.
+- R64 is an offline rectangular-padding upper bound, not measured GPU idle
+  time. H1b's efficiency association does not establish causality.
+- Mixed-batch interference was N/A because no cell retained 30 supported mixed
+  steps inside both leave-one-pattern-out pure-fit supports.
+- Results cover one model, BF16, H20, mostly TP1, and 20/24 cells. P03/C11,
+  P05/C00, P10/C00-TP2, and P11/C00 are absent.
+- The capture validation is one ordered ON/OFF pair. Its padding endpoint is
+  mechanism-valid, but performance deltas have no CI and p95 regressed.
+- Layer 1 did not collect expert-route identities; kernel engineers cannot infer
+  routed-expert imbalance from these artifacts.
+
+## Verification and stop rules for Phase 4 work
+
+Every proposed experiment keeps the original fixed manifest/seed/work, excludes
+warm-up, records Layer-1 accounting, and changes one mechanism. A candidate
+stops on clean failure, footer imbalance/drop, output mismatch, GPU
+contamination, memory regression beyond its declared budget, or violation of
+the 16-H20-hour campaign cap. Throughput, latency, memory, and correctness are
+reported together; no metric shopping or silent pattern substitution is
+allowed.
+
+## GPU accounting
+
+The optional capture pair consumed **0.296389 H20-hours**, taking cumulative
+campaign use from 14.025875 to **14.322265 H20-hours**. Remaining headroom is
+**1.677735 H20-hours**. Both arms returned GPU0 to zero, all eight GPUs were
+0 MiB/0% at final inspection, and no other-user process appeared.
+
+Artifacts are under
+`runs/opprof-phase3/phase4/capture-p09/`; `result.json` SHA-256 is
+`5bb91df28790f6f3c34e4e9ed8e35a1cb8100f93086a4286689d587fd732f2a4`.
+
+## Final ranked one-liners
+
+1. **Prefix affinity (config now):** P08's measured ceiling is **+82.14% req/s** with 62.29% fewer prefill tokens versus matched P07.
+2. **Length-aware scheduler:** raggedness ceiling is **44.79 pp R64 / 44.69% efficiency gap** on P10; smaller confirmed bounds apply to P09/P06.
+3. **Exact capture sizes (config now):** ceiling **5.26 pp P10 / 4.98 pp P09 padding**; P09 validation removed 4.980 pp but did not prove p95 gain.
+4. **MNS64 pattern pools (config now):** measured ceiling **+3.70% req/s P10 / +3.37% P06**, with a −24.27% P01 counterexample.
+5. **Ragged kernels (kernel backlog):** share rank 2's bound; no additive E2E bound is supported while H1a is inconclusive.
+6. **MBT8192 guardrail (config now):** avoids measured regressions up to **11.64%**; MBT2048 has no general positive case.
+
+## Data sanity block
+
+| Numeric family | n | finite | missing | min | max | distinct | Invariant/result |
+|---|---:|---:|---:|---:|---:|---:|---|
+| Ranked items | 6 | 6 | 0 | rank 1 | rank 6 | 6 | Three tiers represented; bounds not summed |
+| Sentinel saturation config deltas | 11 | 11 | 0 | −30.216% | +3.704% | 11 | Both gains and regressions retained |
+| Passing R64 contrast effects | 5 | 5 | 0 | 0.230148 | 0.447872 | 4 | Ratios in [0,1]; duplicate P10 controls expected |
+| Capture-arm padding fraction | 2 | 2 | 0 | 0.035659 | 0.085456 | 2 | Non-negative; ON < OFF |
+| Capture-arm token efficiency | 2 | 2 | 0 | 4.554051 | 4.562222 | 2 | Positive; +0.179% ON |
+| Capture-arm throughput (req/s) | 2 | 2 | 0 | 4.933333 | 4.958333 | 2 | Same offered rate 4.920833 req/s |
+| Capture-arm clean failures | 2 | 2 | 0 | 0 | 0 | 1 expected | Exact 240 s and zero failures |
+| Capture-arm Layer-1 records | 2 | 2 | 0 | 16,491 | 17,579 | 2 | Every footer/sidecar invariant true; zero drops |
+| Optional validation GPU-hours | 1 | 1 | 0 | 0.296389 | 0.296389 | 1 | Positive; cumulative 14.322265 < 16 |
+| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | Cleanup passed |
+
+Checked invariants: Phase-3 metrics remain frozen; every cited number resolves
+to accepted metrics or the checksum-recorded validation; config comparisons
+use saturation rather than normalized moderate throughput; padding/raggedness
+bounds are not presented as throughput; duplicate/non-independent bounds are
+not added; both validation arms use identical work and offered rate; clean
+failures are zero; output work, Layer-1 schema, step continuity, footer/sidecar
+balance, and zero drops pass; ratios lie in their declared domains; all GPU
+memory returned to zero; and cumulative GPU use stays below 16 H20-hours. No
+data-sanity red flag remains.