aituner/docs/opprof/phase4-optimization-plan.md

# OpProf Phase 4 measured optimization plan

Status: **PROPOSAL FROM ACCEPTED PHASE-3 DATA; NO UNMEASURED GAIN CLAIMS**.

Date: 2026-07-12. This plan uses only the accepted Phase-3 40/52-run,
20/24-cell dataset and the single optional Phase-4 capture-size validation.
Phase-3 protocol and results are frozen. Bounds below are ceilings in the
units actually measured; padding or raggedness percentages are not relabeled
as end-to-end throughput gains.

## Pinned implementation context

- Model/hardware: Qwen3-30B-A3B BF16, one H20, TP1 primary.
- vLLM source: accepted OpProf tip
  `23450fb21ac255b0cf710f4ee965ee694921975d` on v0.24.0.
- vLLM 0.24.0 exposes `--cudagraph-capture-sizes`,
  `--max-cudagraph-capture-size`, `--max-num-seqs`, and
  `--max-num-batched-tokens` (`vllm/engine/arg_utils.py:1390-1467`).
- An explicit capture-size list replaces the inferred list. The default is
  `[1,2,4]`, multiples of eight below 256, then multiples of sixteen through
  the maximum, normally 512 (`vllm/config/vllm.py:1669-1792`).
- Chunked prefill remains enabled. vLLM schedules decode first, then fills the
  remaining MBT budget with prefill and chunks an over-budget prefill
  (`docs/configuration/optimization.md:45-59`).

The ranking is by measured opportunity, readiness, and downside together—not
by a normalized composite score. Items that share the same raggedness bound
are explicitly non-additive.

## Ranked optimization list

| Rank | Tier | Target and affected regime | Measured bound | Owner | Decision |
|---:|---|---|---|---|---|
| 1 | Config now / scheduler backlog | Preserve prefix affinity and prefix caching for P08-like shared-prefix traffic | P08 vs matched P07: **+82.14% saturation req/s**, 80% prefix-hit ratio, **62.29% fewer prefill tokens** | Serving/config owner now; cache-aware dispatch upstream | Deploy behind a workload classifier; do not apply to no-sharing traffic |
| 2 | Scheduler | Length-aware cohorting/admission for ragged P10/P09/P06 | At most **44.79 pp** R64 contrast and **44.69%** measured efficiency gap on P10; 39.62 pp/8.32% on P09; 35.44 pp/22.85% on P06 | Upstream vLLM scheduler | Highest structural backlog item; preserve fairness and arrival semantics |
| 3 | Config now | Add exact small CUDAGraph sizes for P09/P10-like moderate decode batches | Distributional bound **4.98 pp P09 / 5.26 pp P10** padding; P09 validation achieved **4.980 pp** | Serving/config owner | Mechanism confirmed; canary only because p95 was +3.01% in one pair |
| 4 | Config now | Route P06/P10-like pools to MNS=64 | Saturation throughput **+3.37% P06 / +3.70% P10** versus C00 | Serving/config owner | Pattern-specific trial only; same setting was −24.27% on P01 |
| 5 | Kernel backlog | Ragged-aware MoE GEMM and attention over the measured shape stream | Shares the rank-2 ceiling; no independent additive gain. Descriptive MoE share is 54.04–75.00% on P10/P09/P06 moderate | Engine/kernel colleagues | Optimize exact weighted shapes below; require serving confirmation |
| 6 | Config guardrail | Keep MBT=8192 for short/high-throughput and long-prefill classes; do not globally set 2048 | Avoided saturation regressions up to **11.64% P03**, 6.53% P01, 5.82% P10 | Serving/config owner | Encode as a policy guardrail, not a positive optimization claim |

## Tier A — configuration-level actions deployable today

### A1. Prefix-affine routing with prefix caching

**Measured regime.** P07 and P08 have the same 1,280-token prompt length,
512-token output, burst-of-eight arrival, C00 config, and seed. P08 alone uses
eight 1,024-token shared prefixes plus a unique 256-token suffix. With prefix
caching already enabled:

| Metric, saturation | P07 no sharing | P08 high sharing | Delta |
|---|---:|---:|---:|
| Prefix-query hit ratio | 0.00 | 0.80 | +0.80 |
| Clean prefill tokens | 1,528,968 | 576,512 | **−62.29%** |
| Completed throughput | 5.1083 req/s | 9.3042 req/s | **+82.14%** |

**Root mechanism.** The matched cell changes only controlled prefix sharing;
the observed hit and prefill-token changes directly identify cache reuse. The
82.14% throughput delta is a point observation and the maximum evidence-backed
gain for this exact regime, not a fleet-wide forecast.

**Action now.** Keep `--enable-prefix-caching`; hash an application-known
stable prefix or conversation identity to a replica so related requests do
not destroy affinity through round-robin load balancing. Apply only when the
online prefix-query hit ratio resembles P08, not P07.

**Verification.** Interleave affinity ON/OFF on the same replicas and fixed
request stream. Primary gates are prefix-hit ratio, prefill-token reduction,
completed req/s, TTFT p95, per-replica queue imbalance, and KV occupancy. A
throughput gain with queue/fairness or KV-capacity regression does not pass.

### A2. Measured CUDAGraph capture sizes

**Measured regime.** P09 moderate has decode-batch p50/p95/max 4/16/25. P10
moderate has 1/4/7. Default captures skip sizes 3, 5, 6, 7, and 9, so those
batches pad upward. Replaying the Phase-3 hit distribution predicts that
adding exactly `{3,5,6,7,9}` to the complete default list can remove:

- **4.9776 percentage points** of P09's 8.5421% graph-hit padding; and
- **5.2579 points** of P10's 5.5652% padding.

The list must be `default ∪ {3,5,6,7,9}`; passing only five sizes would replace
and discard the rest of the default list.

**Closed-loop result.** One P09 moderate ON/OFF pair used fresh servers, the
same fixed seed and accepted saturation-rate source, 60 excluded warm-up
seconds, and 240 clean seconds per arm. No Layer-2 profiler ran.

| Metric | ON: exact sizes | OFF: default | ON relative to OFF |
|---|---:|---:|---:|
| Graph-hit padding | **3.5659%** | 8.5456% | **−4.9798 pp; −58.27%** |
| Useful tokens/model-step ms | 4.56222 | 4.55405 | +0.179% |
| Completed throughput | 4.9583 req/s | 4.9333 req/s | +0.507% |
| Mean E2E latency | 1.6123 s | 1.6812 s | −4.10% |
| p95 E2E latency | 3.9930 s | 3.8763 s | **+3.01%** |
| Clean failures | 0 | 0 | equal |

The observed padding reduction differs from the Phase-3 bound by only 0.0022
percentage points, confirming the mechanism. It does **not** establish a broad
performance win: token efficiency and throughput moved less than 1%, p95 moved
the wrong way, and there is one ordered pair with no CI.

Operational cost also matters: ON captured 56 FULL and 56 PIECEWISE sizes
versus 51/51 OFF, estimated graph memory increased 0.64→0.68 GiB, and server
ownership was 76.9 seconds longer. Deploy only as a pattern-specific canary;
require interleaved replication with a p95 non-regression gate before rollout.

### A3. Pattern-specific MNS pools

The complete saturation comparisons show that `--max-num-seqs 64` is an
interaction, not a globally better default:

| Pattern | C10 MNS=64 vs C00 | Interpretation |
|---|---:|---|
| P01 short/short | **−24.27% req/s** | Reject for dense short traffic |
| P03 long/short | −1.41% | No measured benefit |
| P06 bimodal/long burst | **+3.37%** | Candidate pattern pool |
| P10 real long-context | **+3.70%** | Candidate pattern pool |

The evidence identifies the config×pattern interaction but not a lower-level
cause. Do not attribute it to a particular kernel or queue effect without a
new bisection. Verification is five interleaved saturation pairs per intended
class plus TTFT, queue depth, preemption, KV usage, and exact-work checks.

### A4. MBT policy guardrail

`--max-num-batched-tokens 2048` versus the default 8192 changed saturation
throughput by −6.53% P01, −11.64% P03, +0.35% P06, and −5.82% P10. The combined
MNS64/MBT2048 setting was −30.22% on P01. Phase 3 therefore supports retaining
MBT8192 for these classes and rejects a global MBT2048 rollout. The bound is
an avoided regression, not new speedup.

## Tier B — upstream scheduler changes

### B1. Length-aware cohorting without starvation

**Measured mechanism.** R64 is the rectangular padding fraction of the exact
arrival-order prompt stream. It is 0.6923 for P10, 0.7648 for P09, and 0.5988
for P06. Their passing control contrasts are:

| Irregular pattern | Control | R64 excess | Useful-token efficiency loss |
|---|---|---:|---:|
| P10 | P03 | **44.79 pp** | **44.69%** |
| P10 | P04 | **44.79 pp** | **14.26%** |
| P09 | P01 | **39.62 pp** | **8.32%** |
| P06 | P02 | **23.01 pp** | **11.61%** |
| P06 | P04 | **35.44 pp** | **22.85%** |

These are upper bounds on waste a length-aware path could avoid. R64 is not
observed GPU time, and efficiency association is not causal. The scheduler
change should maintain several ready queues by remaining prompt/context band,
select a less-ragged cohort subject to the existing decode-first token budget,
and impose a finite age/fairness bound. It must not rewrite request arrivals or
drop long requests.

**Verification.** Add a runtime per-step raggedness counter rather than using
manifest R64 as a surrogate. Compare fixed-arrival ON/OFF runs for useful
tokens/model-step ms, TTFT/E2E p95, queue age, starvation count, preemption,
KV occupancy, and the full length histogram. The gain cannot exceed the
corresponding R64/efficiency bounds above, and it is non-additive with a
ragged-aware kernel.

### B2. Automatic cache-aware dispatch

The upstream form of A1 is a scheduler/replica dispatcher that chooses a live
prefix-cache owner while respecting load. Its collaboration contract is the
measured P07/P08 tuple: 1,024 shared + 256 unique prompt tokens, eight prefix
IDs, burst size eight, output 512, target hit ratio 0.80. The load-balancing
penalty and lost cache hits must be reported together; a synthetic cache hit
increase without end-to-end balance is insufficient.

### B3. Histogram-driven capture-list generation

Static A2 proves that Layer-1 can choose useful sizes. An upstream controller
could select a bounded number of exact sizes from padding contribution
`count(size) * (next_bucket-size)`, while retaining the default list and a
memory/startup budget. P09's top five `{3,5,6,7,9}` recovered 4.98 points at a
0.04-GiB graph-memory and 76.9-second server-lifetime cost in this run. The
selector must freeze its list before measurement and never continually tune on
the scored window.

## Tier C — kernel-level backlog and collaboration interface

H1a is inconclusive, so Phase 3 does not prove a universal top operator. The
available moderate windows are still useful shape inputs: descriptive MoE-GEMM
shares are 57.52% P06, 75.00% P09, and 54.04% P10; attention shares are 31.60%,
16.19%, and 30.61%. Only P04's operator windows pass inference gates, where
attention is 47.88% and MoE GEMM 40.64%. Kernel work must therefore claim
shape-local improvement, not a resolved global bottleneck.

### Exact shape stream for kernel engineers

`P`, `D`, and `N` below are per-step prefill tokens, decode tokens, and
scheduled requests. Counts are from clean C00-moderate Layer-1 records.
Context mix is the fraction of scheduled request-context observations in
`<=1024 / 1025–8192 / 8193–32768 / >32768` bins.

| Pattern | Model steps: decode / mixed / prefill | N p50 / p95 / max | Dominant exact `(P,D,N): count` | Context mix | Chunk signal |
|---|---|---|---|---|---|
| P01 | 5,760: 114 / 5,646 / 0 | 69 / 74 / 77 | `(0,66,66):18`, `(0,67,67):18`; mixed P p50=334, D p50=68 | 100.00 / 0 / 0 / 0% | 6,235 unsplit; sizes 129–512 dominate |
| P04 | 12,455: 12,291 / 141 / 23 | 8 / 8 / 16 | `(0,8,8):12,129`, `(8191,1,3):23` | 0.03 / 94.62 / 5.35 / 0% | 238/319 chunks >2,048; first/final 122/123 |
| P06 | 17,464: 17,310 / 152 / 2 | 8 / 16 / 16 | `(0,8,8):13,103`, `(0,16,16):4,098` | 55.11 / 42.93 / 1.96 / 0% | 174/422 >2,048; 136 in 257–512 |
| P09 | 12,837: 11,783 / 1,054 / 0 | 4 / 16 / 25 | `(0,3,3):3,397`, `(0,2,2):1,549`, `(0,4,4):1,526`, `(0,5,5):1,050` | 51.70 / 48.26 / 0.04 / 0% | 228/1,193 >2,048; 345 in 1,025–2,048 |
| P10 | 18,130: 17,941 / 92 / 97 | 1 / 4 / 7 | `(0,1,1):13,572`, `(0,2,2):2,675`, `(0,3,3):792`, `(0,4,4):524`, `(8192,0,1):38` | 12.25 / 39.94 / 47.76 / 0.05% | 138/191 >2,048; first/middle/final/unsplit 49/29/49/64 |

The exported kernel-benchmark interface should be a prompt-free weighted table
with `(P,D,N,context_bin,chunk_class,chunk_size_bin,runtime_mode,count)` plus
step duration and useful tokens. Use the frozen histogram edges already emitted
by Layer 1: context `128..131072` powers of two and chunk `16..2048` powers of
two. Preserve the joint tuples; independent marginal sampling would erase the
mixed-batch structure.

### Kernel targets and acceptance

1. **Ragged MoE GEMM:** accept variable token counts without padding every
   expert/layer tile to the largest sequence. Weight microbenchmarks by the P06,
   P09, and P10 tuples above. The ceiling is the same R64/efficiency opportunity
   as B1, not an additional gain.
2. **Attention:** retain P04 `(D,N)=(8,8)` as the valid long rectangular control
   and test P10's mostly 1–4 decode batches plus 8,192-token chunks. Report
   useful-token time, workspace, and graph compatibility.
3. **Serving confirmation:** kernel time must improve on the exact weighted
   stream, then pass a fixed-arrival serving A/B for throughput, TTFT/p95,
   memory, and correctness. A rectangular-only kernel win does not close the
   Phase-3 finding.

`moe_expert_load` was unavailable in Phase 3. No expert-imbalance mechanism or
gain is claimed; expert-specific packing requires a new low-overhead route
histogram before implementation.

## Honest limits and Phase-5 measurement requirement

- H1a remains inconclusive: only P04 had two representative/recovered operator
  windows. Eight other completed moderate patterns failed window validity even
  though kernel classifiability was 97.05–99.64%.
- A Phase 5 operator study needs longer, time-stratified samples that reproduce
  clean scheduled-token, prefill-fraction, decode-batch, and graph-mode
  distributions, plus a lower-perturbation per-op timer. It must demonstrate
  overhead before using shares for optimization; the Phase-2 Kineto active
  window perturbed throughput by 51.3%.
- Confirmation runs are absent. The MNS and MBT config effects are single-run
  point estimates and require replication before production decisions.
- R64 is an offline rectangular-padding upper bound, not measured GPU idle
  time. H1b's efficiency association does not establish causality.
- Mixed-batch interference was N/A because no cell retained 30 supported mixed
  steps inside both leave-one-pattern-out pure-fit supports.
- Results cover one model, BF16, H20, mostly TP1, and 20/24 cells. P03/C11,
  P05/C00, P10/C00-TP2, and P11/C00 are absent.
- The capture validation is one ordered ON/OFF pair. Its padding endpoint is
  mechanism-valid, but performance deltas have no CI and p95 regressed.
- Layer 1 did not collect expert-route identities; kernel engineers cannot infer
  routed-expert imbalance from these artifacts.

## Verification and stop rules for Phase 4 work

Every proposed experiment keeps the original fixed manifest/seed/work, excludes
warm-up, records Layer-1 accounting, and changes one mechanism. A candidate
stops on clean failure, footer imbalance/drop, output mismatch, GPU
contamination, memory regression beyond its declared budget, or violation of
the 16-H20-hour campaign cap. Throughput, latency, memory, and correctness are
reported together; no metric shopping or silent pattern substitution is
allowed.

## GPU accounting

The optional capture pair consumed **0.296389 H20-hours**, taking cumulative
campaign use from 14.025875 to **14.322265 H20-hours**. Remaining headroom is
**1.677735 H20-hours**. Both arms returned GPU0 to zero, all eight GPUs were
0 MiB/0% at final inspection, and no other-user process appeared.

Artifacts are under
`runs/opprof-phase3/phase4/capture-p09/`; `result.json` SHA-256 is
`5bb91df28790f6f3c34e4e9ed8e35a1cb8100f93086a4286689d587fd732f2a4`.

## Final ranked one-liners

1. **Prefix affinity (config now):** P08's measured ceiling is **+82.14% req/s** with 62.29% fewer prefill tokens versus matched P07.
2. **Length-aware scheduler:** raggedness ceiling is **44.79 pp R64 / 44.69% efficiency gap** on P10; smaller confirmed bounds apply to P09/P06.
3. **Exact capture sizes (config now):** ceiling **5.26 pp P10 / 4.98 pp P09 padding**; P09 validation removed 4.980 pp but did not prove p95 gain.
4. **MNS64 pattern pools (config now):** measured ceiling **+3.70% req/s P10 / +3.37% P06**, with a −24.27% P01 counterexample.
5. **Ragged kernels (kernel backlog):** share rank 2's bound; no additive E2E bound is supported while H1a is inconclusive.
6. **MBT8192 guardrail (config now):** avoids measured regressions up to **11.64%**; MBT2048 has no general positive case.

## Data sanity block

| Numeric family | n | finite | missing | min | max | distinct | Invariant/result |
|---|---:|---:|---:|---:|---:|---:|---|
| Ranked items | 6 | 6 | 0 | rank 1 | rank 6 | 6 | Three tiers represented; bounds not summed |
| Sentinel saturation config deltas | 11 | 11 | 0 | −30.216% | +3.704% | 11 | Both gains and regressions retained |
| Passing R64 contrast effects | 5 | 5 | 0 | 0.230148 | 0.447872 | 4 | Ratios in [0,1]; duplicate P10 controls expected |
| Capture-arm padding fraction | 2 | 2 | 0 | 0.035659 | 0.085456 | 2 | Non-negative; ON < OFF |
| Capture-arm token efficiency | 2 | 2 | 0 | 4.554051 | 4.562222 | 2 | Positive; +0.179% ON |
| Capture-arm throughput (req/s) | 2 | 2 | 0 | 4.933333 | 4.958333 | 2 | Same offered rate 4.920833 req/s |
| Capture-arm clean failures | 2 | 2 | 0 | 0 | 0 | 1 expected | Exact 240 s and zero failures |
| Capture-arm Layer-1 records | 2 | 2 | 0 | 16,491 | 17,579 | 2 | Every footer/sidecar invariant true; zero drops |
| Optional validation GPU-hours | 1 | 1 | 0 | 0.296389 | 0.296389 | 1 | Positive; cumulative 14.322265 < 16 |
| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | Cleanup passed |

Checked invariants: Phase-3 metrics remain frozen; every cited number resolves
to accepted metrics or the checksum-recorded validation; config comparisons
use saturation rather than normalized moderate throughput; padding/raggedness
bounds are not presented as throughput; duplicate/non-independent bounds are
not added; both validation arms use identical work and offered rate; clean
failures are zero; output work, Layer-1 schema, step continuity, footer/sidecar
balance, and zero drops pass; ratios lie in their declared domains; all GPU
memory returned to zero; and cumulative GPU use stays below 16 H20-hours. No
data-sanity red flag remains.