aituner/docs/opprof/phase4-optimization-plan.md

OpProf Phase 4 measured optimization plan

Status: PROPOSAL FROM ACCEPTED PHASE-3 DATA; NO UNMEASURED GAIN CLAIMS.

Date: 2026-07-12. This plan uses only the accepted Phase-3 40/52-run, 20/24-cell dataset and the single optional Phase-4 capture-size validation. Phase-3 protocol and results are frozen. Bounds below are ceilings in the units actually measured; padding or raggedness percentages are not relabeled as end-to-end throughput gains.

Pinned implementation context

• Model/hardware: Qwen3-30B-A3B BF16, one H20, TP1 primary.
• vLLM source: accepted OpProf tip 23450fb21ac255b0cf710f4ee965ee694921975d on v0.24.0.
• vLLM 0.24.0 exposes --cudagraph-capture-sizes, --max-cudagraph-capture-size, --max-num-seqs, and --max-num-batched-tokens (vllm/engine/arg_utils.py:1390-1467).
• An explicit capture-size list replaces the inferred list. The default is [1,2,4], multiples of eight below 256, then multiples of sixteen through the maximum, normally 512 (vllm/config/vllm.py:1669-1792).
• Chunked prefill remains enabled. vLLM schedules decode first, then fills the remaining MBT budget with prefill and chunks an over-budget prefill (docs/configuration/optimization.md:45-59).

The ranking is by measured opportunity, readiness, and downside together—not by a normalized composite score. Items that share the same raggedness bound are explicitly non-additive.

Ranked optimization list

Rank	Tier	Target and affected regime	Measured bound	Owner	Decision
1	Config now / scheduler backlog	Preserve prefix affinity and prefix caching for P08-like shared-prefix traffic	P08 vs matched P07: +82.14% saturation req/s, 80% prefix-hit ratio, 62.29% fewer prefill tokens	Serving/config owner now; cache-aware dispatch upstream	Deploy behind a workload classifier; do not apply to no-sharing traffic
2	Scheduler	Length-aware cohorting/admission for ragged P10/P09/P06	At most 44.79 pp R64 contrast and 44.69% measured efficiency gap on P10; 39.62 pp/8.32% on P09; 35.44 pp/22.85% on P06	Upstream vLLM scheduler	Highest structural backlog item; preserve fairness and arrival semantics
3	Config now	Add exact small CUDAGraph sizes for P09/P10-like moderate decode batches	Distributional bound 4.98 pp P09 / 5.26 pp P10 padding; P09 validation achieved 4.980 pp	Serving/config owner	Mechanism confirmed; canary only because p95 was +3.01% in one pair
4	Config now	Route P06/P10-like pools to MNS=64	Saturation throughput +3.37% P06 / +3.70% P10 versus C00	Serving/config owner	Pattern-specific trial only; same setting was −24.27% on P01
5	Kernel backlog	Ragged-aware MoE GEMM and attention over the measured shape stream	Shares the rank-2 ceiling; no independent additive gain. Descriptive MoE share is 54.04–75.00% on P10/P09/P06 moderate	Engine/kernel colleagues	Optimize exact weighted shapes below; require serving confirmation
6	Config guardrail	Keep MBT=8192 for short/high-throughput and long-prefill classes; do not globally set 2048	Avoided saturation regressions up to 11.64% P03, 6.53% P01, 5.82% P10	Serving/config owner	Encode as a policy guardrail, not a positive optimization claim

Tier A — configuration-level actions deployable today

A1. Prefix-affine routing with prefix caching

Measured regime. P07 and P08 have the same 1,280-token prompt length, 512-token output, burst-of-eight arrival, C00 config, and seed. P08 alone uses eight 1,024-token shared prefixes plus a unique 256-token suffix. With prefix caching already enabled:

Metric, saturation	P07 no sharing	P08 high sharing	Delta
Prefix-query hit ratio	0.00	0.80	+0.80
Clean prefill tokens	1,528,968	576,512	−62.29%
Completed throughput	5.1083 req/s	9.3042 req/s	+82.14%

Root mechanism. The matched cell changes only controlled prefix sharing; the observed hit and prefill-token changes directly identify cache reuse. The 82.14% throughput delta is a point observation and the maximum evidence-backed gain for this exact regime, not a fleet-wide forecast.

Action now. Keep --enable-prefix-caching; hash an application-known stable prefix or conversation identity to a replica so related requests do not destroy affinity through round-robin load balancing. Apply only when the online prefix-query hit ratio resembles P08, not P07.

Verification. Interleave affinity ON/OFF on the same replicas and fixed request stream. Primary gates are prefix-hit ratio, prefill-token reduction, completed req/s, TTFT p95, per-replica queue imbalance, and KV occupancy. A throughput gain with queue/fairness or KV-capacity regression does not pass.

A2. Measured CUDAGraph capture sizes

Measured regime. P09 moderate has decode-batch p50/p95/max 4/16/25. P10 moderate has 1/4/7. Default captures skip sizes 3, 5, 6, 7, and 9, so those batches pad upward. Replaying the Phase-3 hit distribution predicts that adding exactly {3,5,6,7,9} to the complete default list can remove:

• 4.9776 percentage points of P09's 8.5421% graph-hit padding; and
• 5.2579 points of P10's 5.5652% padding.

The list must be default ∪ {3,5,6,7,9}; passing only five sizes would replace and discard the rest of the default list.

Closed-loop result. One P09 moderate ON/OFF pair used fresh servers, the same fixed seed and accepted saturation-rate source, 60 excluded warm-up seconds, and 240 clean seconds per arm. No Layer-2 profiler ran.

Metric	ON: exact sizes	OFF: default	ON relative to OFF
Graph-hit padding	3.5659%	8.5456%	−4.9798 pp; −58.27%
Useful tokens/model-step ms	4.56222	4.55405	+0.179%
Completed throughput	4.9583 req/s	4.9333 req/s	+0.507%
Mean E2E latency	1.6123 s	1.6812 s	−4.10%
p95 E2E latency	3.9930 s	3.8763 s	+3.01%
Clean failures	0	0	equal

The observed padding reduction differs from the Phase-3 bound by only 0.0022 percentage points, confirming the mechanism. It does not establish a broad performance win: token efficiency and throughput moved less than 1%, p95 moved the wrong way, and there is one ordered pair with no CI.

Operational cost also matters: ON captured 56 FULL and 56 PIECEWISE sizes versus 51/51 OFF, estimated graph memory increased 0.64→0.68 GiB, and server ownership was 76.9 seconds longer. Deploy only as a pattern-specific canary; require interleaved replication with a p95 non-regression gate before rollout.

A3. Pattern-specific MNS pools

The complete saturation comparisons show that --max-num-seqs 64 is an interaction, not a globally better default:

Pattern	C10 MNS=64 vs C00	Interpretation
P01 short/short	−24.27% req/s	Reject for dense short traffic
P03 long/short	−1.41%	No measured benefit
P06 bimodal/long burst	+3.37%	Candidate pattern pool
P10 real long-context	+3.70%	Candidate pattern pool

The evidence identifies the config×pattern interaction but not a lower-level cause. Do not attribute it to a particular kernel or queue effect without a new bisection. Verification is five interleaved saturation pairs per intended class plus TTFT, queue depth, preemption, KV usage, and exact-work checks.

A4. MBT policy guardrail

--max-num-batched-tokens 2048 versus the default 8192 changed saturation throughput by −6.53% P01, −11.64% P03, +0.35% P06, and −5.82% P10. The combined MNS64/MBT2048 setting was −30.22% on P01. Phase 3 therefore supports retaining MBT8192 for these classes and rejects a global MBT2048 rollout. The bound is an avoided regression, not new speedup.

Tier B — upstream scheduler changes

B1. Length-aware cohorting without starvation

Measured mechanism. R64 is the rectangular padding fraction of the exact arrival-order prompt stream. It is 0.6923 for P10, 0.7648 for P09, and 0.5988 for P06. Their passing control contrasts are:

Irregular pattern	Control	R64 excess	Useful-token efficiency loss
P10	P03	44.79 pp	44.69%
P10	P04	44.79 pp	14.26%
P09	P01	39.62 pp	8.32%
P06	P02	23.01 pp	11.61%
P06	P04	35.44 pp	22.85%

These are upper bounds on waste a length-aware path could avoid. R64 is not observed GPU time, and efficiency association is not causal. The scheduler change should maintain several ready queues by remaining prompt/context band, select a less-ragged cohort subject to the existing decode-first token budget, and impose a finite age/fairness bound. It must not rewrite request arrivals or drop long requests.

Verification. Add a runtime per-step raggedness counter rather than using manifest R64 as a surrogate. Compare fixed-arrival ON/OFF runs for useful tokens/model-step ms, TTFT/E2E p95, queue age, starvation count, preemption, KV occupancy, and the full length histogram. The gain cannot exceed the corresponding R64/efficiency bounds above, and it is non-additive with a ragged-aware kernel.

B2. Automatic cache-aware dispatch

The upstream form of A1 is a scheduler/replica dispatcher that chooses a live prefix-cache owner while respecting load. Its collaboration contract is the measured P07/P08 tuple: 1,024 shared + 256 unique prompt tokens, eight prefix IDs, burst size eight, output 512, target hit ratio 0.80. The load-balancing penalty and lost cache hits must be reported together; a synthetic cache hit increase without end-to-end balance is insufficient.

B3. Histogram-driven capture-list generation

Static A2 proves that Layer-1 can choose useful sizes. An upstream controller could select a bounded number of exact sizes from padding contribution count(size) * (next_bucket-size), while retaining the default list and a memory/startup budget. P09's top five {3,5,6,7,9} recovered 4.98 points at a 0.04-GiB graph-memory and 76.9-second server-lifetime cost in this run. The selector must freeze its list before measurement and never continually tune on the scored window.

Tier C — kernel-level backlog and collaboration interface

H1a is inconclusive, so Phase 3 does not prove a universal top operator. The available moderate windows are still useful shape inputs: descriptive MoE-GEMM shares are 57.52% P06, 75.00% P09, and 54.04% P10; attention shares are 31.60%, 16.19%, and 30.61%. Only P04's operator windows pass inference gates, where attention is 47.88% and MoE GEMM 40.64%. Kernel work must therefore claim shape-local improvement, not a resolved global bottleneck.

Exact shape stream for kernel engineers

P, D, and N below are per-step prefill tokens, decode tokens, and scheduled requests. Counts are from clean C00-moderate Layer-1 records. Context mix is the fraction of scheduled request-context observations in <=1024 / 1025–8192 / 8193–32768 / >32768 bins.

Pattern	Model steps: decode / mixed / prefill	N p50 / p95 / max	Dominant exact (P,D,N): count	Context mix	Chunk signal
P01	5,760: 114 / 5,646 / 0	69 / 74 / 77	(0,66,66):18, (0,67,67):18; mixed P p50=334, D p50=68	100.00 / 0 / 0 / 0%	6,235 unsplit; sizes 129–512 dominate
P04	12,455: 12,291 / 141 / 23	8 / 8 / 16	(0,8,8):12,129, (8191,1,3):23	0.03 / 94.62 / 5.35 / 0%	238/319 chunks >2,048; first/final 122/123
P06	17,464: 17,310 / 152 / 2	8 / 16 / 16	(0,8,8):13,103, (0,16,16):4,098	55.11 / 42.93 / 1.96 / 0%	174/422 >2,048; 136 in 257–512
P09	12,837: 11,783 / 1,054 / 0	4 / 16 / 25	(0,3,3):3,397, (0,2,2):1,549, (0,4,4):1,526, (0,5,5):1,050	51.70 / 48.26 / 0.04 / 0%	228/1,193 >2,048; 345 in 1,025–2,048
P10	18,130: 17,941 / 92 / 97	1 / 4 / 7	(0,1,1):13,572, (0,2,2):2,675, (0,3,3):792, (0,4,4):524, (8192,0,1):38	12.25 / 39.94 / 47.76 / 0.05%	138/191 >2,048; first/middle/final/unsplit 49/29/49/64

The exported kernel-benchmark interface should be a prompt-free weighted table with (P,D,N,context_bin,chunk_class,chunk_size_bin,runtime_mode,count) plus step duration and useful tokens. Use the frozen histogram edges already emitted by Layer 1: context 128..131072 powers of two and chunk 16..2048 powers of two. Preserve the joint tuples; independent marginal sampling would erase the mixed-batch structure.

Kernel targets and acceptance

1. Ragged MoE GEMM: accept variable token counts without padding every expert/layer tile to the largest sequence. Weight microbenchmarks by the P06, P09, and P10 tuples above. The ceiling is the same R64/efficiency opportunity as B1, not an additional gain.
2. Attention: retain P04 (D,N)=(8,8) as the valid long rectangular control and test P10's mostly 1–4 decode batches plus 8,192-token chunks. Report useful-token time, workspace, and graph compatibility.
3. Serving confirmation: kernel time must improve on the exact weighted stream, then pass a fixed-arrival serving A/B for throughput, TTFT/p95, memory, and correctness. A rectangular-only kernel win does not close the Phase-3 finding.

moe_expert_load was unavailable in Phase 3. No expert-imbalance mechanism or gain is claimed; expert-specific packing requires a new low-overhead route histogram before implementation.

Honest limits and Phase-5 measurement requirement

• H1a remains inconclusive: only P04 had two representative/recovered operator windows. Eight other completed moderate patterns failed window validity even though kernel classifiability was 97.05–99.64%.
• A Phase 5 operator study needs longer, time-stratified samples that reproduce clean scheduled-token, prefill-fraction, decode-batch, and graph-mode distributions, plus a lower-perturbation per-op timer. It must demonstrate overhead before using shares for optimization; the Phase-2 Kineto active window perturbed throughput by 51.3%.
• Confirmation runs are absent. The MNS and MBT config effects are single-run point estimates and require replication before production decisions.
• R64 is an offline rectangular-padding upper bound, not measured GPU idle time. H1b's efficiency association does not establish causality.
• Mixed-batch interference was N/A because no cell retained 30 supported mixed steps inside both leave-one-pattern-out pure-fit supports.
• Results cover one model, BF16, H20, mostly TP1, and 20/24 cells. P03/C11, P05/C00, P10/C00-TP2, and P11/C00 are absent.
• The capture validation is one ordered ON/OFF pair. Its padding endpoint is mechanism-valid, but performance deltas have no CI and p95 regressed.
• Layer 1 did not collect expert-route identities; kernel engineers cannot infer routed-expert imbalance from these artifacts.

Verification and stop rules for Phase 4 work

Every proposed experiment keeps the original fixed manifest/seed/work, excludes warm-up, records Layer-1 accounting, and changes one mechanism. A candidate stops on clean failure, footer imbalance/drop, output mismatch, GPU contamination, memory regression beyond its declared budget, or violation of the 16-H20-hour campaign cap. Throughput, latency, memory, and correctness are reported together; no metric shopping or silent pattern substitution is allowed.

GPU accounting

The optional capture pair consumed 0.296389 H20-hours, taking cumulative campaign use from 14.025875 to 14.322265 H20-hours. Remaining headroom is 1.677735 H20-hours. Both arms returned GPU0 to zero, all eight GPUs were 0 MiB/0% at final inspection, and no other-user process appeared.

Artifacts are under runs/opprof-phase3/phase4/capture-p09/; result.json SHA-256 is 5bb91df28790f6f3c34e4e9ed8e35a1cb8100f93086a4286689d587fd732f2a4.

Final ranked one-liners

1. Prefix affinity (config now): P08's measured ceiling is +82.14% req/s with 62.29% fewer prefill tokens versus matched P07.
2. Length-aware scheduler: raggedness ceiling is 44.79 pp R64 / 44.69% efficiency gap on P10; smaller confirmed bounds apply to P09/P06.
3. Exact capture sizes (config now): ceiling 5.26 pp P10 / 4.98 pp P09 padding; P09 validation removed 4.980 pp but did not prove p95 gain.
4. MNS64 pattern pools (config now): measured ceiling +3.70% req/s P10 / +3.37% P06, with a −24.27% P01 counterexample.
5. Ragged kernels (kernel backlog): share rank 2's bound; no additive E2E bound is supported while H1a is inconclusive.
6. MBT8192 guardrail (config now): avoids measured regressions up to 11.64%; MBT2048 has no general positive case.

Data sanity block

Numeric family	n	finite	missing	min	max	distinct	Invariant/result
Ranked items	6	6	0	rank 1	rank 6	6	Three tiers represented; bounds not summed
Sentinel saturation config deltas	11	11	0	−30.216%	+3.704%	11	Both gains and regressions retained
Passing R64 contrast effects	5	5	0	0.230148	0.447872	4	Ratios in [0,1]; duplicate P10 controls expected
Capture-arm padding fraction	2	2	0	0.035659	0.085456	2	Non-negative; ON < OFF
Capture-arm token efficiency	2	2	0	4.554051	4.562222	2	Positive; +0.179% ON
Capture-arm throughput (req/s)	2	2	0	4.933333	4.958333	2	Same offered rate 4.920833 req/s
Capture-arm clean failures	2	2	0	0	0	1 expected	Exact 240 s and zero failures
Capture-arm Layer-1 records	2	2	0	16,491	17,579	2	Every footer/sidecar invariant true; zero drops
Optional validation GPU-hours	1	1	0	0.296389	0.296389	1	Positive; cumulative 14.322265 < 16
Final GPU memory (MiB)	8	8	0	0	0	1 expected	Cleanup passed

Checked invariants: Phase-3 metrics remain frozen; every cited number resolves to accepted metrics or the checksum-recorded validation; config comparisons use saturation rather than normalized moderate throughput; padding/raggedness bounds are not presented as throughput; duplicate/non-independent bounds are not added; both validation arms use identical work and offered rate; clean failures are zero; output work, Layer-1 schema, step continuity, footer/sidecar balance, and zero drops pass; ratios lie in their declared domains; all GPU memory returned to zero; and cumulative GPU use stays below 16 H20-hours. No data-sanity red flag remains.