Gahow Wang
d5b276180d
Workload-conditioned operator profiling on patched vLLM 0.24.0 + Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64 raggedness, 8-45% token-efficiency loss vs rectangular controls); mechanism decomposition kills the padding narrative and finds the arrival-uniformization artifact (-12.9%); cross-version churn surface shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held. Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer sidecars and metrics are tracked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
15 KiB
OpProf Phase 3 dash0 results
Status: FINAL A-P3-7 PARTIAL-MATRIX ANALYSIS — H1a INCONCLUSIVE; H1b PASS.
Date: 2026-07-12. The matrix is permanently frozen at 40/52 accepted measured runs and 20/24 complete pattern/config cells. A-P3-7 permits existential confirmation on complete comparisons but forbids refutation from incomplete coverage. The result is therefore PARTIAL: H1b is confirmed by five evaluable contrasts, while H1a cannot be evaluated on a valid pattern pair.
The final machine result is
runs/opprof-phase3/phase3/metrics.json (SHA-256
dcc44941bfe1eb56c7a069148e5565eddd1ccf966e6914aa16b64bd67d02f0f0).
The final analyzer SHA-256 is
483c170838a1e81a170d8425b4be22cc1be02ea4bc9492b474daf7ae86186d67;
7/7 no-GPU analysis tests passed, including ResourceWarning-as-error.
Data-sanity result first
All 23/23 machine-checked invariants pass before inference: 40 unique accepted primary markers selected only through completed-stage records; exact 240-second clean windows; zero clean failures; finite moderate rates within 5%; balanced Layer-1 footer/sidecar accounting; ratios in range; 72 accepted trace files with the registered eight-file first-wave deviation; zero other GPU processes; exact 20-cell coverage; and the exact A-P3-7 missing-cell and missing-contrast sets.
The accepted data contain 77,109 clean completions, 352,907 clean Layer-1 steps, and 542,350 total Layer-1 records. Device-time classifiability is 97.05–99.64%. No malformed trace, accounting mismatch, counter underflow, private-text leak, or unexpected identical-result family was found.
Declared limitations are not hidden as sanity passes: eight accepted saturation trace files were lost during the registered first-wave controller cleanup; all four confirmation runs are absent; MoE per-layer CV is unavailable because layer scopes cover less than 80% of MoE GEMM time; and only one of nine completed C00-moderate patterns passes the Layer-2 inference gates.
Frozen coverage and A-P3-7 logic
The four incomplete cells, each missing saturation and moderate, are exactly:
- P03/C11;
- P05/C00;
- P10/C00-TP2; and
- P11/C00.
The four missing confirmations are P10, P06, P03, and P01 C00-moderate. Formal H1a coverage is therefore the nine completed C00 patterns P01, P02, P03, P04, P06, P07, P08, P09, and P10. Formal H1b has six evaluable and two non-evaluable frozen contrasts. A-P3-7's asymmetry is mandatory: one valid hit can confirm an existential hypothesis, but absence of a hit cannot support the original complete-matrix null.
H1a — operator composition
Verdict: INCONCLUSIVE; refutation is not allowed.
All moderate traces were parseable and classifiable, but only P04's two
windows jointly passed classifiability, abs(SMD)<=0.25 representativeness,
and fixed recovery. P01/P02/P03/P09 failed representativeness in both windows;
P06/P07/P08/P10 additionally failed at least one recovery. Thus only one of
nine completed patterns is inferentially evaluable, leaving zero pattern pairs,
zero ranking tests with data, and no Kendall tau-b pairs. No qualifying
inversion can be accepted, and this is not evidence for a shared ranking.
Descriptively, P04 is attention-led at both loads. P06 and P10 change from attention-led at saturation to MoE-GEMM-led at moderate load; the other available patterns remain MoE-GEMM-led. These are ranking changes in sampled windows, not H1a evidence because their window gates fail.
Per-pattern operator shares, both load points
Values are the mean percentage across the two Layer-2 windows. E means the
row is inferentially evaluable; D means data are available but descriptive
only because a registered window/recovery gate failed; NE-cell means the cell
is missing; NE-trace means the accepted run has no retained trace. Other is
unclassified device activity. Collective, sampler, and dense-GEMM shares round
to 0.00% in every available C00 row.
| Pattern | Load | Use | Attention | MoE GEMM | Router | Collective | Sampler | Dense GEMM | Norm/elt | KV | Other | Top family |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| P01 | saturation | D | 16.32 | 77.81 | 0.85 | 0.00 | 0.00 | 0.00 | 3.90 | 0.10 | 1.02 | MoE GEMM |
| P01 | moderate | D | 12.09 | 82.43 | 1.03 | 0.00 | 0.00 | 0.00 | 3.38 | 0.11 | 0.96 | MoE GEMM |
| P02 | saturation | D | 32.67 | 62.39 | 1.05 | 0.00 | 0.00 | 0.00 | 2.98 | 0.16 | 0.74 | MoE GEMM |
| P02 | moderate | D | 20.91 | 72.53 | 1.64 | 0.00 | 0.00 | 0.00 | 3.68 | 0.25 | 1.00 | MoE GEMM |
| P03 | saturation | D | 39.11 | 56.60 | 0.44 | 0.00 | 0.00 | 0.00 | 2.98 | 0.03 | 0.84 | MoE GEMM |
| P03 | moderate | D | 23.32 | 57.45 | 3.54 | 0.00 | 0.00 | 0.00 | 12.29 | 0.83 | 2.57 | MoE GEMM |
| P04 | saturation | D | 65.88 | 29.77 | 1.16 | 0.00 | 0.00 | 0.00 | 2.30 | 0.17 | 0.72 | Attention |
| P04 | moderate | E | 47.88 | 40.64 | 3.28 | 0.00 | 0.00 | 0.00 | 5.85 | 0.45 | 1.90 | Attention |
| P05 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — |
| P05 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — |
| P06 | saturation | D | 56.59 | 39.35 | 1.04 | 0.00 | 0.00 | 0.00 | 2.25 | 0.15 | 0.62 | Attention |
| P06 | moderate | D | 31.60 | 57.52 | 2.97 | 0.00 | 0.00 | 0.00 | 5.69 | 0.43 | 1.80 | MoE GEMM |
| P07 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — |
| P07 | moderate | D | 28.67 | 62.18 | 2.42 | 0.00 | 0.00 | 0.00 | 4.86 | 0.37 | 1.49 | MoE GEMM |
| P08 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — |
| P08 | moderate | D | 34.59 | 57.06 | 2.22 | 0.00 | 0.00 | 0.00 | 4.44 | 0.34 | 1.34 | MoE GEMM |
| P09 | saturation | D | 26.23 | 69.03 | 0.36 | 0.00 | 0.00 | 0.00 | 3.43 | 0.00 | 0.95 | MoE GEMM |
| P09 | moderate | D | 16.19 | 75.00 | 2.14 | 0.00 | 0.00 | 0.00 | 4.83 | 0.35 | 1.49 | MoE GEMM |
| P10 | saturation | D | 67.80 | 28.90 | 0.76 | 0.00 | 0.00 | 0.00 | 1.87 | 0.10 | 0.57 | Attention |
| P10 | moderate | D | 30.61 | 54.04 | 2.81 | 0.00 | 0.00 | 0.00 | 9.80 | 0.68 | 2.06 | MoE GEMM |
| P11 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — |
| P11 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — |
Clean CUDA-graph modes at moderate load
| Pattern | FULL | PIECEWISE | NONE/eager |
|---|---|---|---|
| P01 | 1.98% | 73.19% | 24.83% |
| P02 | 83.16% | 13.00% | 3.84% |
| P03 | 96.37% | 0.00% | 3.63% |
| P04 | 98.32% | 0.02% | 1.66% |
| P05 | — | — | — |
| P06 | 99.10% | 0.05% | 0.85% |
| P07 | 98.76% | 0.00% | 1.24% |
| P08 | 98.64% | 0.01% | 1.36% |
| P09 | 91.79% | 3.69% | 4.52% |
| P10 | 98.33% | 0.12% | 1.55% |
| P11 | — | — | — |
P01 is the clear mode outlier: PIECEWISE covers 73.19% of clean moderate steps. Full per-mode operator segments and active-step counts remain in the machine result; no mode with fewer than eight sampled steps is summarized.
H1b — rectangular-benchmark miss
Verdict: PASS. Five of six evaluable frozen contrasts pass at least one
waste threshold with a simultaneous positive bound and at least 5% useful-token
efficiency loss. The two P05 contrasts are NOT EVALUABLE; no value is
imputed. Holm correction retains the original eight planned contrasts per
metric. Reported simultaneous intervals below are percentage-point effects;
passing bootstrap p-values are zero at 100,000-resample resolution.
| Frozen contrast | Status | Passing evidence | Useful-token efficiency loss |
|---|---|---|---|
| P05 vs P01 | NOT EVALUABLE | P05/C00 missing | — |
| P05 vs P03 | NOT EVALUABLE | P05/C00 missing | — |
| P06 vs P02 | PASS | R64 +23.015 pp, simultaneous 95% [22.345, 23.683] | 11.609% |
| P06 vs P04 | PASS | R64 +35.440 pp [34.826, 36.059] | 22.846% |
| P09 vs P01 | PASS | padding +6.673 pp [5.887, 7.492]; R64 +39.616 pp [39.065, 40.166] | 8.325% |
| P09 vs P03 | EVALUABLE, NO HIT | padding +8.542 pp and R64 +52.041 pp are material, but association gate fails | 3.840%, below 5% |
| P10 vs P03 | PASS | padding +5.565 pp [2.010, 10.341]; R64 +44.787 pp [43.635, 45.971] | 44.693% |
| P10 vs P04 | PASS | padding +5.399 pp [1.868, 10.213]; R64 +44.787 pp [43.638, 45.949] | 14.260% |
This confirms the existential H1b claim on completed contrasts. It cannot support a null claim about P05 or the missing matrix.
Waste accounting
The preregistered contrast thresholds are 5 percentage points for padding, 10 points for graph miss/overflow, 0.15 absolute for R64, 0.10 for mixed-batch interference, and 0.15 for MoE layer CV, plus the 5% efficiency/residual association gate. Values below are clean C00-moderate results; efficiency is scheduled useful tokens per model-step millisecond.
| Pattern | Padding | Graph miss | Overflow | R64 | Mixed interference | Efficiency | Supported/total mixed steps |
|---|---|---|---|---|---|---|---|
| P01 | 1.869% | 24.826% | 24.826% | 0.3687 | N/A | 4.9673 | 0/5,646 |
| P02 | 1.998% | 3.837% | 3.837% | 0.3687 | N/A | 2.6664 | 0/1,536 |
| P03 | 0.000% | 1.736% | 1.736% | 0.2444 | N/A | 4.7356 | 0/138 |
| P04 | 0.167% | 1.301% | 1.301% | 0.2444 | N/A | 3.0547 | 0/141 |
| P05 | — | — | — | — | — | — | — |
| P06 | 0.276% | 0.830% | 0.830% | 0.5988 | N/A | 2.3568 | 0/152 |
| P07 | 0.140% | 1.240% | 1.240% | 0.0000 | N/A | 2.7513 | 0/186 |
| P08 | 0.005% | 1.357% | 1.357% | 0.0000 | N/A | 2.1478 | 0/178 |
| P09 | 8.542% | 4.518% | 4.518% | 0.7648 | N/A | 4.5537 | 0/1,054 |
| P10 | 5.565% | 0.921% | 0.921% | 0.6923 | N/A | 2.6191 | 0/92 |
| P11 | — | — | — | — | — | — | — |
No irregular-versus-control graph-miss or overflow contrast reaches the positive 10-point H1b threshold. P01 has the largest absolute miss rate but is a rectangular control; P09 actually reduces miss versus P01 by 20.308 points.
The registered leave-one-pattern-out robust fits were applied for mixed-batch
interference, but no pattern retained 30 supported mixed steps inside both
pure-fit convex supports; every result is N/A rather than extrapolated. No
separate LOAO operator-ranking procedure was preregistered, so none was added
post hoc. The required two-window rule is used unchanged, and confirmation-run
robustness is NOT EVALUABLE because all four confirmations are missing.
Top three waste findings
- P10 real trace: R64 is 44.787 points above both long rectangular controls; padding is 5.565/5.399 points higher; efficiency is 44.693% worse than P03 and 14.260% worse than P04.
- P09 production-shaped mix: versus P01, R64 rises 39.616 points and padding 6.673 points with 8.325% lower efficiency. Its still-larger P03 contrast does not pass because efficiency loss is only 3.840%.
- P06 bimodal long-output burst: R64 rises 23.015 points versus P02 and 35.440 points versus P04, paired with 11.609% and 22.846% efficiency loss.
Pattern-conditioned operational findings
- P10/TP2 non-stabilization: the fresh run completed 17 warm-up requests, but trailing scheduled-token throughput fell 18,022.4 → 16,384.0 → 14,062.6 tokens/s. Its 36.76% fitted drift exceeds A-P3-6's 10% limit; same-wave synthetic P11 and P03 completed 512 and 102 warm-up requests and passed their applicable registered gates. The orchestrator accepts this as a pattern-conditioned finding, not an accepted throughput measurement.
- Long-context drain: P10/C01 saturation naturally drained for 288.619 seconds. The clean window and accounting were valid; the original 120-second short-pattern watchdog was miscalibrated, while the amended 600-second P10 budget correctly retained the run.
- Failure boundary: P01/C01 moderate had 5,828 clean successes and zero
clean failures. Two
ServerDisconnectedErrorrecords occurred only in excluded post-profile time; separating clean and auxiliary windows prevented valid data from being discarded. - Layer-2 perturbation: despite 97%+ kernel classifiability, eight of nine completed moderate patterns failed representativeness or recovery. Together with the Phase-2 51.3% active-window perturbation, this shows that sparse Kineto windows are operationally fragile for these serving patterns.
These incidents consistently show long-context and mixed serving patterns breaking assumptions calibrated on shorter rectangular loads: fixed drain watchdogs, completion-count warm-up, global failure scope, and profile-window representativeness.
Matrix and GPU accounting
| Item | Final value |
|---|---|
| Planned measured runs | 52 |
| Accepted measured runs | 40 |
| Complete cells | 20/24 |
| Accepted confirmations | 0/4 |
| Accepted clean failures | 0 |
| Drain-quarantined accepted runs | 0 |
| Accepted Layer-2 trace files | 72 |
| Registered missing trace files | 8 |
| Other-user GPU processes | 0 |
| Cumulative GPU use | 14.025875 H20-hours |
| Reserved/unused headroom | 1.974125 H20-hours |
No GPU work was run for A-P3-7 analysis. The reserved headroom remains unused, and all eight GPUs were at 0 MiB/0% before and after the CPU-only analysis.
Data sanity block
| Numeric family | n | finite | missing | min | max | distinct | Check |
|---|---|---|---|---|---|---|---|
| Accepted throughput (req/s) | 40 | 40 | 0 | 0.445833 | 43.300000 | 37 | finite, positive, not identical |
| Token efficiency (tokens/ms) | 40 | 40 | 0 | 2.147827 | 8.295849 | 40 | finite, positive, distinct |
| Drain duration (s) | 40 | 40 | 0 | 0.305419 | 288.619108 | 40 | non-negative; no quarantine |
| Clean Layer-1 steps/run | 40 | 40 | 0 | 718 | 20,171 | 40 | sum 352,907; continuous per run |
| Operator classifiable fraction | 72 | 72 | 0 | 0.970458 | 0.996378 | 72 | all >=0.70 and within [0,1] |
| Operator-family shares | 576 | 576 | 0 | 0.000000 | 0.873417 | 348 | within [0,1] |
| Waste ratios | 240 | 240 | 0 | 0.000000 | 1.000000 | 168 | within [0,1] |
| Waste-contrast effects | 36 | 24 | 12 expected | -0.203082 | 0.520413 | 17 | signed; N/A preserved |
| Kendall tau-b | 0 | 0 | 0 | — | — | 0 | expected: only one valid pattern |
| Accepted clean failures | 40 | 40 | 0 | 0 | 0 | 1 expected | sum zero |
| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | cleanup passed |
Checked invariants: accepted IDs unique; exact 40-run/20-cell coverage; exact four missing cells and two missing contrasts; 240-second clean windows; zero clean failures; moderate rate within 5%; balanced Layer-1 accounting and zero drops; trace parsing and classifiability; ratios in range; exact 72+8 trace accounting; clock/load snapshots present; no other-user process; drain quarantine below 20%; non-identical pattern results; and exact reproduction of the 36.76% A-P3-6 operational result. No data-sanity red flag remains. H1a is inconclusive because of declared Layer-2 validity limits; H1b passes on five complete contrasts; the compound hypothesis is partial and cannot be refuted.