aituner/docs/opprof/phase3-results.md

# OpProf Phase 3 dash0 results

Status: **FINAL A-P3-7 PARTIAL-MATRIX ANALYSIS — H1a INCONCLUSIVE; H1b PASS**.

Date: 2026-07-12. The matrix is permanently frozen at 40/52 accepted measured
runs and 20/24 complete pattern/config cells. A-P3-7 permits existential
confirmation on complete comparisons but forbids refutation from incomplete
coverage. The result is therefore **PARTIAL**: H1b is confirmed by five
evaluable contrasts, while H1a cannot be evaluated on a valid pattern pair.

The final machine result is
`runs/opprof-phase3/phase3/metrics.json` (SHA-256
`dcc44941bfe1eb56c7a069148e5565eddd1ccf966e6914aa16b64bd67d02f0f0`).
The final analyzer SHA-256 is
`483c170838a1e81a170d8425b4be22cc1be02ea4bc9492b474daf7ae86186d67`;
7/7 no-GPU analysis tests passed, including ResourceWarning-as-error.

## Data-sanity result first

All **23/23** machine-checked invariants pass before inference: 40 unique
accepted primary markers selected only through completed-stage records; exact
240-second clean windows; zero clean failures; finite moderate rates within
5%; balanced Layer-1 footer/sidecar accounting; ratios in range; 72 accepted
trace files with the registered eight-file first-wave deviation; zero other
GPU processes; exact 20-cell coverage; and the exact A-P3-7 missing-cell and
missing-contrast sets.

The accepted data contain 77,109 clean completions, 352,907 clean Layer-1
steps, and 542,350 total Layer-1 records. Device-time classifiability is
97.05–99.64%. No malformed trace, accounting mismatch, counter underflow,
private-text leak, or unexpected identical-result family was found.

Declared limitations are not hidden as sanity passes: eight accepted
saturation trace files were lost during the registered first-wave controller
cleanup; all four confirmation runs are absent; MoE per-layer CV is unavailable
because layer scopes cover less than 80% of MoE GEMM time; and only one of nine
completed C00-moderate patterns passes the Layer-2 inference gates.

## Frozen coverage and A-P3-7 logic

The four incomplete cells, each missing saturation and moderate, are exactly:

- P03/C11;
- P05/C00;
- P10/C00-TP2; and
- P11/C00.

The four missing confirmations are P10, P06, P03, and P01 C00-moderate.
Formal H1a coverage is therefore the nine completed C00 patterns P01, P02,
P03, P04, P06, P07, P08, P09, and P10. Formal H1b has six evaluable and two
non-evaluable frozen contrasts. A-P3-7's asymmetry is mandatory: one valid hit
can confirm an existential hypothesis, but absence of a hit cannot support the
original complete-matrix null.

## H1a — operator composition

**Verdict: INCONCLUSIVE; refutation is not allowed.**

All moderate traces were parseable and classifiable, but only P04's two
windows jointly passed classifiability, `abs(SMD)<=0.25` representativeness,
and fixed recovery. P01/P02/P03/P09 failed representativeness in both windows;
P06/P07/P08/P10 additionally failed at least one recovery. Thus only one of
nine completed patterns is inferentially evaluable, leaving zero pattern pairs,
zero ranking tests with data, and no Kendall tau-b pairs. No qualifying
inversion can be accepted, and this is not evidence for a shared ranking.

Descriptively, P04 is attention-led at both loads. P06 and P10 change from
attention-led at saturation to MoE-GEMM-led at moderate load; the other
available patterns remain MoE-GEMM-led. These are ranking changes in sampled
windows, not H1a evidence because their window gates fail.

### Per-pattern operator shares, both load points

Values are the mean percentage across the two Layer-2 windows. `E` means the
row is inferentially evaluable; `D` means data are available but descriptive
only because a registered window/recovery gate failed; `NE-cell` means the cell
is missing; `NE-trace` means the accepted run has no retained trace. `Other` is
unclassified device activity. Collective, sampler, and dense-GEMM shares round
to 0.00% in every available C00 row.

| Pattern | Load | Use | Attention | MoE GEMM | Router | Collective | Sampler | Dense GEMM | Norm/elt | KV | Other | Top family |
|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| P01 | saturation | D | 16.32 | 77.81 | 0.85 | 0.00 | 0.00 | 0.00 | 3.90 | 0.10 | 1.02 | MoE GEMM |
| P01 | moderate | D | 12.09 | 82.43 | 1.03 | 0.00 | 0.00 | 0.00 | 3.38 | 0.11 | 0.96 | MoE GEMM |
| P02 | saturation | D | 32.67 | 62.39 | 1.05 | 0.00 | 0.00 | 0.00 | 2.98 | 0.16 | 0.74 | MoE GEMM |
| P02 | moderate | D | 20.91 | 72.53 | 1.64 | 0.00 | 0.00 | 0.00 | 3.68 | 0.25 | 1.00 | MoE GEMM |
| P03 | saturation | D | 39.11 | 56.60 | 0.44 | 0.00 | 0.00 | 0.00 | 2.98 | 0.03 | 0.84 | MoE GEMM |
| P03 | moderate | D | 23.32 | 57.45 | 3.54 | 0.00 | 0.00 | 0.00 | 12.29 | 0.83 | 2.57 | MoE GEMM |
| P04 | saturation | D | 65.88 | 29.77 | 1.16 | 0.00 | 0.00 | 0.00 | 2.30 | 0.17 | 0.72 | Attention |
| P04 | moderate | **E** | 47.88 | 40.64 | 3.28 | 0.00 | 0.00 | 0.00 | 5.85 | 0.45 | 1.90 | Attention |
| P05 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — |
| P05 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — |
| P06 | saturation | D | 56.59 | 39.35 | 1.04 | 0.00 | 0.00 | 0.00 | 2.25 | 0.15 | 0.62 | Attention |
| P06 | moderate | D | 31.60 | 57.52 | 2.97 | 0.00 | 0.00 | 0.00 | 5.69 | 0.43 | 1.80 | MoE GEMM |
| P07 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — |
| P07 | moderate | D | 28.67 | 62.18 | 2.42 | 0.00 | 0.00 | 0.00 | 4.86 | 0.37 | 1.49 | MoE GEMM |
| P08 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — |
| P08 | moderate | D | 34.59 | 57.06 | 2.22 | 0.00 | 0.00 | 0.00 | 4.44 | 0.34 | 1.34 | MoE GEMM |
| P09 | saturation | D | 26.23 | 69.03 | 0.36 | 0.00 | 0.00 | 0.00 | 3.43 | 0.00 | 0.95 | MoE GEMM |
| P09 | moderate | D | 16.19 | 75.00 | 2.14 | 0.00 | 0.00 | 0.00 | 4.83 | 0.35 | 1.49 | MoE GEMM |
| P10 | saturation | D | 67.80 | 28.90 | 0.76 | 0.00 | 0.00 | 0.00 | 1.87 | 0.10 | 0.57 | Attention |
| P10 | moderate | D | 30.61 | 54.04 | 2.81 | 0.00 | 0.00 | 0.00 | 9.80 | 0.68 | 2.06 | MoE GEMM |
| P11 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — |
| P11 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — |

### Clean CUDA-graph modes at moderate load

| Pattern | FULL | PIECEWISE | NONE/eager |
|---|---:|---:|---:|
| P01 | 1.98% | 73.19% | 24.83% |
| P02 | 83.16% | 13.00% | 3.84% |
| P03 | 96.37% | 0.00% | 3.63% |
| P04 | 98.32% | 0.02% | 1.66% |
| P05 | — | — | — |
| P06 | 99.10% | 0.05% | 0.85% |
| P07 | 98.76% | 0.00% | 1.24% |
| P08 | 98.64% | 0.01% | 1.36% |
| P09 | 91.79% | 3.69% | 4.52% |
| P10 | 98.33% | 0.12% | 1.55% |
| P11 | — | — | — |

P01 is the clear mode outlier: PIECEWISE covers 73.19% of clean moderate
steps. Full per-mode operator segments and active-step counts remain in the
machine result; no mode with fewer than eight sampled steps is summarized.

## H1b — rectangular-benchmark miss

**Verdict: PASS.** Five of six evaluable frozen contrasts pass at least one
waste threshold with a simultaneous positive bound and at least 5% useful-token
efficiency loss. The two P05 contrasts are `NOT EVALUABLE`; no value is
imputed. Holm correction retains the original eight planned contrasts per
metric. Reported simultaneous intervals below are percentage-point effects;
passing bootstrap p-values are zero at 100,000-resample resolution.

| Frozen contrast | Status | Passing evidence | Useful-token efficiency loss |
|---|---|---|---:|
| P05 vs P01 | **NOT EVALUABLE** | P05/C00 missing | — |
| P05 vs P03 | **NOT EVALUABLE** | P05/C00 missing | — |
| P06 vs P02 | **PASS** | R64 +23.015 pp, simultaneous 95% [22.345, 23.683] | 11.609% |
| P06 vs P04 | **PASS** | R64 +35.440 pp [34.826, 36.059] | 22.846% |
| P09 vs P01 | **PASS** | padding +6.673 pp [5.887, 7.492]; R64 +39.616 pp [39.065, 40.166] | 8.325% |
| P09 vs P03 | EVALUABLE, NO HIT | padding +8.542 pp and R64 +52.041 pp are material, but association gate fails | **3.840%**, below 5% |
| P10 vs P03 | **PASS** | padding +5.565 pp [2.010, 10.341]; R64 +44.787 pp [43.635, 45.971] | 44.693% |
| P10 vs P04 | **PASS** | padding +5.399 pp [1.868, 10.213]; R64 +44.787 pp [43.638, 45.949] | 14.260% |

This confirms the existential H1b claim on completed contrasts. It cannot
support a null claim about P05 or the missing matrix.

## Waste accounting

The preregistered contrast thresholds are 5 percentage points for padding,
10 points for graph miss/overflow, 0.15 absolute for R64, 0.10 for mixed-batch
interference, and 0.15 for MoE layer CV, plus the 5% efficiency/residual
association gate. Values below are clean C00-moderate results; efficiency is
scheduled useful tokens per model-step millisecond.

| Pattern | Padding | Graph miss | Overflow | R64 | Mixed interference | Efficiency | Supported/total mixed steps |
|---|---:|---:|---:|---:|---|---:|---:|
| P01 | 1.869% | 24.826% | 24.826% | 0.3687 | N/A | 4.9673 | 0/5,646 |
| P02 | 1.998% | 3.837% | 3.837% | 0.3687 | N/A | 2.6664 | 0/1,536 |
| P03 | 0.000% | 1.736% | 1.736% | 0.2444 | N/A | 4.7356 | 0/138 |
| P04 | 0.167% | 1.301% | 1.301% | 0.2444 | N/A | 3.0547 | 0/141 |
| P05 | — | — | — | — | — | — | — |
| P06 | 0.276% | 0.830% | 0.830% | 0.5988 | N/A | 2.3568 | 0/152 |
| P07 | 0.140% | 1.240% | 1.240% | 0.0000 | N/A | 2.7513 | 0/186 |
| P08 | 0.005% | 1.357% | 1.357% | 0.0000 | N/A | 2.1478 | 0/178 |
| P09 | **8.542%** | 4.518% | 4.518% | **0.7648** | N/A | 4.5537 | 0/1,054 |
| P10 | **5.565%** | 0.921% | 0.921% | **0.6923** | N/A | 2.6191 | 0/92 |
| P11 | — | — | — | — | — | — | — |

No irregular-versus-control graph-miss or overflow contrast reaches the
positive 10-point H1b threshold. P01 has the largest absolute miss rate but is
a rectangular control; P09 actually reduces miss versus P01 by 20.308 points.

The registered leave-one-pattern-out robust fits were applied for mixed-batch
interference, but no pattern retained 30 supported mixed steps inside both
pure-fit convex supports; every result is N/A rather than extrapolated. No
separate LOAO operator-ranking procedure was preregistered, so none was added
post hoc. The required two-window rule is used unchanged, and confirmation-run
robustness is `NOT EVALUABLE` because all four confirmations are missing.

### Top three waste findings

1. **P10 real trace:** R64 is 44.787 points above both long rectangular
   controls; padding is 5.565/5.399 points higher; efficiency is 44.693% worse
   than P03 and 14.260% worse than P04.
2. **P09 production-shaped mix:** versus P01, R64 rises 39.616 points and
   padding 6.673 points with 8.325% lower efficiency. Its still-larger P03
   contrast does not pass because efficiency loss is only 3.840%.
3. **P06 bimodal long-output burst:** R64 rises 23.015 points versus P02 and
   35.440 points versus P04, paired with 11.609% and 22.846% efficiency loss.

## Pattern-conditioned operational findings

- **P10/TP2 non-stabilization:** the fresh run completed 17 warm-up requests,
  but trailing scheduled-token throughput fell 18,022.4 → 16,384.0 → 14,062.6
  tokens/s. Its 36.76% fitted drift exceeds A-P3-6's 10% limit; same-wave
  synthetic P11 and P03 completed 512 and 102 warm-up requests and passed their
  applicable registered gates. The orchestrator accepts this as a
  pattern-conditioned finding, not an accepted throughput measurement.
- **Long-context drain:** P10/C01 saturation naturally drained for 288.619
  seconds. The clean window and accounting were valid; the original 120-second
  short-pattern watchdog was miscalibrated, while the amended 600-second P10
  budget correctly retained the run.
- **Failure boundary:** P01/C01 moderate had 5,828 clean successes and zero
  clean failures. Two `ServerDisconnectedError` records occurred only in
  excluded post-profile time; separating clean and auxiliary windows prevented
  valid data from being discarded.
- **Layer-2 perturbation:** despite 97%+ kernel classifiability, eight of nine
  completed moderate patterns failed representativeness or recovery. Together
  with the Phase-2 51.3% active-window perturbation, this shows that sparse
  Kineto windows are operationally fragile for these serving patterns.

These incidents consistently show long-context and mixed serving patterns
breaking assumptions calibrated on shorter rectangular loads: fixed drain
watchdogs, completion-count warm-up, global failure scope, and profile-window
representativeness.

## Matrix and GPU accounting

| Item | Final value |
|---|---:|
| Planned measured runs | 52 |
| Accepted measured runs | **40** |
| Complete cells | **20/24** |
| Accepted confirmations | 0/4 |
| Accepted clean failures | 0 |
| Drain-quarantined accepted runs | 0 |
| Accepted Layer-2 trace files | 72 |
| Registered missing trace files | 8 |
| Other-user GPU processes | 0 |
| Cumulative GPU use | **14.025875 H20-hours** |
| Reserved/unused headroom | **1.974125 H20-hours** |

No GPU work was run for A-P3-7 analysis. The reserved headroom remains unused,
and all eight GPUs were at 0 MiB/0% before and after the CPU-only analysis.

## Data sanity block

| Numeric family | n | finite | missing | min | max | distinct | Check |
|---|---:|---:|---:|---:|---:|---:|---|
| Accepted throughput (req/s) | 40 | 40 | 0 | 0.445833 | 43.300000 | 37 | finite, positive, not identical |
| Token efficiency (tokens/ms) | 40 | 40 | 0 | 2.147827 | 8.295849 | 40 | finite, positive, distinct |
| Drain duration (s) | 40 | 40 | 0 | 0.305419 | 288.619108 | 40 | non-negative; no quarantine |
| Clean Layer-1 steps/run | 40 | 40 | 0 | 718 | 20,171 | 40 | sum 352,907; continuous per run |
| Operator classifiable fraction | 72 | 72 | 0 | 0.970458 | 0.996378 | 72 | all >=0.70 and within [0,1] |
| Operator-family shares | 576 | 576 | 0 | 0.000000 | 0.873417 | 348 | within [0,1] |
| Waste ratios | 240 | 240 | 0 | 0.000000 | 1.000000 | 168 | within [0,1] |
| Waste-contrast effects | 36 | 24 | 12 expected | -0.203082 | 0.520413 | 17 | signed; N/A preserved |
| Kendall tau-b | 0 | 0 | 0 | — | — | 0 | expected: only one valid pattern |
| Accepted clean failures | 40 | 40 | 0 | 0 | 0 | 1 expected | sum zero |
| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | cleanup passed |

Checked invariants: accepted IDs unique; exact 40-run/20-cell coverage; exact
four missing cells and two missing contrasts; 240-second clean windows; zero
clean failures; moderate rate within 5%; balanced Layer-1 accounting and zero
drops; trace parsing and classifiability; ratios in range; exact 72+8 trace
accounting; clock/load snapshots present; no other-user process; drain
quarantine below 20%; non-identical pattern results; and exact reproduction of
the 36.76% A-P3-6 operational result. **No data-sanity red flag remains.** H1a
is inconclusive because of declared Layer-2 validity limits; H1b passes on five
complete contrasts; the compound hypothesis is partial and cannot be refuted.