Add OpProf campaign: protocols, results, patches, run evidence (P0-P6)
Workload-conditioned operator profiling on patched vLLM 0.24.0 + Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64 raggedness, 8-45% token-efficiency loss vs rectangular controls); mechanism decomposition kills the padding narrative and finds the arrival-uniformization artifact (-12.9%); cross-version churn surface shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held. Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer sidecars and metrics are tracked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
223
docs/opprof/phase5-results.md
Normal file
223
docs/opprof/phase5-results.md
Normal file
@@ -0,0 +1,223 @@
|
||||
# OpProf Phase 5 dash0 results
|
||||
|
||||
Status: **FINAL — INCONCLUSIVE MECHANISM LEDGER; NO DOMINANCE CALL**.
|
||||
|
||||
Date: 2026-07-12. All 15 registered P10 primary runs completed with three
|
||||
replicates per arm and passed A-P5-1. The fresh A3 bridge passed, so the frozen
|
||||
P3 P03/P04 controls were reused. The ledger is nevertheless inconclusive:
|
||||
P04's reused-control denominator is unstable, and A2, A3, and A4 fail their
|
||||
predeclared manipulation checks. The protocol therefore permits one official
|
||||
share (A1 under P03), requires the other lines to be N/A, and forbids a
|
||||
dual-control dominance conclusion.
|
||||
|
||||
The machine result is `runs/opprof-phase5/phase5/metrics.json` (SHA-256
|
||||
`fe16934866806ac542f7974cac061beeb1f1c58f5bfb36698efd8aeaa99b8cb7`).
|
||||
The final analyzer SHA-256 is
|
||||
`8b0ff8658614227ee00cf8d10c82c4b5edf6f6210db172fd6efc9cc149e51e10`;
|
||||
its no-GPU tools and analysis tests pass.
|
||||
|
||||
This remains explicitly a **recorded-arrival P5 bridge ledger anchored to P3
|
||||
controls**, not a literal decomposition of P3's already-uniform P10 gap.
|
||||
|
||||
## Data-sanity result first
|
||||
|
||||
**RED FLAG: the P04 control denominator is unstable.** P5 recorded-arrival
|
||||
base has `E_token=3.013752`, while reused P3 P04 has `E_token=3.054704`, leaving
|
||||
only a `0.040952` gap with bootstrap 95% CI `[-0.649096, 0.752594]`; 45.353% of
|
||||
denominator draws are nonpositive. This violates every registered stability
|
||||
form: the CI contains zero and the nonpositive fraction exceeds 5%.
|
||||
|
||||
P03 remains stable: its gap is `1.721848`, CI `[1.459092, 1.988772]`, with zero
|
||||
nonpositive draws. All execution-validity checks pass: 15/15 primary runs,
|
||||
three replicates per arm, exact 240-second clean windows, zero clean failures,
|
||||
offered rates within 5%, continuous/balanced Layer-1 accounting, identical
|
||||
request IDs and token sums, all A-P5-1 cold-start gates, no drain quarantine,
|
||||
and GPU spend below the cap. The red flag is reported before the ledger and no
|
||||
conclusion is built on P04's raw ratio draws.
|
||||
|
||||
## Bridge and absolute efficiency
|
||||
|
||||
A3 reproduces the P3 uniform-arrival P10 base: `delta_E=+0.007041`, 95% CI
|
||||
`[-0.329799, 0.354859]`, or 0.269% absolute relative difference. The CI contains
|
||||
zero and the point difference is below 3%, so the bridge passes and P3 control
|
||||
reuse is valid under the frozen rule. Conditional control reruns were not
|
||||
launched.
|
||||
|
||||
| Arm | `E_token` | Bootstrap 95% CI | Delta from P5 base |
|
||||
|---|---:|---:|---:|
|
||||
| Base, recorded | 3.013752 | [2.754462, 3.267964] | — |
|
||||
| A1 length-binned | 3.078999 | [2.801659, 3.349307] | +0.065247 |
|
||||
| A2 capture sizes | 3.062128 | [2.802812, 3.314276] | +0.048376 |
|
||||
| A3 uniform arrival | 2.626154 | [2.451852, 2.792684] | -0.387598 |
|
||||
| A4 prefix cache off | 3.020959 | [2.760494, 3.274105] | +0.007207 |
|
||||
|
||||
## Manipulation checks and Holm family
|
||||
|
||||
| Arm | Manipulation result | Efficiency delta 95% CI | Raw p | Holm p |
|
||||
|---|---|---:|---:|---:|
|
||||
| A1 | **PASS:** R16 0.641744 -> 0.473409; max delay 62.743 s; prefix-hit-ratio delta -0.296 pp | [-0.309461, 0.442565] | 0.73108 | 1.00000 |
|
||||
| A2 | **FAIL:** support coverage 98.879% <99%; padding reduction 82.541% <90% | [-0.315361, 0.411446] | 0.79022 | 1.00000 |
|
||||
| A3 | **FAIL:** decode-B CV falls 0.8706 -> 0.6551, but waiting CV rises 6.0145 -> 6.8557 | [-0.693444, -0.078141] | 0.01384 | 0.05536 |
|
||||
| A4 | **FAIL:** disabled arm still reports 3,093,324 local prefix queries and 52,896 hits, not zero | [-0.357637, 0.369309] | 0.97230 | 1.00000 |
|
||||
|
||||
Per protocol, a failed manipulation makes that mechanism's official share N/A
|
||||
even if its raw efficiency contrast is nonzero. A3's uncorrected contrast is
|
||||
negative, and it also misses family-wise significance after Holm correction.
|
||||
|
||||
## A-P5-1 arrival evidence
|
||||
|
||||
The four originally invalidated arms supplied the evidence for A-P5-1. Under
|
||||
the superseded throughput-drift calculation, recorded base/A1/A4 span
|
||||
190.55%-216.50% normalized drift, while uniform A3 is 13.23%. This gap is
|
||||
retained as direct evidence that the old gate measured arrival shape in a
|
||||
rate-following experiment; it is not converted into an efficiency share.
|
||||
|
||||
All 15 replacement/continued primary runs pass the amended cold-start gate:
|
||||
25-28 warm-up completions, 12-16 completions with input at least 8192 tokens,
|
||||
all compile/capture first occurrences before clean, zero clean-window
|
||||
first-capture events, and zero uncovered Layer-1 graph descriptors. The 10%
|
||||
drift gate was retained unchanged for the P06 saturation background arm.
|
||||
|
||||
## Operational findings
|
||||
|
||||
- The detached controller completed three cache burn-ins, four four-way waves,
|
||||
all drains, validation, budget accounting, and cleanup. The four invalidated
|
||||
first-wave arms were rerun exactly; none of their superseded measurements
|
||||
enters the ledger.
|
||||
- The first exact rerun produced complete clean artifacts but hit a post-hoc
|
||||
validator `NameError` from a missing `math` import. Immutable artifacts were
|
||||
CPU-re-adjudicated under A-P5-1, the import was repaired, and no third GPU run
|
||||
was charged.
|
||||
- CPU analysis repairs admitted legitimate idle fixed-time blocks as `[0,0]`,
|
||||
supported the older P3 control schema, converted one NumPy boolean at the
|
||||
JSON boundary, and explicitly separated diagnostic from reportable shares.
|
||||
These changes alter no GPU data, estimator, seed, threshold, or resample
|
||||
count.
|
||||
- P10 drains range from 0.676 to 6.004 seconds. The separate P06 saturation
|
||||
background run drained naturally in 87.903 seconds, below the 600-second
|
||||
gate, with 564/564 clean completions and zero failures.
|
||||
- Public Phase-5 artifacts occupy 286,205,365 bytes (304 MiB), below the 3 GB
|
||||
stop threshold; CPFS retained 1,286 GB free.
|
||||
- MoE routing skew remains content-bound and receives no causal share. The
|
||||
optional routed-expert analysis-only run was not required for this primary
|
||||
ledger and was not launched.
|
||||
|
||||
## Launch echo
|
||||
|
||||
```text
|
||||
LAUNCH_ECHO utc=2026-07-12T12:07:54Z host=dash0 gpus=0-3 cpus=0-79 source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0@4b253fd manifests=/home/admin/cpfs/wjh/opprof-phase5-private/manifests outputs=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5 runs=3burnin+15primary+1background rate=0.4725 warmup=60s clean=240s drain=600s est_wall=35-60min est_gpu=1.7-2.1_H20h hard_cap=6.0_H20h conditional_controls=6_if_bridge_fails
|
||||
```
|
||||
|
||||
Resume after A-P5-1 reused the completed burn-ins and compile/cudagraph caches.
|
||||
The first launch through final controller completion took approximately 57.9
|
||||
minutes wall time, including the adjudication/resume interval.
|
||||
|
||||
## Run completion stats
|
||||
|
||||
| Item | Count/result |
|
||||
|---|---:|
|
||||
| Valid primary P10 runs | **15/15** |
|
||||
| Replicates | **3 per arm** |
|
||||
| Valid P06 background runs | 1 |
|
||||
| Completed burn-ins | 3 |
|
||||
| Superseded pre-amendment arms | 4 |
|
||||
| Conditional fresh controls | 0; bridge passed, P3 reused |
|
||||
| Optional secondary/routed-expert runs | 0 |
|
||||
| A-P5-1 primary passes | 15/15 |
|
||||
| Clean failures | 0 |
|
||||
| Drain quarantines | 0 |
|
||||
|
||||
## Mechanism ledger
|
||||
|
||||
Official shares follow the frozen N/A rules. Values in parentheses are raw
|
||||
diagnostic ratios retained for audit, not causal/reportable shares.
|
||||
|
||||
| Mechanism | P03 share (95% CI) | P04 share (95% CI) | Disposition |
|
||||
|---|---|---|---|
|
||||
| A1 length raggedness | **0.0379 [-0.2011, 0.2344]** | N/A; unstable denominator (diagnostic 1.5933 [-6.8925, 7.0636]) | Manipulation passes; no significant recovery |
|
||||
| A2 capture mismatch | N/A; manipulation failed (diagnostic 0.0281 [-0.2049, 0.2170]) | N/A; unstable denominator (diagnostic 1.1813 [-6.2871, 6.4924]) | Config did not meet 99%/90% removal gates |
|
||||
| A3 arrival dynamics | N/A; manipulation failed (diagnostic -0.2251 [-0.4625, -0.0403]) | N/A; unstable denominator (diagnostic -9.4647 [-17.2450, 17.4102]) | Queue-variance gate failed; Holm p=0.05536 |
|
||||
| A4 prefix structure | N/A; manipulation failed (diagnostic 0.0042 [-0.2340, 0.1948]) | N/A; unstable denominator (diagnostic 0.1760 [-6.2590, 6.4925]) | Zero-query/hit gate failed |
|
||||
|
||||
Shares are neither clipped nor renormalized. P04's extreme raw ratios are the
|
||||
registered consequence of dividing by a near-zero, sign-unstable denominator,
|
||||
not evidence of large mechanisms.
|
||||
|
||||
## Dominance verdicts
|
||||
|
||||
| Mechanism | P03 rule | P04 rule | Dual-control verdict |
|
||||
|---|---|---|---|
|
||||
| A1 | Fails share/CI/Holm thresholds | Not evaluable | **NOT EVALUABLE** |
|
||||
| A2 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
|
||||
| A3 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
|
||||
| A4 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
|
||||
|
||||
No mechanism may be called dominant because the frozen rule requires both
|
||||
control denominators. This is an inconclusive verdict, not evidence that every
|
||||
mechanism is small.
|
||||
|
||||
## Residual + interaction line
|
||||
|
||||
The official residual is N/A under both controls because the official share
|
||||
ledger is incomplete. The unchanged arithmetic over all raw diagnostic shares
|
||||
is reported only for transparency:
|
||||
|
||||
| Control | Diagnostic share sum | Diagnostic residual + interaction (95% CI) | Status |
|
||||
|---|---:|---:|---|
|
||||
| P03 | -0.1549 | 1.1549 [0.5536, 1.9397] | Diagnostic only |
|
||||
| P04 | -6.5142 | 7.5142 [-21.3965, 22.8767] | Diagnostic only; unstable denominator |
|
||||
|
||||
Nothing is forced to 100%. The residual can include interactions, unablated
|
||||
MoE routing, chunked-prefill effects, double-counting, and measurement error.
|
||||
|
||||
## Config-tier A2 measured delta
|
||||
|
||||
A2 added the P3-derived sizes `{3,5,6,7}` to the full default capture list.
|
||||
Its measured `delta_E` is **+0.048376**, 95% CI
|
||||
`[-0.315361, 0.411446]`, or **+1.605%** relative to base. Pure-decode padding
|
||||
falls from 13.7405% to 2.3989%, an 82.541% reduction. New recorded-arrival
|
||||
runs expose decode-B support 1-13, so uncaptured 9-13 leave coverage at 98.879%;
|
||||
both this and the sub-90% padding reduction invalidate the causal A2 share.
|
||||
|
||||
The config cost is reproducible: base graph capture is 11 seconds and 0.80 GiB
|
||||
in all three runs; A2 is 12-14 seconds and 0.84 GiB, a 1-3 second and 0.04 GiB
|
||||
increase. The measured config-tier result is therefore a small, statistically
|
||||
uncertain efficiency gain with a failed preregistered removal gate.
|
||||
|
||||
## GPU total
|
||||
|
||||
New Phase-5 spend is **2.4388167443 H20-hours** of the 6.0-hour cap, including
|
||||
the superseded first wave, exact reruns, continued primary matrix, burn-ins,
|
||||
and background arm. **3.5611832557 H20-hours remain unspent.** Final state on
|
||||
all eight H20s is 0 MiB and 0% utilization with zero compute processes.
|
||||
|
||||
## Sanity block
|
||||
|
||||
**Anomaly first:** P04's gap CI contains zero and 45.353% of its denominator
|
||||
draws are nonpositive; the dual-control ledger is inconclusive and inferential
|
||||
claims stop there.
|
||||
|
||||
| Numeric family | n | Finite | Missing | Min | Max | Distinct | Checked invariant/result |
|
||||
|---|---:|---:|---:|---:|---:|---:|---|
|
||||
| Primary run `E_token` | 15 | 15 | 0 | 2.623534 | 3.083263 | 15 | Positive; per-arm results not all identical |
|
||||
| Clean Layer-1 steps | 15 | 15 | 0 | 11,178 | 18,400 | 15 | Non-negative; continuous/balanced per run |
|
||||
| Clean duration (s) | 15 | 15 | 0 | 240 | 240 | 1 expected | Exact `3*80`; zero clean failures |
|
||||
| Offered rate (req/s) | 15 | 15 | 0 | 0.470833 | 0.487500 | 2 | Every run within 5% of 0.4725 |
|
||||
| Warm-up completions | 15 | 15 | 0 | 25 | 28 | 2 | Every run >=16 |
|
||||
| Warm-up long completions | 15 | 15 | 0 | 12 | 16 | 3 | Every run >=1 with input >=8192 |
|
||||
| Clean first-capture events | 15 | 15 | 0 | 0 | 0 | 1 expected | Zero events and zero uncovered descriptors |
|
||||
| P10 drain duration (s) | 15 | 15 | 0 | 0.676181 | 6.003959 | 15 | Non-negative; all below 600 |
|
||||
| Control gap `E_token` | 2 | 2 | 0 | 0.040952 | 1.721848 | 2 | P03 stable; **P04 unstable red flag** |
|
||||
| Raw diagnostic shares | 8 | 8 | 0 | -9.464746 | 1.593268 | 8 | Finite; intentionally not constrained to [0,1] |
|
||||
| Diagnostic residuals | 2 | 2 | 0 | 1.154931 | 7.514211 | 2 | Finite; never forced to close |
|
||||
| New H20-hours | 1 | 1 | 0 | 2.438817 | 2.438817 | 1 | Non-negative; below 6.0 |
|
||||
|
||||
Checked invariants: source/helper/protocol provenance; identical request IDs,
|
||||
content/token totals, and prompt-preserving manifests; C00-TP1 four-way
|
||||
placement; detached controller ownership; exact clean windows; zero failures;
|
||||
offered-rate tolerance; A-P5-1 compile/capture ordering and coverage; Layer-1
|
||||
footer/sidecar balance, continuous step indices, token composition, and zero
|
||||
drops; non-negative counters; drain gates; bridge decision; no public prompt or
|
||||
generated-text leakage; public-disk/free-space limits; GPU hard-cap compliance;
|
||||
and zero-process/zero-memory cleanup. Manipulation failures are A2, A3, and A4;
|
||||
the sole machine sanity red flag is P04 denominator instability.
|
||||
Reference in New Issue
Block a user