gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
39b767e384fc49da53b99ad06a3e2ca1b6ac37d6
aituner/docs/opprof/phase6-results.md
Gahow Wang d5b276180d Add OpProf campaign: protocols, results, patches, run evidence (P0-P6) 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 11:06:10 +08:00

227 lines
12 KiB
Markdown
Raw Blame History

# OpProf Phase 6 dash0 results
Status: **COMPLETE, DATA-VALID A-P6-2 SOLO TIER; ARGMAX/RANKING/TRAP INCONCLUSIVE UNDER THE FROZEN RULES**.
Phase 6 completed all 12 C1 cells on patched vLLM
`0.24.1.dev3+g668cfb7e2` (`4b253fd`). A-P6-2 makes the serialized solo tier
authoritative because the v0.20 baseline was solo and the original W2/W3
confirmations exposed large pass-rate and queue changes after neighboring wave
clients became idle. The authoritative campaign contains 37 primary anchors,
29 same-placement confirmations, and 12 valid Layer-1 streams.
The machine-readable result is
`runs/opprof-phase6/phase6/metrics.json`, SHA-256
`290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16`.
Co-located W1-W3 artifacts are preserved beside the solo tree rather than
overwritten. Public artifacts contain no prompt or generated text.
## Data sanity first
There are **no data-sanity red flags**. All 37 authoritative primary anchors
reproduce the pinned v0.20 request count exactly; all client invariants pass;
all 12 solo cells have one graceful footer and agreeing final sidecar; Layer-1
indices are contiguous with zero drops; every anchor interval is represented;
and all cold-start compile/capture events precede measurement.
The scientific decision gate is separate from data validity. Eight cells have
bounded monotonic solo frontiers. TP2/MNS16, TP4/MNS32, and TP4/MNS64 pass the
highest recorded request set and are right-censored at the pinned-history edge.
TP4/MNS16 is non-monotonic: L resolves feasible, P resolves infeasible, and U
resolves feasible. These four cells prohibit a full-surface floor-bucket rank,
tau-b, and trap decision; no value is imputed beyond the recorded history.
## Launch echo
```text
A-P6-2_LAUNCH_ECHO host=dash0 engine=vllm-0.24.1.dev3+g668cfb7e2@4b253fd authoritative=solo cells=12 mandatory_anchors=25 crawl=recorded-only confirmations=2of3 prior_h20h=2.291173 new_est_h20h=3.440 cumulative_est_h20h=5.731173 cap_h20h=6.0 est_wall=45-70min
SOLO_WAVE_ECHO order=TP4/MNS32,TP4/MNS64,TP2/MNS32,TP2/MNS64,TP4/MNS16,TP2/MNS8,TP2/MNS16,TP4/MNS8,TP1/MNS8,TP1/MNS16,TP1/MNS32,TP1/MNS64 placement=one_server_one_client
FINAL_ACCOUNTING status=complete cells=12 primary=37 confirmations=29 solo_h20h=3.353371 campaign_h20h=5.644544 cap_h20h=6.0
```
The exact per-cell echoes, GPU placement, trace and ground-truth paths, spend,
and projection are retained in
`runs/opprof-phase6/phase6/solo-authoritative/launch-echo.log`.
## Run completion stats
| Item | Result |
|---|---:|
| Historical selections reconstructed | 92/92 exact |
| Authoritative solo cells | 12/12 |
| Solo primary anchors | 37 |
| Solo confirmations | 29 |
| Solo anchor trials | 66 |
| Solo warm-ups | 12; exactly 16 completions each |
| Long prompts per warm-up | 2-8 raw inputs `>4096` |
| Bounded solo frontiers | 8/12 |
| History-edge right-censored cells | 3 |
| Non-monotonic cells | 1 |
| Prior co-located primary/confirm trials | 21 + 3 |
| Total accepted anchor trials across attempts | 90 |
| Co-located spend before A-P6-2 | 2.291173 H20-hours |
| A-P6-2 solo spend | 3.353371 H20-hours |
| Total campaign spend | **5.644544 H20-hours** |
| Artifact footprint | 467 MiB, below 3 GiB |
## Final paired surface
`A*` marks the authoritative A-P6-2 solo tier. Exact values are bounded
frontiers. `>=` is a recorded-history lower bound. The TP4/MNS16 value is not a
frontier because its feasibility sequence is non-monotonic.
| Cell | Tier | v0.20 frontier | v0.24 authoritative result | Drift | Status |
|---|---|---:|---:|---:|---|
| TP1/MNS8 | A* solo | 2.1000 | **2.3833** | **+13.49%** | bounded up |
| TP1/MNS16 | A* solo | 2.3500 | 2.3833 | +1.42% | bounded up |
| TP1/MNS32 | A* solo | 2.2833 | 2.3833 | +4.38% | bounded up |
| TP1/MNS64 | A* solo | 2.2833 | 2.3833 | +4.38% | bounded up |
| TP2/MNS8 | A* solo | 2.2750 | 2.2417 | -1.47% | bounded down |
| TP2/MNS16 | A* solo | 2.2750 | >=2.3000 | >=+1.10% | right-censored history edge |
| TP2/MNS32 | A* solo | **3.2833** | **3.2583** | -0.76% | bounded down |
| TP2/MNS64 | A* solo | 3.2583 | **2.3000** | **-29.41%** | bounded down |
| TP4/MNS8 | A* solo | 1.2833 | 1.3208 | +2.92% | bounded up |
| TP4/MNS16 | A* solo | 2.4417 | N/E; observed max feasible 2.5000 | N/E | **non-monotonic: L pass, peak fail, U pass** |
| TP4/MNS32 | A* solo | 2.4417 | >=2.5000 | >=+2.39% | right-censored history edge |
| TP4/MNS64 | A* solo | 2.4417 | >=2.5000 | >=+2.39% | right-censored history edge |
The solo tier validates two material frontier changes under the predeclared 5%
threshold: TP1/MNS8 improves by 13.49%, while TP2/MNS64 declines by 29.41%.
The previous TP2/MNS64 decline is now quotable because both its passing 2.3000
anchor and failing 3.2583 anchor were repeated solo.
## Frozen decision-rule verdicts
- **ARGMAX: INCONCLUSIVE.** TP2/MNS32 is the highest bounded observed frontier
at 3.2583 req/s/GPU, but the frozen rule requires a bounded 12-cell subset.
TP2/MNS16 and TP4/MNS32/64 terminate at passing history-edge anchors, and
TP4/MNS16 has no coherent frontier. The top floor bucket is therefore not
defined for all contenders.
- **RANKING: INCONCLUSIVE.** Only 8/12 frontiers are bounded. Per protocol,
Kendall tau-b is **not evaluable** (`tau_b=null`) rather than computed from
censored lower bounds. The `tau-b>=0.8` survival rule cannot be applied.
- **TRAP: INCONCLUSIVE.** TP4/MNS16 is non-monotonic and its TP4/MNS32 neighbor
is right-censored. Neither trap persistence nor escape satisfies the frozen
floor-bucket rule.
## Solo versus co-located exact-anchor deltas
The co-located column is the simultaneous-wave primary. The solo column is the
accepted median after same-placement 2-of-3 adjudication. Delta is percentage
points. `P/F` are SLO feasible/infeasible at 0.95.
| Cell | Anchor | Co-located primary | Solo authoritative | Delta | State change |
|---|---:|---:|---:|---:|---|
| TP1/MNS8 | 0.21875000 | 0.992 | 0.992 | +0.00pp | P->P |
| TP1/MNS8 | 0.22656250 | 0.206 | 0.992 | **+78.57pp** | F->P |
| TP1/MNS16 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
| TP1/MNS16 | 0.25000000 | 1.000 | 1.000 | +0.00pp | P->P |
| TP1/MNS32 | 0.24218750 | 1.000 | 1.000 | +0.00pp | P->P |
| TP1/MNS32 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
| TP1/MNS64 | 0.24218750 | 1.000 | 1.000 | +0.00pp | P->P |
| TP1/MNS64 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
| TP2/MNS8 | 0.49218750 | 0.691 | 1.000 | +30.86pp | F->P |
| TP2/MNS8 | 0.49609375 | 0.092 | 0.581 | +48.90pp | F->F |
| TP2/MNS16 | 0.49218750 | 1.000 | 1.000 | +0.00pp | P->P |
| TP2/MNS16 | 0.49609375 | 0.088 | 1.000 | **+91.21pp** | F->P |
| TP2/MNS32 | 0.75000000 | 0.412 | 1.000 | +58.82pp | F->P |
| TP2/MNS32 | 0.75390625 | 0.028 | 0.575 | +54.70pp | F->F |
| TP2/MNS64 | 0.50000000 | 0.460 | 1.000 | +53.99pp | F->P |
| TP2/MNS64 | 0.75000000 | 0.141 | 0.561 | +42.07pp | F->F |
| TP4/MNS8 | 0.016055910008 | 1.000 | 1.000 | +0.00pp | P->P |
| TP4/MNS8 | 0.016591107009 | 0.071 | 1.000 | **+92.86pp** | F->P |
| TP4/MNS16 | 0.033182214016 | 0.609 | 1.000 | +39.13pp | F->P |
| TP4/MNS16 | 0.033717411016 | 0.036 | 0.238 | +20.22pp | F->F |
| TP4/MNS16 | 0.034252608017 | 0.847 | 1.000 | +15.33pp | F->P |
The deltas range from 0 to +92.86pp across 21 exact pairs. Four TP1/MNS16-64
pairs are unchanged, while several TP2/TP4 and TP1/MNS8 anchors flip. Thus
co-location validity is **metric- and cell-dependent**, not a universal offset.
Throughput-only values may co-locate, but SLO-frontier feasibility cannot be
mixed across placement tiers.
## Layer-1 mechanism notes for material drift
Layer-1 is mechanism context, not a matched v0.20 causal decomposition. P3 P10
is TP1 at a different workload/output contract. All 37 solo primaries have
zero preemptions.
| Cell/anchor | State | Waiting mean/max | Decode-B mean | KV mean | `NONE` graph share | Padding |
|---|---|---:|---:|---:|---:|---:|
| TP1/MNS8, 0.25 | frontier pass | 0.74 / 8 | 4.25 | 0.0407 | 3.26% | 0.34% |
| TP1/MNS8, 0.50 | next fail | 11.66 / 27 | 7.19 | 0.0644 | 5.04% | 0.02% |
| TP2/MNS64, 0.50 | frontier pass | 0.00 / 0 | 3.28 | 0.0084 | 0.00% | 18.88% |
| TP2/MNS64, 0.75 | old-peak fail, primary | 0.00 / 0 | 18.05 | 0.0448 | 9.47% | 0.52% |
| TP2/MNS64, 0.75 | old-peak fail, confirm | 0.0002 / 1 | 11.52 | 0.0285 | 4.57% | 1.13% |
For TP1/MNS8, the new frontier remains feasible despite a larger decode batch
and modest queueing; failure at 0.5 coincides with a 16x waiting-mean increase,
higher KV usage, and more eager/non-full graph execution. For TP2/MNS64, the
passing lower anchor has small decode batches and full/piecewise graphs despite
high padding, whereas both old-peak trials fail with much larger decode batches
and graph fallback. Zero preemptions and near-zero queueing rule out preemption
and server waiting as the main TP2/MNS64 explanation, but the unmatched v0.20
telemetry prevents assigning causality uniquely to graph mode or batch shape.
## Operational findings and full attempt history
1. The initial ownership-monitor false start charged 0.132918 H20-hours without
measurement. The repaired W1 charged 0.299393; A-P6-1 later accepted its
eight anchors from balanced checkpoint sidecars without rerun.
2. Co-located W2/W3 completed 13 primary anchors and three confirmations but
exposed placement-sensitive SLO results. The 3.0-hour projection stopped W4
at 2.291173 total; those values remain indicative and preserved.
3. A-P6-2 completed W4 and remeasured the full surface solo. Every solo server
used `--shutdown-timeout 120`, emitted a graceful footer, and passed Layer-1
accounting.
4. Placement was not the only transient. TP4/MNS32 and TP4/MNS64 first full
replays failed at 0.031/0.114 but subsequent identical solo trials passed;
TP4/MNS16 produced an order-dependent non-monotonic sequence. The 16-request
A-P5-1 gate removes compile/capture cold start but not necessarily the first
full-trace transient. The predeclared 2-of-3 rule prevented single-trial
conclusions, but a future protocol should use a full-trace burn-in or
randomized/reversed anchor order.
5. Two cleanup checks briefly observed 1-4 MiB at 0% with zero compute PIDs.
Memory decayed to 0 MiB within seconds, graceful footers validated, and no
rerun was needed. Cleanup now polls for 30 seconds rather than treating
transient driver bookkeeping as a failed experiment.
## GPU total
The complete campaign charged **5.6445441643 H20-hours** against the
user-approved **6.0** cap:
- original co-located attempts: 2.2911728495;
- A-P6-2 authoritative solo tier: 3.3533713148;
- remaining headroom: 0.3554558357 H20-hours.
Final dash0 state is 0 MiB and 0% utilization on all eight H20s, with zero
controller, client, vLLM, EngineCore, or worker processes.
## Sanity block
| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
|---|---:|---:|---:|---:|---|
| Historical selection checks | 92 | 66 requests | 600 requests | 34 | 92/92 exact |
| Solo primary pass rates | 37 | 0.0307 | 1.0000 | 17 | Ratios in `[0,1]`; not identical |
| Solo selected requests | 37 | 121 | 600 | 18 | Exact v0.20 count per anchor |
| Layer-1 steps per primary | 37 | 343 | 12,103 | 37 | Contiguous; non-negative; zero drops |
| Layer-1 records per cell | 12 | 14,174 | 58,725 | 12 | Footer/sidecar balanced |
| Solo preemptions | 37 | 0 | 0 | 1 expected | Zero throughout |
| Reported v0.24 frontier values/bounds | 11 | 1.3208 | 3.2583 | 6 | TP4/MNS16 omitted as non-monotonic |
| Frontier drift values/bounds | 12 | -29.41% | +13.49% | 9 | 11 finite; one non-monotonic missing |
| Bounded/censored cells | 12 | 8 bounded | 4 unbounded | 2 | Censoring never imputed |
| Exact solo/co-located deltas | 21 | +0.00pp | +92.86pp | 13 | Same request set and anchor |
| Solo confirmation pass rates | 29 | 0.4130 | 1.0000 | 10 | Frozen 2-of-3 applied |
| Solo cell H20-hours | 12 | 0.0845 | 0.5534 | 12 | Non-negative; serialized placement |
| Total campaign H20-hours | 1 | 5.644544 | 5.644544 | 1 | Below 6.0 hard cap |
| Final GPU memory/utilization | 8 | 0 MiB / 0% | 0 MiB / 0% | 1 expected | Zero compute processes |
Checked invariants: exact cap-before-threshold selection; raw-length SLO
evaluation; exact 128-token completions or counted failures; nondecreasing
arrivals; all selected outcomes represented in the denominator; one solo
server/client at a time; compile/capture outside measured intervals; 12/12
graceful footer balances; non-negative counters; pass ratios in range; no prompt
text in public artifacts; complete attempt-history retention; authoritative
tier separation; explicit censoring/non-monotonicity; tau-b suppression when
the full surface is unbounded; hard-cap compliance; and complete GPU cleanup.
Reference in New Issue View Git Blame Copy Permalink