aituner/docs/opprof/phase6-results.md

OpProf Phase 6 dash0 results

Status: COMPLETE, DATA-VALID A-P6-2 SOLO TIER; ARGMAX/RANKING/TRAP INCONCLUSIVE UNDER THE FROZEN RULES.

Phase 6 completed all 12 C1 cells on patched vLLM 0.24.1.dev3+g668cfb7e2 (4b253fd). A-P6-2 makes the serialized solo tier authoritative because the v0.20 baseline was solo and the original W2/W3 confirmations exposed large pass-rate and queue changes after neighboring wave clients became idle. The authoritative campaign contains 37 primary anchors, 29 same-placement confirmations, and 12 valid Layer-1 streams.

The machine-readable result is runs/opprof-phase6/phase6/metrics.json, SHA-256 290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16. Co-located W1-W3 artifacts are preserved beside the solo tree rather than overwritten. Public artifacts contain no prompt or generated text.

Data sanity first

There are no data-sanity red flags. All 37 authoritative primary anchors reproduce the pinned v0.20 request count exactly; all client invariants pass; all 12 solo cells have one graceful footer and agreeing final sidecar; Layer-1 indices are contiguous with zero drops; every anchor interval is represented; and all cold-start compile/capture events precede measurement.

The scientific decision gate is separate from data validity. Eight cells have bounded monotonic solo frontiers. TP2/MNS16, TP4/MNS32, and TP4/MNS64 pass the highest recorded request set and are right-censored at the pinned-history edge. TP4/MNS16 is non-monotonic: L resolves feasible, P resolves infeasible, and U resolves feasible. These four cells prohibit a full-surface floor-bucket rank, tau-b, and trap decision; no value is imputed beyond the recorded history.

Launch echo

A-P6-2_LAUNCH_ECHO host=dash0 engine=vllm-0.24.1.dev3+g668cfb7e2@4b253fd authoritative=solo cells=12 mandatory_anchors=25 crawl=recorded-only confirmations=2of3 prior_h20h=2.291173 new_est_h20h=3.440 cumulative_est_h20h=5.731173 cap_h20h=6.0 est_wall=45-70min
SOLO_WAVE_ECHO order=TP4/MNS32,TP4/MNS64,TP2/MNS32,TP2/MNS64,TP4/MNS16,TP2/MNS8,TP2/MNS16,TP4/MNS8,TP1/MNS8,TP1/MNS16,TP1/MNS32,TP1/MNS64 placement=one_server_one_client
FINAL_ACCOUNTING status=complete cells=12 primary=37 confirmations=29 solo_h20h=3.353371 campaign_h20h=5.644544 cap_h20h=6.0

The exact per-cell echoes, GPU placement, trace and ground-truth paths, spend, and projection are retained in runs/opprof-phase6/phase6/solo-authoritative/launch-echo.log.

Run completion stats

Item	Result
Historical selections reconstructed	92/92 exact
Authoritative solo cells	12/12
Solo primary anchors	37
Solo confirmations	29
Solo anchor trials	66
Solo warm-ups	12; exactly 16 completions each
Long prompts per warm-up	2-8 raw inputs >4096
Bounded solo frontiers	8/12
History-edge right-censored cells	3
Non-monotonic cells	1
Prior co-located primary/confirm trials	21 + 3
Total accepted anchor trials across attempts	90
Co-located spend before A-P6-2	2.291173 H20-hours
A-P6-2 solo spend	3.353371 H20-hours
Total campaign spend	5.644544 H20-hours
Artifact footprint	467 MiB, below 3 GiB

Final paired surface

A* marks the authoritative A-P6-2 solo tier. Exact values are bounded frontiers. >= is a recorded-history lower bound. The TP4/MNS16 value is not a frontier because its feasibility sequence is non-monotonic.

Cell	Tier	v0.20 frontier	v0.24 authoritative result	Drift	Status
TP1/MNS8	A* solo	2.1000	2.3833	+13.49%	bounded up
TP1/MNS16	A* solo	2.3500	2.3833	+1.42%	bounded up
TP1/MNS32	A* solo	2.2833	2.3833	+4.38%	bounded up
TP1/MNS64	A* solo	2.2833	2.3833	+4.38%	bounded up
TP2/MNS8	A* solo	2.2750	2.2417	-1.47%	bounded down
TP2/MNS16	A* solo	2.2750	>=2.3000	>=+1.10%	right-censored history edge
TP2/MNS32	A* solo	3.2833	3.2583	-0.76%	bounded down
TP2/MNS64	A* solo	3.2583	2.3000	-29.41%	bounded down
TP4/MNS8	A* solo	1.2833	1.3208	+2.92%	bounded up
TP4/MNS16	A* solo	2.4417	N/E; observed max feasible 2.5000	N/E	non-monotonic: L pass, peak fail, U pass
TP4/MNS32	A* solo	2.4417	>=2.5000	>=+2.39%	right-censored history edge
TP4/MNS64	A* solo	2.4417	>=2.5000	>=+2.39%	right-censored history edge

The solo tier validates two material frontier changes under the predeclared 5% threshold: TP1/MNS8 improves by 13.49%, while TP2/MNS64 declines by 29.41%. The previous TP2/MNS64 decline is now quotable because both its passing 2.3000 anchor and failing 3.2583 anchor were repeated solo.

Frozen decision-rule verdicts

• ARGMAX: INCONCLUSIVE. TP2/MNS32 is the highest bounded observed frontier at 3.2583 req/s/GPU, but the frozen rule requires a bounded 12-cell subset. TP2/MNS16 and TP4/MNS32/64 terminate at passing history-edge anchors, and TP4/MNS16 has no coherent frontier. The top floor bucket is therefore not defined for all contenders.
• RANKING: INCONCLUSIVE. Only 8/12 frontiers are bounded. Per protocol, Kendall tau-b is not evaluable (tau_b=null) rather than computed from censored lower bounds. The tau-b>=0.8 survival rule cannot be applied.
• TRAP: INCONCLUSIVE. TP4/MNS16 is non-monotonic and its TP4/MNS32 neighbor is right-censored. Neither trap persistence nor escape satisfies the frozen floor-bucket rule.

Solo versus co-located exact-anchor deltas

The co-located column is the simultaneous-wave primary. The solo column is the accepted median after same-placement 2-of-3 adjudication. Delta is percentage points. P/F are SLO feasible/infeasible at 0.95.

Cell	Anchor	Co-located primary	Solo authoritative	Delta	State change
TP1/MNS8	0.21875000	0.992	0.992	+0.00pp	P->P
TP1/MNS8	0.22656250	0.206	0.992	+78.57pp	F->P
TP1/MNS16	0.24609375	1.000	1.000	+0.00pp	P->P
TP1/MNS16	0.25000000	1.000	1.000	+0.00pp	P->P
TP1/MNS32	0.24218750	1.000	1.000	+0.00pp	P->P
TP1/MNS32	0.24609375	1.000	1.000	+0.00pp	P->P
TP1/MNS64	0.24218750	1.000	1.000	+0.00pp	P->P
TP1/MNS64	0.24609375	1.000	1.000	+0.00pp	P->P
TP2/MNS8	0.49218750	0.691	1.000	+30.86pp	F->P
TP2/MNS8	0.49609375	0.092	0.581	+48.90pp	F->F
TP2/MNS16	0.49218750	1.000	1.000	+0.00pp	P->P
TP2/MNS16	0.49609375	0.088	1.000	+91.21pp	F->P
TP2/MNS32	0.75000000	0.412	1.000	+58.82pp	F->P
TP2/MNS32	0.75390625	0.028	0.575	+54.70pp	F->F
TP2/MNS64	0.50000000	0.460	1.000	+53.99pp	F->P
TP2/MNS64	0.75000000	0.141	0.561	+42.07pp	F->F
TP4/MNS8	0.016055910008	1.000	1.000	+0.00pp	P->P
TP4/MNS8	0.016591107009	0.071	1.000	+92.86pp	F->P
TP4/MNS16	0.033182214016	0.609	1.000	+39.13pp	F->P
TP4/MNS16	0.033717411016	0.036	0.238	+20.22pp	F->F
TP4/MNS16	0.034252608017	0.847	1.000	+15.33pp	F->P

The deltas range from 0 to +92.86pp across 21 exact pairs. Four TP1/MNS16-64 pairs are unchanged, while several TP2/TP4 and TP1/MNS8 anchors flip. Thus co-location validity is metric- and cell-dependent, not a universal offset. Throughput-only values may co-locate, but SLO-frontier feasibility cannot be mixed across placement tiers.

Layer-1 mechanism notes for material drift

Layer-1 is mechanism context, not a matched v0.20 causal decomposition. P3 P10 is TP1 at a different workload/output contract. All 37 solo primaries have zero preemptions.

Cell/anchor	State	Waiting mean/max	Decode-B mean	KV mean	NONE graph share	Padding
TP1/MNS8, 0.25	frontier pass	0.74 / 8	4.25	0.0407	3.26%	0.34%
TP1/MNS8, 0.50	next fail	11.66 / 27	7.19	0.0644	5.04%	0.02%
TP2/MNS64, 0.50	frontier pass	0.00 / 0	3.28	0.0084	0.00%	18.88%
TP2/MNS64, 0.75	old-peak fail, primary	0.00 / 0	18.05	0.0448	9.47%	0.52%
TP2/MNS64, 0.75	old-peak fail, confirm	0.0002 / 1	11.52	0.0285	4.57%	1.13%

For TP1/MNS8, the new frontier remains feasible despite a larger decode batch and modest queueing; failure at 0.5 coincides with a 16x waiting-mean increase, higher KV usage, and more eager/non-full graph execution. For TP2/MNS64, the passing lower anchor has small decode batches and full/piecewise graphs despite high padding, whereas both old-peak trials fail with much larger decode batches and graph fallback. Zero preemptions and near-zero queueing rule out preemption and server waiting as the main TP2/MNS64 explanation, but the unmatched v0.20 telemetry prevents assigning causality uniquely to graph mode or batch shape.

Operational findings and full attempt history

1. The initial ownership-monitor false start charged 0.132918 H20-hours without measurement. The repaired W1 charged 0.299393; A-P6-1 later accepted its eight anchors from balanced checkpoint sidecars without rerun.
2. Co-located W2/W3 completed 13 primary anchors and three confirmations but exposed placement-sensitive SLO results. The 3.0-hour projection stopped W4 at 2.291173 total; those values remain indicative and preserved.
3. A-P6-2 completed W4 and remeasured the full surface solo. Every solo server used --shutdown-timeout 120, emitted a graceful footer, and passed Layer-1 accounting.
4. Placement was not the only transient. TP4/MNS32 and TP4/MNS64 first full replays failed at 0.031/0.114 but subsequent identical solo trials passed; TP4/MNS16 produced an order-dependent non-monotonic sequence. The 16-request A-P5-1 gate removes compile/capture cold start but not necessarily the first full-trace transient. The predeclared 2-of-3 rule prevented single-trial conclusions, but a future protocol should use a full-trace burn-in or randomized/reversed anchor order.
5. Two cleanup checks briefly observed 1-4 MiB at 0% with zero compute PIDs. Memory decayed to 0 MiB within seconds, graceful footers validated, and no rerun was needed. Cleanup now polls for 30 seconds rather than treating transient driver bookkeeping as a failed experiment.

GPU total

The complete campaign charged 5.6445441643 H20-hours against the user-approved 6.0 cap:

• original co-located attempts: 2.2911728495;
• A-P6-2 authoritative solo tier: 3.3533713148;
• remaining headroom: 0.3554558357 H20-hours.

Final dash0 state is 0 MiB and 0% utilization on all eight H20s, with zero controller, client, vLLM, EngineCore, or worker processes.

Sanity block

Numeric family	n	Min	Max	Distinct	Checked invariant/result
Historical selection checks	92	66 requests	600 requests	34	92/92 exact
Solo primary pass rates	37	0.0307	1.0000	17	Ratios in [0,1]; not identical
Solo selected requests	37	121	600	18	Exact v0.20 count per anchor
Layer-1 steps per primary	37	343	12,103	37	Contiguous; non-negative; zero drops
Layer-1 records per cell	12	14,174	58,725	12	Footer/sidecar balanced
Solo preemptions	37	0	0	1 expected	Zero throughout
Reported v0.24 frontier values/bounds	11	1.3208	3.2583	6	TP4/MNS16 omitted as non-monotonic
Frontier drift values/bounds	12	-29.41%	+13.49%	9	11 finite; one non-monotonic missing
Bounded/censored cells	12	8 bounded	4 unbounded	2	Censoring never imputed
Exact solo/co-located deltas	21	+0.00pp	+92.86pp	13	Same request set and anchor
Solo confirmation pass rates	29	0.4130	1.0000	10	Frozen 2-of-3 applied
Solo cell H20-hours	12	0.0845	0.5534	12	Non-negative; serialized placement
Total campaign H20-hours	1	5.644544	5.644544	1	Below 6.0 hard cap
Final GPU memory/utilization	8	0 MiB / 0%	0 MiB / 0%	1 expected	Zero compute processes

Checked invariants: exact cap-before-threshold selection; raw-length SLO evaluation; exact 128-token completions or counted failures; nondecreasing arrivals; all selected outcomes represented in the denominator; one solo server/client at a time; compile/capture outside measured intervals; 12/12 graceful footer balances; non-negative counters; pass ratios in range; no prompt text in public artifacts; complete attempt-history retention; authoritative tier separation; explicit censoring/non-monotonicity; tau-b suppression when the full surface is unbounded; hard-cap compliance; and complete GPU cleanup.