aituner/docs/opprof/phase6-protocol.md

OpProf Phase 6 pre-registered cross-version churn protocol

Status: ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED.

Date frozen: 2026-07-12 (Asia/Singapore). Phase 6 asks whether the vLLM 0.20.0 best configuration and ranking on the C1 TP={1,2,4} x max-num-seqs={8,16,32,64} surface survive an upgrade to patched vLLM 0.24.0. This is a paired historical-anchor experiment, not a fresh tuner search. It measures churn at the exact recorded C1 request sets and reports where the recorded local frontier moved.

Approved dispositions (orchestrator; 2026-07-12)

All five open decisions are approved exactly as proposed, without changing an anchor, threshold, estimator, validity gate, or budget:

1. The upgrade-path estimand is approved. Resolved default changes are part of observed churn; dash1->dash0 and co-location remain explicit limitations and the result is not described as a pure engine-version causal effect.
2. The adaptive 25-anchor subset is approved: every cell receives its old peak plus the direction-relevant adjacent anchor, and TP4/MNS16 receives both neighbors.
3. The 5% material-frontier threshold, tau-b>=0.8 ranking-survival threshold, and zero-anchored floor-bucket argmax/trap rules are approved.
4. One primary observation per historical anchor plus the 0.35-H20-hour confirmation reserve is approved.
5. The A-P5-1-class warm-up long tier is approved as raw input >4096 tokens for this 0-8192 workload.

This approval authorizes the later execution only after pinned-input, detached ownership, cold-start, echo-before-launch, placement, privacy, cleanup, and projected-budget gates pass. The 3.0-H20-hour cap remains absolute.

Amendment A-P6-1 — checkpoint-sidecar accounting and conditional cap

The first Phase-6 W1 request replays were incorrectly rejected by requiring a clean in-stream footer and final=true sidecar after a non-graceful shutdown. That contradicts the already accepted A-P3-5/P5 accounting rule. Before any resume, all four W1 cell streams and their eight measured anchor intervals must be CPU-re-adjudicated against the latest atomic checkpoint sidecar. With no in-stream footer, the sidecar is authoritative when every complete JSONL line decodes, step indices are contiguous, on-disk data-record count equals written_records, final data-line step equals last_step_index, encoded_records = written_records + dropped_records, dropped_records=0, and the checkpoint is within the configured one-second flush interval (plus the accepted 100-ms scheduling tolerance) of the recorded shutdown boundary. At most one post-checkpoint flush interval may be absent. A balanced stream accepts all covered W1 replay intervals without a GPU rerun.

All subsequent waves use the graceful P5 path: start vllm serve with positive --shutdown-timeout, signal each API parent with SIGINT, wait up to 150 seconds for EngineCore drain/finalization, and use process-group SIGTERM/SIGKILL only as fallback. The preferred result is one in-stream footer plus an agreeing final=true sidecar; the atomic checkpoint rule remains the crash fallback.

Only if any W1 sidecar fails the CPU balance/coverage gate does A-P6-1c activate: the hard cap becomes 3.5 H20-hours, the activation is logged before launch, and W1 is rerun under graceful shutdown. If all W1 sidecars balance, A-P6-1c is not activated, W1 is accepted in place, its already charged 0.299393 H20-hours is retained, and execution resumes at W2 under the original 3.0-hour cap.

Amendment A-P6-2 — authoritative solo frontier tier and 6.0-hour cap

The user raises the cumulative Phase-6 hard cap from 3.0 to 6.0 H20-hours. This amendment responds to the W2/W3 confirmation evidence: when neighboring wave clients became idle, pass rate changed from 0.412 -> 1.000 and 0.028 -> 0.957 at TP2/MNS32 and from 0.036 -> 1.000 at TP4/MNS16, while Layer-1 waiting means collapsed from 1.22 -> 0.00, 8.50 -> 0.35, and 28.33 -> 0.06. Because the v0.20 C1 baseline ran one cell at a time on dash1, co-located SLO feasibility is not a valid authoritative tier for the paired frontier. Co-location remains an explicitly reported, metric-dependent methodological ablation; its throughput-only observations are indicative.

Every SLO-frontier-decisive Phase-6 anchor is therefore run solo: exactly one vLLM server and one replay client are active on dash0, no other Phase-6 server/client is resident, and only the cell's TP GPUs are allocated. Each solo cell gets a fresh server process, the frozen exact-selection client, an independent 16-request long-tier warm-up, A-P5-1 cold-start gates, Layer-1 telemetry, and graceful A-P6-1 shutdown. The solo result is authoritative for feasibility, frontier, floor bucket, argmax, tau-b, and trap decisions. A co-located-only result may not fill a missing solo frontier or support a paper claim.

The mandatory solo set has no cell exclusions:

Cell set	Mandatory solo anchors	Inclusion reason
TP1/MNS8	existing L+P	co-located peak failed
TP1/MNS16,32,64	existing P+U	co-located U passed and censored the frontier
TP2/MNS8,16	existing L+P	co-located peak failed; L distinguishes bracket vs left censoring
TP2/MNS32	existing L+P	old argmax; both anchors split primary/confirmation
TP2/MNS64	existing L+P	the indicative >29.4% down bound is unquotable until solo-confirmed
TP4/MNS8	existing L+P	co-located peak failed; trap-neighbor comparison
TP4/MNS16	existing L+P+U	named trap; peak split and L failed co-located
TP4/MNS32,64	P plus direction-relevant L/U	W4 completion; unmeasured under the 3.0-hour cap

After these 25 anchors, the controller may crawl only the pinned 92-anchor history: upward while the highest solo anchor passes or downward while the lowest solo anchor fails, stopping at the first opposite-feasibility anchor. There is no interpolation and no unrecorded request set. A pass rate in [0.93,0.97], or a solo/co-located disagreement that changes argmax/trap or a material-frontier claim, triggers a same-solo-placement confirmation; a split gets a third trial for the frozen 2-of-3 rule if projected budget permits. Unresolved or history-edge frontiers remain explicitly censored.

Priority is W4, TP2/MNS32, TP2/MNS64, TP4/MNS16, then the remaining failed or censored cells, so a budget stop preserves the most decision-relevant evidence. All twelve cells are nevertheless included. The planning envelope is 3.44 new H20-hours for serialized startup, warm-up, 25 mandatory anchors, bounded crawl, graceful drain, and a 0.20-hour safety reserve. Added to the already charged 2.291173 hours, the launch projection is 5.731173/6.0 H20-hours. Before each one-cell wave, the controller echoes cell, anchors, GPU placement, input paths, charged spend, remaining projection, and cap. Projected or actual spend reaching 6.0 stops further GPU work.

Final analysis preserves the complete W1-W3 co-located attempt history and reports a solo-versus-co-located delta table for every exact anchor pair. ARGMAX, RANKING, and TRAP use solo values only. RANKING remains inconclusive and tau-b non-evaluable unless all 12 solo frontiers are bounded under the pinned history.

Headline claim and falsifiable outcomes

The paper-facing question is:

Does the C1 vLLM 0.20.0 per-GPU SLO-feasible ranking, especially global best TP2/MNS32 and local trap TP4/MNS16, remain valid on vLLM 0.24.0 when the exact historical request-selection and SLO semantics are replayed?

Phase 6 can produce four distinct outcomes:

1. SURVIVES: TP2/MNS32 remains in the top floor bucket, the full-surface Kendall tau-b is at least 0.8, and the TP4/MNS16 trap persists.
2. ARGMAX MOVED: TP2/MNS32 is not in the top v0.24 measured-frontier bucket.
3. RANKING MOVED WITHOUT ARGMAX MOVEMENT: the old best survives but tau-b is below 0.8 or the trap changes status.
4. INCONCLUSIVE/PARTIAL: a decision-critical frontier is unbracketed, an anchor is not reproducible, repeated boundary verdicts disagree, the 12-cell surface is incomplete, or the 3.0-H20-hour cap stops required work.

The experiment quantifies observed upgrade churn. Because engine version, host, resolved defaults, and co-location differ together, it does not identify a pure causal effect of one vLLM commit.

Frozen vLLM 0.20.0 surface

The historical objective is the maximum measured SLO-feasible offered rate divided by TP:

f20(c) = max feasible selected_request_count / 60 seconds / TP

TP	MNS8	MNS16	MNS32	MNS64
1	2.1000	2.3500	2.2833	2.2833
2	2.2750	2.2750	3.2833	3.2583
4	1.2833	2.4417	2.4417	2.4417

Values are request/s/GPU. The global best is TP2/MNS32. TP4/MNS16 is the registered local trap: it is the first point on a TP4 plateau, has no improving adjacent MNS move, but is below the global best.

The pinned ground-truth asset is /home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json, SHA-256 23ff3e6f6b88df8632bf37d37c0ff87bfef9d1676db81592948c08e898f7a670. It contains 92 probe observations: eight anchors for each TP1/TP2 cell and seven for each TP4 cell.

Comparability contract

Exact workload and selection semantics

The private materialized workload is trace_windows/traces/chat_w20260311_1000.jsonl, 32,606 rows, SHA-256 f539f38eb0ee0f750e3c23ff47df6eed3faf723a25f1444d55665a85871750b9. Its window record is trace_windows/windows.json, SHA-256 23c432e7439508b07991cabfe1db9977ebb55f8ddc39ca214a627fc5f5ae4725: window [0,600] seconds becomes [0,60] under replay scale 0.1.

The later helper must call or byte-for-byte reproduce the following pinned AITuner implementations, rather than use the Phase-3/5 client:

Source	SHA-256	Frozen role
src/aituner/trace.py	6bb19f0e3eb42b1aecc6969a5f20d65eb505a91564dd117dd149155a6114eb35	load, stable time sort, cap/downsample, threshold selection
src/aituner/worker.py	18337fe273a48d1cbb2bc364f774d100f8ffa2be5f9cd6a7c58c8ed4717c5e48	rate-following submit loop, early stop, denominator accounting
src/aituner/slo.py	1d2e5b1ed9d0d6d6c07f8b5b6fa56d36ed9bb62710be54c125e9b2890f4f96d5	stepped SLO evaluation
src/aituner/http_client.py	213b80d5b37cc74333517f46ab4e8cdba1a274bdbfabdfc14207b314f61799ac	urllib/SSE TTFT, TPOT, usage accounting

The exact order is normative:

1. Read all JSONL rows in file order. Retain raw input_length in [0,8192].
2. Build the same chat body and stable-sort by scaled arrival time only. Stable ties retain source order; sorting by (timestamp,sampling_u,...) is forbidden.
3. For TP1/TP2, even-downsample the full filtered, time-sorted list to 512 entries before threshold selection, using index floor(i*N/512) for i=0..511. TP4 is uncapped.
4. Select exactly the requests with sampling_u <= anchor. Preserve their selected service order. Do not resample, wrap, shuffle, or interpolate.
5. Submit at each row's materialized timestamp*0.1; do not uniformize, rebase individual gaps, or replace the recorded burst process.
6. Send the original messages through /v1/chat/completions with stream=true, stream_options.include_usage=true, and the row's temperature when present. Set both min_tokens=128 and max_tokens=128.
7. Use maximum concurrency 64. Client request timeout remains 900 seconds.
8. Verify usage.completion_tokens==128; missing usage, a different token count, HTTP failure, timeout, and every request not submitted because the SLO verdict became unrecoverable are failures in the original selected-request denominator.
9. Preserve original early-stop logic: after more than N-ceil(0.95*N) failed evaluations, synthesize failures for the remaining selected requests. Disable adaptive Stop-A; it was absent in C1.

An offline, prompt-free preflight must regenerate all 92 historical request counts exactly. The local reconstruction already yields 17,710 filtered rows, 512 capped rows, and 92/92 count matches. Any future mismatch in count, request-ID hash, arrival hash, raw-length hash, or requested-token sum stops the GPU launch.

Exact SLO and score

Request feasibility uses the original raw trace input length, not tokenizer usage after chat templating:

raw input <= 4096:  TTFT <= 2000 ms
raw input <= 32768: TTFT <= 4000 ms
otherwise:          TTFT <= 6000 ms
all requests:       TPOT <= 50 ms
anchor feasible:    SLO-passing selected requests / all selected requests >= 0.95

Although the C1 filter limits inputs to 8192, the complete three-step rule is retained. TTFT is client wall time from request start to first nonempty content. TPOT is (last_content_time-first_content_time)/(usage_completion_tokens-1). The paired surface score is always selected count divided by 60 seconds and TP; completed throughput is a diagnostic and never replaces the historical score.

Serving configuration

The v0.24 command repeats the explicit C1 launch surface:

MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
VENV=/tmp/wjh-opprof-phase2-dash0-20260711/.venv
RUN_ROOT=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6

taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU_IDS" \
  VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
  "$VENV/bin/vllm" serve "$MODEL" \
  --host 127.0.0.1 --port "$PORT" \
  --served-model-name qwen3-30b-a3b-community \
  --max-num-batched-tokens 8192 \
  --max-num-seqs "$MNS" \
  --tensor-parallel-size "$TP" \
  --shutdown-timeout 120

--shutdown-timeout is operational and outside the measured interval. As in C1, do not explicitly set block size, GPU-memory utilization, prefix caching, chunked prefill, CUDA-graph capture sizes, DP, or EP. Their resolved v0.24 values must be parsed from startup logs and reported. Allowing version defaults to change is intentional: this experiment asks whether an operator's old explicit configuration remains good after upgrading, not whether internal defaults can be manually made identical.

Preflight records vllm.__version__, source commit, patch checksums, Python, PyTorch, CUDA, NCCL, Triton, driver, model/tokenizer file hashes, MoE backend, resolved scheduler/cache/graph config, compile-cache key, and exact command. The expected source is the existing Phase-3/5 patched vLLM 0.24 environment; any missing OpProf patch or unexpected model path is a stop condition.

Client equivalence, not a new client covariate

The later scripts/opprof_phase6_client.py is permitted only as a thin anchor driver around the four pinned AITuner modules. Its interface is normative:

P6C='python scripts/opprof_phase6_client.py'

$P6C preflight \
  --trace trace_windows/traces/chat_w20260311_1000.jsonl \
  --windows trace_windows/windows.json --window-id chat_w20260311_1000 \
  --ground-truth /home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json

$P6C run-anchor --cell "$CELL" --anchor "$U" \
  --base-url "http://127.0.0.1:$PORT" \
  --model qwen3-30b-a3b-community --completion-tokens 128 \
  --replay-time-scale 0.1 --max-concurrency 64 \
  --max-requests "$CAP_OR_NONE" --target-pass-rate 0.95 \
  --ttft-step-ms '4096:2000,32768:4000,inf:6000' --tpot-ms 50 \
  --request-timeout-seconds 900 --probe-deadline-ceiling-seconds 900 \
  --result-dir "$ANCHOR_DIR"

Before GPU execution, a no-server golden test must prove identical selected IDs, order, arrivals, raw input lengths, serialized bodies excluding model/URL, and SLO decisions for synthetic outcomes. The wrapper may add interval markers and sanity JSON, but may not change HTTP transport, scheduling, completion checks, early stop, or latency formulas.

Covariate ledger and estimand limit

Covariate	C1 v0.20	Phase 6 v0.24	Treatment/handling
Engine	community vLLM 0.20.0	patched vLLM 0.24.0	Primary intended churn dimension; exact builds recorded
Host	dash1	dash0	Both 8x H20 96 GB class; CPU/NUMA/clocks/driver/kernel recorded, not assumed identical
GPU visibility	TP1 C1 env exposed GPUs 0,1, TP1 consumed one	expose exactly TP GPUs	Declared operational covariate; H20-hour charge follows TP
Trial placement	recovered C1 harness trials	up to four co-located servers	CPU/NUMA pinned; reject outside GPU processes and report co-location limitation
HTTP client	pinned AITuner urllib/SSE	same code paths through thin wrapper	Eliminated as a semantic covariate by golden tests
Workload	materialized C1 chat window	byte-identical file/hash	Fixed
Selection/cap	cap-before-threshold; TP4 uncapped	exact same	Fixed and 92-count preflighted
SLO length	raw input length	raw input length	Fixed; server usage length is diagnostic only
Explicit flags	TP, MNS, MBT8192	same	Fixed
Unspecified defaults	v0.20 defaults	v0.24 defaults	Intentionally allowed to churn; resolved values reported
Telemetry	no Layer 1	zero-overhead OpProf Layer 1	Added measurement; overhead evidence -0.04196%, CI [-0.17443%, 0.04550%]

Therefore “version churn” means the observed paired upgrade path under these documented platform covariates. It must not be rewritten as a same-host causal microbenchmark.

Budget-constrained anchor subset

Adaptive adjacent-anchor design

The primary subset improves on measuring next-infeasible anchors only for a chosen top four:

1. Measure the recorded v0.20 peak anchor for all 12 cells.
2. For every non-trap cell, select exactly one adjacent recorded anchor by a result-independent rule fixed now:
   • if the v0.20 peak remains feasible on v0.24, measure the nearest higher recorded anchor, which was infeasible on v0.20;
   • if the peak becomes infeasible, measure the nearest lower recorded anchor, which was feasible on v0.20.
3. For TP4/MNS16, measure lower, peak, and higher anchors unconditionally. This directly brackets the registered trap.

Thus exactly 25 primary anchors are measured: TP1 eight, TP2 eight, and TP4 nine. The design covers every cell and spends the second anchor in the direction where its frontier could have moved. It also includes the suggested top-four test. Under the floor-bucket rule, the nominal top-four cutoff expands to five cells because TP4/MNS16/32/64 share one bucket; all five receive an upward test whenever their old peak still passes.

If an adjacent anchor does not bracket the transition—higher also passes or lower also fails—the cell is censored. Remaining budget may crawl one recorded anchor farther outward, in this priority order: old global best TP2/MNS32, TP2/MNS64, the TP4 plateau MNS16/32/64, then other cells by old score. Crawling uses only that cell's recorded history and stops at the first transition. No new sampling_u value may be invented. Failure to close a decision-critical bracket before the budget stop makes the applicable argmax/ranking verdict inconclusive.

Boundary confirmations

The original surface has one observation per anchor. Phase 6 keeps that paired unit but reserves at most 0.35 H20-hours for decision-triggered confirmation. Repeat an anchor if its v0.24 pass rate lies in [0.93,0.97], or if its single verdict alone changes argmax or trap status. Priority is argmax, trap, material frontier calls, then other cells. Agreement across two runs freezes the verdict; disagreement requests a third run if budget permits and uses 2-of-3. Without a third run, that anchor and every dependent headline verdict are inconclusive. Request outcomes are arrival-correlated, so a binomial CI is descriptive only and cannot replace these repeat rules.

Execution plan

Preflight and detached ownership

The Phase-6 controller follows A-P3-2 detached ownership: one controller PID, per-stage atomic state, process-group/environment markers, resumable completed anchors, exact command logs, and cleanup restricted to its own markers. Before launch it must verify dash0 has eight H20s, no outside compute processes on assigned GPUs, at least 100 GB CPFS free, pinned source/workload/helper hashes, all 92 selection counts, legal TP placement, and a projected total below 3.0 H20-hours.

The autonomous launch log must echo one resolved line before starting:

LAUNCH_ECHO host=dash0 engine=vllm-0.24.0-patched model=Qwen3-30B-A3B trace_sha=f539f38e... cells=12 primary_anchors=25 waves=4 gpus=0-7 warmup=16 clean_horizon=60s drain=derived<=120s est_wall=25-35min est_gpu=2.65_H20h confirm_reserve=0.35_H20h hard_cap=3.0_H20h

Rate-following cold-start and drain gates

The throughput-drift stationarity gate is forbidden for these rate-following anchors. A-P5-1-class cold-start gates are applied instead:

1. Every torch.compile/CUDA-graph first-occurrence event must precede the first measured anchor, verified from server logs and Layer 1.
2. A deterministic unmeasured warm set must complete at least 16 exact-output requests, including at least one request with raw input >4096 tokens. The peak-selected sets contain 43-184 such requests, so this gate is feasible.
3. No first-occurrence compile/capture event may occur in any measured anchor; all graph-hit (runtime_mode,bucket_tokens) descriptors must be covered by startup capture logs. A violation invalidates that anchor only.
4. Before the next anchor, all selected requests must be accounted for and three consecutive controller samples must show running/waiting/deferred queues at zero. Prefix cache is not explicitly enabled; nevertheless a full drain is required.

The probe deadline reproduces _probe_drain_deadline: last selected arrival + raw-length TTFT budget + 128*50 ms + 30 seconds, capped by the historical 900-second ceiling. With raw inputs <=8192 this is about 100.4 seconds from probe start; the controller has a 120-second class watchdog for cleanup. A deadline miss is an SLO-infeasible anchor, not permission to omit denominator requests. The server then shuts down through the official 120-second path.

Four-wave packing

Wave	Cells	Placement	Per-server measured anchors	Planned wall
W1	TP1/MNS8,16,32,64	four servers on GPUs 0,1,2,3; GPUs 4-7 idle	2 each	4-6 min
W2	TP2/MNS8,16,32,64	four servers on GPU pairs 0-1,2-3,4-5,6-7	2 each	4-6 min
W3	TP4/MNS8 and TP4/MNS16 trap	GPU quads 0-3 and 4-7	2 and 3	5-7 min
W4	TP4/MNS32 and TP4/MNS64	GPU quads 0-3 and 4-7	2 each	4-6 min

Each server starts once, passes warm-up once, runs its peak first, drains, then runs the predeclared adjacent anchor. Trap lower/upper order after peak is counterbalanced by execution seed 20260717. Anchor confirmation occurs before that server is released when possible. The controller records GPU clocks, power, utilization, memory, CPU load, NUMA placement, and outside processes through every wave. Any contamination invalidates the whole co-located wave.

Actual H20-hours are charged from server launch through verified cleanup for every allocated GPU, including idle time while a paired TP4 cell finishes. Before each wave or confirmation, the controller recomputes spent + current + conservative_remaining; it stops before launch if the projection can reach 3.0. Optional crawling and confirmations are sacrificed before any hard-cap violation.

Measurement and mechanism notes

For every anchor, report selected/admitted/completed/exact-output/SLO-pass counts; early-stop reason; offered and completed req/s; TTFT/TPOT mean and p50/p90/p95/p99; failures by raw-length TTFT bucket; elapsed/drain time; and peak GPU memory/utilization. The surface objective remains offered req/s/GPU at the highest measured feasible recorded anchor.

Layer 1 is active over the complete anchor service interval, from the first submission through the last selected-request accounting event. Aggregate:

• prefill, decode, and mixed-step counts and token shares;
• decode-batch distribution and 5-second CV;
• FULL/PIECEWISE/NONE graph-mode shares, bucket padding, and uncovered support;
• running/waiting/deferred queues and queue CV;
• preemption count/rate;
• KV blocks used/total, mean/max usage, and allocation pressure; and
• prefix queries/hits as a resolved-default diagnostic.

Mechanism notes compare v0.24 cells/anchors and may relate a frontier change to preemptions, KV pressure, batching, or graph modes. C1 lacks Layer-1 telemetry, so these notes cannot claim that a measured composition field itself changed from v0.20 or caused the version drift. Historical v0.20 TTFT/TPOT failure reasons may be paired directly; device-mechanism attribution remains descriptive.

Statistical and ranking analysis

Floor buckets

For each unrounded score vector s, use the SimFid inventory rule:

tol(s) = max(1e-9, 1e-6 * max_c(abs(s(c))))
bucket_s(c) = floor(s(c) / tol(s))

Two scores tie only when their integer buckets match. Bucketing occurs before display rounding, pair signs, tau-b, top selection, or trap evaluation. Report tol, bucket integers, and occupied intervals [j*tol,(j+1)*tol). For v0.20, tol20=3.283333333333333e-6; the TP4 MNS16/32/64 tie makes nominal top-4 an effective top-5 set.

Per-cell frontier

For each cell, define the v0.24 measured recorded-anchor frontier:

f24(c) = max_{measured u with accepted feasible verdict} N_c(u) / 60 / TP
drift(c) = (f24(c) - f20(c)) / f20(c)

The predeclared material threshold is X=5% relative, large enough to avoid calling one-request anchor quantization a paper-level churn effect. Report two orthogonal labels:

• BOUNDARY DOWN/UP: old peak flips feasible->infeasible, or the old adjacent infeasible anchor flips to feasible, respectively;
• MATERIAL FRONTIER MOVED: abs(drift)>5%; otherwise report the boundary flip as sub-material.

If peak passes and higher fails, and abs(drift)<=5%, the local recorded frontier is STABLE. If peak and lower both fail or peak and every affordable higher anchor pass, report a directional bound and UNBRACKETED, never an invented peak.

Argmax, ranking, and trap

• ARGMAX MOVED when TP2/MNS32 is not in the top bucket_f24 among all 12 decision-bounded cells. ARGMAX SURVIVED requires it to share the top bucket and no censored competitor capable of overtaking it. Otherwise it is inconclusive.
• Compute Kendall tau-b over all 12 floor-bucketed f20/f24 scores and report concordant, discordant, v0.20-tied, v0.24-tied, and comparable-pair counts. RANKING SURVIVES requires an evaluable 12-cell surface, argmax survival, tau-b>=0.8, and no >5% top-bucket pair reversal. tau-b<0.8 is RANKING MOVED. A partial surface receives no suite-level tau claim.
• TRAP PERSISTS when TP4/MNS16 is in a bucket at least as high as both adjacent TP4 cells MNS8 and MNS32, yet below the global-best bucket. It ESCAPES if an adjacent TP4 move is strictly better, and CEASES TO BE A TRAP if it joins the global-best bucket. Missing/unbracketed inputs make trap status inconclusive.

No arithmetic average of normalized scores is reported. Absolute rates, relative drift, the paired table, and every excluded/censored anchor accompany all ranking summaries.

Later deliverables and artifact contract

Execution, after approval, must produce:

• docs/opprof/phase6-results.md;
• runs/opprof-phase6/phase6/metrics.json containing the complete paired surface, anchor/run values, selection hashes, feasibility, floor buckets, frontier/argmax/ranking/trap verdicts, confirmation history, Layer-1 notes, covariates, and GPU accounting;
• exact commands, controller state, environment/machine fingerprints, server logs, client outcomes, Layer-1 JSONL/footer/sidecar, monitor samples, and per-anchor sanity JSON; and
• an operational-findings section covering startup/capture gates, drains, early stops, co-location, default changes, retries, contamination, and budget.

The results begin with data sanity and stop inferential claims on any red flag. Every numeric family ends with n, finite/missing, min, max, distinct count, and applicable invariants. Prompt text, messages, generated content, source substrings, and private request bodies stay in mode-0700 storage. Public files contain only hashes, IDs, lengths, timestamps, outcomes, counters, and summaries.

Anchor-subset table

Notation is u / selected N / recorded req/s/GPU. P is always measured. For non-trap cells, measure U if P passes on v0.24, otherwise L. TP4/MNS16 measures all L+P+U. Planning cost per anchor includes the 60-second replay and 20 seconds expected drain/accounting; shared startup/warm-up is budgeted below.

Cell	L: nearest recorded feasible below P	P: recorded peak	U: nearest recorded infeasible above P	Primary selection	H20-hour cost each
TP1/MNS8	0.21875 / 121 / 2.0167	0.2265625 / 126 / 2.1000	0.23046875 / 130 / 2.1667	P + (U if pass else L)	0.0222
TP1/MNS16	0.2421875 / 137 / 2.2833	0.24609375 / 141 / 2.3500	0.25 / 143 / 2.3833	P + (U if pass else L)	0.0222
TP1/MNS32	0.234375 / 132 / 2.2000	0.2421875 / 137 / 2.2833	0.24609375 / 141 / 2.3500	P + (U if pass else L)	0.0222
TP1/MNS64	0.234375 / 132 / 2.2000	0.2421875 / 137 / 2.2833	0.24609375 / 141 / 2.3500	P + (U if pass else L)	0.0222
TP2/MNS8	0.4921875 / 269 / 2.2417	0.49609375 / 273 / 2.2750	0.5 / 276 / 2.3000	P + (U if pass else L)	0.0444
TP2/MNS16	0.4921875 / 269 / 2.2417	0.49609375 / 273 / 2.2750	0.5 / 276 / 2.3000	P + (U if pass else L)	0.0444
TP2/MNS32	0.75 / 391 / 3.2583	0.75390625 / 394 / 3.2833	0.7578125 / 396 / 3.3000	P + (U if pass else L)	0.0444
TP2/MNS64	0.5 / 276 / 2.3000	0.75 / 391 / 3.2583	0.75390625 / 394 / 3.2833	P + (U if pass else L)	0.0444
TP4/MNS8	0.016055910008 / 301 / 1.2542	0.016591107009 / 308 / 1.2833	0.017126304009 / 317 / 1.3208	P + (U if pass else L)	0.0889
TP4/MNS16 trap	0.033182214016 / 575 / 2.3958	0.033717411016 / 586 / 2.4417	0.034252608017 / 600 / 2.5000	L + P + U	0.0889
TP4/MNS32	0.033182214016 / 575 / 2.3958	0.033717411016 / 586 / 2.4417	0.034252608017 / 600 / 2.5000	P + (U if pass else L)	0.0889
TP4/MNS64	0.033182214016 / 575 / 2.3958	0.033717411016 / 586 / 2.4417	0.034252608017 / 600 / 2.5000	P + (U if pass else L)	0.0889

Total GPU estimate versus 3.0 cap

The 25 primary anchors contain 8 TP1, 8 TP2, and 9 TP4 replays. At the 80-second per-anchor planning charge they consume 0.1778 + 0.3556 + 0.8000 = 1.3334 H20-hours. Four shared startup/warm-up/cleanup envelopes plus runtime variance are budgeted at 1.3166 hours, for a 2.65-H20-hour primary plan. The remaining 0.35 H20-hours is reserved for boundary confirmations or decision-critical outward crawling. The absolute launch cap is 3.0 H20-hours; projected or actual spend reaching the cap stops all further work. Expected wall time is 25-35 minutes over four waves. Expected public disk is 0.3-0.6 GB and must remain below 3 GB.

Decision rules

1. Anchor feasibility is exact-output request pass rate >=0.95 using raw-length stepped TTFT and 50-ms TPOT; absent/early-stopped requests remain failures.
2. Boundary movement is an old peak feasible->infeasible flip or an old adjacent infeasible->feasible flip. Material frontier churn is abs(drift)>5%.
3. Apply zero-anchored floor buckets with set-wide tol=max(1e-9,1e-6*max|s|) before every tie/rank decision.
4. ARGMAX MOVED iff TP2/MNS32 is outside the v0.24 top bucket on a bounded 12-cell subset; censored contenders make survival inconclusive.
5. RANKING SURVIVES only with all 12 cells, argmax survival, tau-b >=0.8, and no

   5% top-bucket pair reversal. Tau-b <0.8 is RANKING MOVED.

6. TP4/MNS16 trap persists only when it is no worse than adjacent MNS8/MNS32 and remains below the global best; an improving neighbor means escape.
7. Boundary [0.93,0.97] or verdict-changing anchors require confirmation when budget permits. Disagreement without a 2-of-3 resolution is inconclusive.

Open decisions for orchestrator

1. Approve the upgrade-path estimand: resolved default changes are part of churn, while dash1->dash0 and co-location remain explicit limitations rather than pretending this is a pure engine-version causal effect.
2. Approve the adaptive 25-anchor subset, which gives every cell a direction-relevant neighbor and the trap both neighbors, instead of spending next-infeasible anchors only on a tie-ambiguous top four.
3. Approve 5% material frontier drift, tau-b>=0.8, and the stated argmax/trap floor-bucket rules.
4. Approve one primary observation per historical anchor with the 0.35-hour reserve for [0.93,0.97]/verdict-changing confirmations, rather than reducing surface coverage to replicate every anchor.
5. Approve the A-P5-1-class warm-up adaptation from input>=8192 to raw input >4096, the longest SLO-relevant tier available reliably in this 0-8192 C1 workload.

Sanity block

Numeric family	n	Min	Max	Distinct	Checked invariant/result
C1 cells	12	TP=1,MNS=8	TP=4,MNS=64	12	Exact 3x4 Cartesian surface
Historical probe observations	92	7/cell	8/cell	2 per-cell counts	8*8 + 4*7 = 92
Materialized trace rows	1 file	32,606	32,606	1 expected	SHA pinned; prompt content not emitted
Raw-length-filtered rows	1 reconstruction	17,710	17,710	1 expected	Raw length 22-8192; non-negative
Capped TP1/TP2 source rows	1 set	512	512	1 expected	Stable time sort, then even downsample
Reconstructed historical counts	92	66	600	34	92/92 match ground truth
C1 peak scores (req/s/GPU)	12	1.283333	3.283333	8	Finite, positive, not all identical
v0.20 floor tolerance	1 vector	3.283333e-6	3.283333e-6	1	Applied before display rounding
Nominal top-4 effective size	1 cutoff	5	5	1	TP4 three-way cutoff tie retained
Primary peak anchors	12	1/cell	1/cell	1 expected	No cell omitted
Realized adjacent/trap anchors	13	1/non-trap	2/trap	2	Predeclared direction rule; total 25
Primary anchors by TP	25	8 at TP1/TP2	9 at TP4	2 counts	8+8+9=25
Selected requests in anchor candidates	36 candidates	121	600	>1 expected	TP1/2 cap-before-threshold; TP4 uncapped
Per-anchor planning cost (H20-hour)	3 TP classes	0.0222	0.0889	3	Proportional to TP for 80 seconds
Primary planned H20-hours	1 plan	2.65	2.65	1	Non-negative; below 3.0
Confirmation/crawl reserve	1 plan	0.35	0.35	1	Primary plus reserve equals hard cap
GPU work in this protocol turn	1 turn	0	0	1 expected	Protocol-only requirement satisfied

Checked invariants: 12=3*4; 92=8*8+4*7; all 92 local selection counts match; cap precedes threshold; arrivals span the fixed 60-second replay; raw input length, not retokenized usage, selects TTFT thresholds; completion is exactly 128; SLO denominator includes all selected requests; surface scores use 60 seconds and divide by TP; floor buckets precede ranking; the top-four cutoff expands to five; 25=12+11+2; planned 2.65+0.35=3.0; ratios and counters have their declared domains; and no GPU run, helper, manifest, remote artifact, or results file was created in this protocol-only turn.