aituner/docs/opprof/phase6-protocol.md

# OpProf Phase 6 pre-registered cross-version churn protocol

Status: **ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED**.

Date frozen: 2026-07-12 (Asia/Singapore). Phase 6 asks whether the vLLM 0.20.0
best configuration and ranking on the C1
`TP={1,2,4} x max-num-seqs={8,16,32,64}` surface survive an upgrade to patched
vLLM 0.24.0. This is a paired historical-anchor experiment, not a fresh tuner
search. It measures churn at the exact recorded C1 request sets and reports
where the recorded local frontier moved.

## Approved dispositions (orchestrator; 2026-07-12)

All five open decisions are approved exactly as proposed, without changing an
anchor, threshold, estimator, validity gate, or budget:

1. The upgrade-path estimand is approved. Resolved default changes are part of
   observed churn; dash1->dash0 and co-location remain explicit limitations and
   the result is not described as a pure engine-version causal effect.
2. The adaptive 25-anchor subset is approved: every cell receives its old peak
   plus the direction-relevant adjacent anchor, and TP4/MNS16 receives both
   neighbors.
3. The 5% material-frontier threshold, `tau-b>=0.8` ranking-survival threshold,
   and zero-anchored floor-bucket argmax/trap rules are approved.
4. One primary observation per historical anchor plus the 0.35-H20-hour
   confirmation reserve is approved.
5. The A-P5-1-class warm-up long tier is approved as raw input `>4096` tokens
   for this 0-8192 workload.

This approval authorizes the later execution only after pinned-input, detached
ownership, cold-start, echo-before-launch, placement, privacy, cleanup, and
projected-budget gates pass. The 3.0-H20-hour cap remains absolute.

## Amendment A-P6-1 — checkpoint-sidecar accounting and conditional cap

The first Phase-6 W1 request replays were incorrectly rejected by requiring a
clean in-stream footer and `final=true` sidecar after a non-graceful shutdown.
That contradicts the already accepted A-P3-5/P5 accounting rule. Before any
resume, all four W1 cell streams and their eight measured anchor intervals must
be CPU-re-adjudicated against the latest atomic checkpoint sidecar. With no
in-stream footer, the sidecar is authoritative when every complete JSONL line
decodes, step indices are contiguous, on-disk data-record count equals
`written_records`, final data-line step equals `last_step_index`,
`encoded_records = written_records + dropped_records`, `dropped_records=0`, and
the checkpoint is within the configured one-second flush interval (plus the
accepted 100-ms scheduling tolerance) of the recorded shutdown boundary. At
most one post-checkpoint flush interval may be absent. A balanced stream accepts
all covered W1 replay intervals without a GPU rerun.

All subsequent waves use the graceful P5 path: start `vllm serve` with positive
`--shutdown-timeout`, signal each API parent with SIGINT, wait up to 150 seconds
for EngineCore drain/finalization, and use process-group SIGTERM/SIGKILL only as
fallback. The preferred result is one in-stream footer plus an agreeing
`final=true` sidecar; the atomic checkpoint rule remains the crash fallback.

Only if any W1 sidecar fails the CPU balance/coverage gate does A-P6-1c activate:
the hard cap becomes **3.5 H20-hours**, the activation is logged before launch,
and W1 is rerun under graceful shutdown. If all W1 sidecars balance, A-P6-1c is
not activated, W1 is accepted in place, its already charged 0.299393 H20-hours
is retained, and execution resumes at W2 under the original 3.0-hour cap.

## Amendment A-P6-2 — authoritative solo frontier tier and 6.0-hour cap

The user raises the cumulative Phase-6 hard cap from **3.0 to 6.0 H20-hours**.
This amendment responds to the W2/W3 confirmation evidence: when neighboring
wave clients became idle, pass rate changed from `0.412 -> 1.000` and `0.028 ->
0.957` at TP2/MNS32 and from `0.036 -> 1.000` at TP4/MNS16, while Layer-1
waiting means collapsed from `1.22 -> 0.00`, `8.50 -> 0.35`, and `28.33 ->
0.06`. Because the v0.20 C1 baseline ran one cell at a time on dash1,
co-located SLO feasibility is not a valid authoritative tier for the paired
frontier. Co-location remains an explicitly reported, metric-dependent
methodological ablation; its throughput-only observations are indicative.

Every SLO-frontier-decisive Phase-6 anchor is therefore run **solo**: exactly
one vLLM server and one replay client are active on dash0, no other Phase-6
server/client is resident, and only the cell's `TP` GPUs are allocated. Each
solo cell gets a fresh server process, the frozen exact-selection client, an
independent 16-request long-tier warm-up, A-P5-1 cold-start gates, Layer-1
telemetry, and graceful A-P6-1 shutdown. The solo result is authoritative for
feasibility, frontier, floor bucket, argmax, tau-b, and trap decisions. A
co-located-only result may not fill a missing solo frontier or support a paper
claim.

The mandatory solo set has no cell exclusions:

| Cell set | Mandatory solo anchors | Inclusion reason |
|---|---|---|
| TP1/MNS8 | existing L+P | co-located peak failed |
| TP1/MNS16,32,64 | existing P+U | co-located U passed and censored the frontier |
| TP2/MNS8,16 | existing L+P | co-located peak failed; L distinguishes bracket vs left censoring |
| TP2/MNS32 | existing L+P | old argmax; both anchors split primary/confirmation |
| TP2/MNS64 | existing L+P | the indicative `>29.4% down` bound is unquotable until solo-confirmed |
| TP4/MNS8 | existing L+P | co-located peak failed; trap-neighbor comparison |
| TP4/MNS16 | existing L+P+U | named trap; peak split and L failed co-located |
| TP4/MNS32,64 | P plus direction-relevant L/U | W4 completion; unmeasured under the 3.0-hour cap |

After these 25 anchors, the controller may crawl only the pinned 92-anchor
history: upward while the highest solo anchor passes or downward while the
lowest solo anchor fails, stopping at the first opposite-feasibility anchor.
There is no interpolation and no unrecorded request set. A pass rate in
`[0.93,0.97]`, or a solo/co-located disagreement that changes argmax/trap or a
material-frontier claim, triggers a same-solo-placement confirmation; a split
gets a third trial for the frozen 2-of-3 rule if projected budget permits.
Unresolved or history-edge frontiers remain explicitly censored.

Priority is W4, TP2/MNS32, TP2/MNS64, TP4/MNS16, then the remaining failed or
censored cells, so a budget stop preserves the most decision-relevant evidence.
All twelve cells are nevertheless included. The planning envelope is 3.44 new
H20-hours for serialized startup, warm-up, 25 mandatory anchors, bounded crawl,
graceful drain, and a 0.20-hour safety reserve. Added to the already charged
2.291173 hours, the launch projection is **5.731173/6.0 H20-hours**. Before
each one-cell wave, the controller echoes cell, anchors, GPU placement, input
paths, charged spend, remaining projection, and cap. Projected or actual spend
reaching 6.0 stops further GPU work.

Final analysis preserves the complete W1-W3 co-located attempt history and
reports a solo-versus-co-located delta table for every exact anchor pair.
ARGMAX, RANKING, and TRAP use solo values only. RANKING remains inconclusive
and tau-b non-evaluable unless all 12 solo frontiers are bounded under the
pinned history.

## Headline claim and falsifiable outcomes

The paper-facing question is:

> Does the C1 vLLM 0.20.0 per-GPU SLO-feasible ranking, especially global best
> TP2/MNS32 and local trap TP4/MNS16, remain valid on vLLM 0.24.0 when the exact
> historical request-selection and SLO semantics are replayed?

Phase 6 can produce four distinct outcomes:

1. **SURVIVES:** TP2/MNS32 remains in the top floor bucket, the full-surface
   Kendall tau-b is at least 0.8, and the TP4/MNS16 trap persists.
2. **ARGMAX MOVED:** TP2/MNS32 is not in the top v0.24 measured-frontier bucket.
3. **RANKING MOVED WITHOUT ARGMAX MOVEMENT:** the old best survives but tau-b is
   below 0.8 or the trap changes status.
4. **INCONCLUSIVE/PARTIAL:** a decision-critical frontier is unbracketed, an
   anchor is not reproducible, repeated boundary verdicts disagree, the 12-cell
   surface is incomplete, or the 3.0-H20-hour cap stops required work.

The experiment quantifies observed upgrade churn. Because engine version,
host, resolved defaults, and co-location differ together, it does not identify
a pure causal effect of one vLLM commit.

## Frozen vLLM 0.20.0 surface

The historical objective is the maximum measured SLO-feasible offered rate
divided by TP:

```text
f20(c) = max feasible selected_request_count / 60 seconds / TP
```

| TP | MNS8 | MNS16 | MNS32 | MNS64 |
|---:|---:|---:|---:|---:|
| 1 | 2.1000 | 2.3500 | 2.2833 | 2.2833 |
| 2 | 2.2750 | 2.2750 | **3.2833** | 3.2583 |
| 4 | 1.2833 | **2.4417** | 2.4417 | 2.4417 |

Values are request/s/GPU. The global best is TP2/MNS32. TP4/MNS16 is the
registered local trap: it is the first point on a TP4 plateau, has no improving
adjacent MNS move, but is below the global best.

The pinned ground-truth asset is
`/home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json`, SHA-256
`23ff3e6f6b88df8632bf37d37c0ff87bfef9d1676db81592948c08e898f7a670`.
It contains 92 probe observations: eight anchors for each TP1/TP2 cell and seven
for each TP4 cell.

## Comparability contract

### Exact workload and selection semantics

The private materialized workload is
`trace_windows/traces/chat_w20260311_1000.jsonl`, 32,606 rows, SHA-256
`f539f38eb0ee0f750e3c23ff47df6eed3faf723a25f1444d55665a85871750b9`.
Its window record is `trace_windows/windows.json`, SHA-256
`23c432e7439508b07991cabfe1db9977ebb55f8ddc39ca214a627fc5f5ae4725`:
window `[0,600]` seconds becomes `[0,60]` under replay scale 0.1.

The later helper must call or byte-for-byte reproduce the following pinned
AITuner implementations, rather than use the Phase-3/5 client:

| Source | SHA-256 | Frozen role |
|---|---|---|
| `src/aituner/trace.py` | `6bb19f0e3eb42b1aecc6969a5f20d65eb505a91564dd117dd149155a6114eb35` | load, stable time sort, cap/downsample, threshold selection |
| `src/aituner/worker.py` | `18337fe273a48d1cbb2bc364f774d100f8ffa2be5f9cd6a7c58c8ed4717c5e48` | rate-following submit loop, early stop, denominator accounting |
| `src/aituner/slo.py` | `1d2e5b1ed9d0d6d6c07f8b5b6fa56d36ed9bb62710be54c125e9b2890f4f96d5` | stepped SLO evaluation |
| `src/aituner/http_client.py` | `213b80d5b37cc74333517f46ab4e8cdba1a274bdbfabdfc14207b314f61799ac` | urllib/SSE TTFT, TPOT, usage accounting |

The exact order is normative:

1. Read all JSONL rows in file order. Retain raw `input_length` in `[0,8192]`.
2. Build the same chat body and stable-sort by scaled arrival time only. Stable
   ties retain source order; sorting by `(timestamp,sampling_u,...)` is forbidden.
3. For TP1/TP2, even-downsample the full filtered, time-sorted list to 512
   entries **before** threshold selection, using index
   `floor(i*N/512)` for `i=0..511`. TP4 is uncapped.
4. Select exactly the requests with `sampling_u <= anchor`. Preserve their
   selected service order. Do not resample, wrap, shuffle, or interpolate.
5. Submit at each row's materialized `timestamp*0.1`; do not uniformize,
   rebase individual gaps, or replace the recorded burst process.
6. Send the original messages through `/v1/chat/completions` with `stream=true`,
   `stream_options.include_usage=true`, and the row's temperature when present.
   Set both `min_tokens=128` and `max_tokens=128`.
7. Use maximum concurrency 64. Client request timeout remains 900 seconds.
8. Verify `usage.completion_tokens==128`; missing usage, a different token count,
   HTTP failure, timeout, and every request not submitted because the SLO verdict
   became unrecoverable are failures in the original selected-request denominator.
9. Preserve original early-stop logic: after more than
   `N-ceil(0.95*N)` failed evaluations, synthesize failures for the remaining
   selected requests. Disable adaptive Stop-A; it was absent in C1.

An offline, prompt-free preflight must regenerate all 92 historical request
counts exactly. The local reconstruction already yields 17,710 filtered rows,
512 capped rows, and **92/92 count matches**. Any future mismatch in count,
request-ID hash, arrival hash, raw-length hash, or requested-token sum stops the
GPU launch.

### Exact SLO and score

Request feasibility uses the original raw trace input length, not tokenizer
usage after chat templating:

```text
raw input <= 4096:  TTFT <= 2000 ms
raw input <= 32768: TTFT <= 4000 ms
otherwise:          TTFT <= 6000 ms
all requests:       TPOT <= 50 ms
anchor feasible:    SLO-passing selected requests / all selected requests >= 0.95
```

Although the C1 filter limits inputs to 8192, the complete three-step rule is
retained. TTFT is client wall time from request start to first nonempty content.
TPOT is `(last_content_time-first_content_time)/(usage_completion_tokens-1)`.
The paired surface score is always selected count divided by 60 seconds and TP;
completed throughput is a diagnostic and never replaces the historical score.

### Serving configuration

The v0.24 command repeats the explicit C1 launch surface:

```bash
MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
VENV=/tmp/wjh-opprof-phase2-dash0-20260711/.venv
RUN_ROOT=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6

taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU_IDS" \
  VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
  "$VENV/bin/vllm" serve "$MODEL" \
  --host 127.0.0.1 --port "$PORT" \
  --served-model-name qwen3-30b-a3b-community \
  --max-num-batched-tokens 8192 \
  --max-num-seqs "$MNS" \
  --tensor-parallel-size "$TP" \
  --shutdown-timeout 120
```

`--shutdown-timeout` is operational and outside the measured interval. As in
C1, do **not** explicitly set block size, GPU-memory utilization, prefix
caching, chunked prefill, CUDA-graph capture sizes, DP, or EP. Their resolved
v0.24 values must be parsed from startup logs and reported. Allowing version
defaults to change is intentional: this experiment asks whether an operator's
old explicit configuration remains good after upgrading, not whether internal
defaults can be manually made identical.

Preflight records `vllm.__version__`, source commit, patch checksums, Python,
PyTorch, CUDA, NCCL, Triton, driver, model/tokenizer file hashes, MoE backend,
resolved scheduler/cache/graph config, compile-cache key, and exact command.
The expected source is the existing Phase-3/5 patched vLLM 0.24 environment;
any missing OpProf patch or unexpected model path is a stop condition.

### Client equivalence, not a new client covariate

The later `scripts/opprof_phase6_client.py` is permitted only as a thin anchor
driver around the four pinned AITuner modules. Its interface is normative:

```bash
P6C='python scripts/opprof_phase6_client.py'

$P6C preflight \
  --trace trace_windows/traces/chat_w20260311_1000.jsonl \
  --windows trace_windows/windows.json --window-id chat_w20260311_1000 \
  --ground-truth /home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json

$P6C run-anchor --cell "$CELL" --anchor "$U" \
  --base-url "http://127.0.0.1:$PORT" \
  --model qwen3-30b-a3b-community --completion-tokens 128 \
  --replay-time-scale 0.1 --max-concurrency 64 \
  --max-requests "$CAP_OR_NONE" --target-pass-rate 0.95 \
  --ttft-step-ms '4096:2000,32768:4000,inf:6000' --tpot-ms 50 \
  --request-timeout-seconds 900 --probe-deadline-ceiling-seconds 900 \
  --result-dir "$ANCHOR_DIR"
```

Before GPU execution, a no-server golden test must prove identical selected IDs,
order, arrivals, raw input lengths, serialized bodies excluding model/URL, and
SLO decisions for synthetic outcomes. The wrapper may add interval markers and
sanity JSON, but may not change HTTP transport, scheduling, completion checks,
early stop, or latency formulas.

### Covariate ledger and estimand limit

| Covariate | C1 v0.20 | Phase 6 v0.24 | Treatment/handling |
|---|---|---|---|
| Engine | community vLLM 0.20.0 | patched vLLM 0.24.0 | Primary intended churn dimension; exact builds recorded |
| Host | dash1 | dash0 | Both 8x H20 96 GB class; CPU/NUMA/clocks/driver/kernel recorded, not assumed identical |
| GPU visibility | TP1 C1 env exposed GPUs `0,1`, TP1 consumed one | expose exactly TP GPUs | Declared operational covariate; H20-hour charge follows TP |
| Trial placement | recovered C1 harness trials | up to four co-located servers | CPU/NUMA pinned; reject outside GPU processes and report co-location limitation |
| HTTP client | pinned AITuner urllib/SSE | same code paths through thin wrapper | Eliminated as a semantic covariate by golden tests |
| Workload | materialized C1 chat window | byte-identical file/hash | Fixed |
| Selection/cap | cap-before-threshold; TP4 uncapped | exact same | Fixed and 92-count preflighted |
| SLO length | raw input length | raw input length | Fixed; server usage length is diagnostic only |
| Explicit flags | TP, MNS, MBT8192 | same | Fixed |
| Unspecified defaults | v0.20 defaults | v0.24 defaults | Intentionally allowed to churn; resolved values reported |
| Telemetry | no Layer 1 | zero-overhead OpProf Layer 1 | Added measurement; overhead evidence -0.04196%, CI [-0.17443%, 0.04550%] |

Therefore “version churn” means the observed paired upgrade path under these
documented platform covariates. It must not be rewritten as a same-host causal
microbenchmark.

## Budget-constrained anchor subset

### Adaptive adjacent-anchor design

The primary subset improves on measuring next-infeasible anchors only for a
chosen top four:

1. Measure the recorded v0.20 peak anchor for all 12 cells.
2. For every non-trap cell, select exactly one adjacent recorded anchor by a
   result-independent rule fixed now:
   - if the v0.20 peak remains feasible on v0.24, measure the nearest higher
     recorded anchor, which was infeasible on v0.20;
   - if the peak becomes infeasible, measure the nearest lower recorded anchor,
     which was feasible on v0.20.
3. For TP4/MNS16, measure lower, peak, and higher anchors unconditionally. This
   directly brackets the registered trap.

Thus exactly 25 primary anchors are measured: TP1 eight, TP2 eight, and TP4
nine. The design covers every cell and spends the second anchor in the direction
where its frontier could have moved. It also includes the suggested top-four
test. Under the floor-bucket rule, the nominal top-four cutoff expands to five
cells because TP4/MNS16/32/64 share one bucket; all five receive an upward test
whenever their old peak still passes.

If an adjacent anchor does not bracket the transition—higher also passes or
lower also fails—the cell is censored. Remaining budget may crawl one recorded
anchor farther outward, in this priority order: old global best TP2/MNS32,
TP2/MNS64, the TP4 plateau MNS16/32/64, then other cells by old score. Crawling
uses only that cell's recorded history and stops at the first transition. No
new `sampling_u` value may be invented. Failure to close a decision-critical
bracket before the budget stop makes the applicable argmax/ranking verdict
inconclusive.

### Boundary confirmations

The original surface has one observation per anchor. Phase 6 keeps that paired
unit but reserves at most 0.35 H20-hours for decision-triggered confirmation.
Repeat an anchor if its v0.24 pass rate lies in `[0.93,0.97]`, or if its single
verdict alone changes argmax or trap status. Priority is argmax, trap, material
frontier calls, then other cells. Agreement across two runs freezes the verdict;
disagreement requests a third run if budget permits and uses 2-of-3. Without a
third run, that anchor and every dependent headline verdict are inconclusive.
Request outcomes are arrival-correlated, so a binomial CI is descriptive only
and cannot replace these repeat rules.

## Execution plan

### Preflight and detached ownership

The Phase-6 controller follows A-P3-2 detached ownership: one controller PID,
per-stage atomic state, process-group/environment markers, resumable completed
anchors, exact command logs, and cleanup restricted to its own markers. Before
launch it must verify dash0 has eight H20s, no outside compute processes on
assigned GPUs, at least 100 GB CPFS free, pinned source/workload/helper hashes,
all 92 selection counts, legal TP placement, and a projected total below 3.0
H20-hours.

The autonomous launch log must echo one resolved line before starting:

```text
LAUNCH_ECHO host=dash0 engine=vllm-0.24.0-patched model=Qwen3-30B-A3B trace_sha=f539f38e... cells=12 primary_anchors=25 waves=4 gpus=0-7 warmup=16 clean_horizon=60s drain=derived<=120s est_wall=25-35min est_gpu=2.65_H20h confirm_reserve=0.35_H20h hard_cap=3.0_H20h
```

### Rate-following cold-start and drain gates

The throughput-drift stationarity gate is forbidden for these rate-following
anchors. A-P5-1-class cold-start gates are applied instead:

1. Every torch.compile/CUDA-graph first-occurrence event must precede the first
   measured anchor, verified from server logs and Layer 1.
2. A deterministic unmeasured warm set must complete at least 16 exact-output
   requests, including at least one request with raw input `>4096` tokens. The
   peak-selected sets contain 43-184 such requests, so this gate is feasible.
3. No first-occurrence compile/capture event may occur in any measured anchor;
   all graph-hit `(runtime_mode,bucket_tokens)` descriptors must be covered by
   startup capture logs. A violation invalidates that anchor only.
4. Before the next anchor, all selected requests must be accounted for and three
   consecutive controller samples must show running/waiting/deferred queues at
   zero. Prefix cache is not explicitly enabled; nevertheless a full drain is
   required.

The probe deadline reproduces `_probe_drain_deadline`: last selected arrival +
raw-length TTFT budget + `128*50 ms` + 30 seconds, capped by the historical
900-second ceiling. With raw inputs <=8192 this is about 100.4 seconds from
probe start; the controller has a 120-second class watchdog for cleanup. A
deadline miss is an SLO-infeasible anchor, not permission to omit denominator
requests. The server then shuts down through the official 120-second path.

### Four-wave packing

| Wave | Cells | Placement | Per-server measured anchors | Planned wall |
|---|---|---|---:|---:|
| W1 | TP1/MNS8,16,32,64 | four servers on GPUs 0,1,2,3; GPUs 4-7 idle | 2 each | 4-6 min |
| W2 | TP2/MNS8,16,32,64 | four servers on GPU pairs 0-1,2-3,4-5,6-7 | 2 each | 4-6 min |
| W3 | TP4/MNS8 and TP4/MNS16 trap | GPU quads 0-3 and 4-7 | 2 and 3 | 5-7 min |
| W4 | TP4/MNS32 and TP4/MNS64 | GPU quads 0-3 and 4-7 | 2 each | 4-6 min |

Each server starts once, passes warm-up once, runs its peak first, drains, then
runs the predeclared adjacent anchor. Trap lower/upper order after peak is
counterbalanced by execution seed 20260717. Anchor confirmation occurs before
that server is released when possible. The controller records GPU clocks,
power, utilization, memory, CPU load, NUMA placement, and outside processes
through every wave. Any contamination invalidates the whole co-located wave.

Actual H20-hours are charged from server launch through verified cleanup for
every allocated GPU, including idle time while a paired TP4 cell finishes.
Before each wave or confirmation, the controller recomputes
`spent + current + conservative_remaining`; it stops before launch if the
projection can reach 3.0. Optional crawling and confirmations are sacrificed
before any hard-cap violation.

## Measurement and mechanism notes

For every anchor, report selected/admitted/completed/exact-output/SLO-pass
counts; early-stop reason; offered and completed req/s; TTFT/TPOT mean and
p50/p90/p95/p99; failures by raw-length TTFT bucket; elapsed/drain time; and
peak GPU memory/utilization. The surface objective remains offered req/s/GPU at
the highest measured feasible recorded anchor.

Layer 1 is active over the complete anchor service interval, from the first
submission through the last selected-request accounting event. Aggregate:

- prefill, decode, and mixed-step counts and token shares;
- decode-batch distribution and 5-second CV;
- FULL/PIECEWISE/NONE graph-mode shares, bucket padding, and uncovered support;
- running/waiting/deferred queues and queue CV;
- preemption count/rate;
- KV blocks used/total, mean/max usage, and allocation pressure; and
- prefix queries/hits as a resolved-default diagnostic.

Mechanism notes compare v0.24 cells/anchors and may relate a frontier change to
preemptions, KV pressure, batching, or graph modes. C1 lacks Layer-1 telemetry,
so these notes cannot claim that a measured composition field itself changed
from v0.20 or caused the version drift. Historical v0.20 TTFT/TPOT failure
reasons may be paired directly; device-mechanism attribution remains
descriptive.

## Statistical and ranking analysis

### Floor buckets

For each unrounded score vector `s`, use the SimFid inventory rule:

```text
tol(s) = max(1e-9, 1e-6 * max_c(abs(s(c))))
bucket_s(c) = floor(s(c) / tol(s))
```

Two scores tie only when their integer buckets match. Bucketing occurs before
display rounding, pair signs, tau-b, top selection, or trap evaluation. Report
`tol`, bucket integers, and occupied intervals `[j*tol,(j+1)*tol)`. For v0.20,
`tol20=3.283333333333333e-6`; the TP4 MNS16/32/64 tie makes nominal top-4 an
effective top-5 set.

### Per-cell frontier

For each cell, define the v0.24 measured recorded-anchor frontier:

```text
f24(c) = max_{measured u with accepted feasible verdict} N_c(u) / 60 / TP
drift(c) = (f24(c) - f20(c)) / f20(c)
```

The predeclared material threshold is **X=5% relative**, large enough to avoid
calling one-request anchor quantization a paper-level churn effect. Report two
orthogonal labels:

- **BOUNDARY DOWN/UP:** old peak flips feasible->infeasible, or the old adjacent
  infeasible anchor flips to feasible, respectively;
- **MATERIAL FRONTIER MOVED:** `abs(drift)>5%`; otherwise report the boundary
  flip as sub-material.

If peak passes and higher fails, and `abs(drift)<=5%`, the local recorded
frontier is **STABLE**. If peak and lower both fail or peak and every affordable
higher anchor pass, report a directional bound and **UNBRACKETED**, never an
invented peak.

### Argmax, ranking, and trap

- **ARGMAX MOVED** when TP2/MNS32 is not in the top `bucket_f24` among all 12
  decision-bounded cells. **ARGMAX SURVIVED** requires it to share the top
  bucket and no censored competitor capable of overtaking it. Otherwise it is
  inconclusive.
- Compute Kendall tau-b over all 12 floor-bucketed `f20/f24` scores and report
  concordant, discordant, v0.20-tied, v0.24-tied, and comparable-pair counts.
  **RANKING SURVIVES** requires an evaluable 12-cell surface, argmax survival,
  `tau-b>=0.8`, and no >5% top-bucket pair reversal. `tau-b<0.8` is **RANKING
  MOVED**. A partial surface receives no suite-level tau claim.
- **TRAP PERSISTS** when TP4/MNS16 is in a bucket at least as high as both
  adjacent TP4 cells MNS8 and MNS32, yet below the global-best bucket. It
  **ESCAPES** if an adjacent TP4 move is strictly better, and **CEASES TO BE A
  TRAP** if it joins the global-best bucket. Missing/unbracketed inputs make trap
  status inconclusive.

No arithmetic average of normalized scores is reported. Absolute rates,
relative drift, the paired table, and every excluded/censored anchor accompany
all ranking summaries.

## Later deliverables and artifact contract

Execution, after approval, must produce:

- `docs/opprof/phase6-results.md`;
- `runs/opprof-phase6/phase6/metrics.json` containing the complete paired
  surface, anchor/run values, selection hashes, feasibility, floor buckets,
  frontier/argmax/ranking/trap verdicts, confirmation history, Layer-1 notes,
  covariates, and GPU accounting;
- exact commands, controller state, environment/machine fingerprints, server
  logs, client outcomes, Layer-1 JSONL/footer/sidecar, monitor samples, and
  per-anchor sanity JSON; and
- an operational-findings section covering startup/capture gates, drains,
  early stops, co-location, default changes, retries, contamination, and budget.

The results begin with data sanity and stop inferential claims on any red flag.
Every numeric family ends with `n`, finite/missing, min, max, distinct count,
and applicable invariants. Prompt text, messages, generated content, source
substrings, and private request bodies stay in mode-0700 storage. Public files
contain only hashes, IDs, lengths, timestamps, outcomes, counters, and summaries.

## Anchor-subset table

Notation is `u / selected N / recorded req/s/GPU`. `P` is always measured. For
non-trap cells, measure `U` if P passes on v0.24, otherwise `L`. TP4/MNS16
measures all `L+P+U`. Planning cost per anchor includes the 60-second replay and
20 seconds expected drain/accounting; shared startup/warm-up is budgeted below.

| Cell | L: nearest recorded feasible below P | P: recorded peak | U: nearest recorded infeasible above P | Primary selection | H20-hour cost each |
|---|---|---|---|---|---:|
| TP1/MNS8 | 0.21875 / 121 / 2.0167 | 0.2265625 / 126 / 2.1000 | 0.23046875 / 130 / 2.1667 | P + (U if pass else L) | 0.0222 |
| TP1/MNS16 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | 0.25 / 143 / 2.3833 | P + (U if pass else L) | 0.0222 |
| TP1/MNS32 | 0.234375 / 132 / 2.2000 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | P + (U if pass else L) | 0.0222 |
| TP1/MNS64 | 0.234375 / 132 / 2.2000 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | P + (U if pass else L) | 0.0222 |
| TP2/MNS8 | 0.4921875 / 269 / 2.2417 | 0.49609375 / 273 / 2.2750 | 0.5 / 276 / 2.3000 | P + (U if pass else L) | 0.0444 |
| TP2/MNS16 | 0.4921875 / 269 / 2.2417 | 0.49609375 / 273 / 2.2750 | 0.5 / 276 / 2.3000 | P + (U if pass else L) | 0.0444 |
| TP2/MNS32 | 0.75 / 391 / 3.2583 | 0.75390625 / 394 / **3.2833** | 0.7578125 / 396 / 3.3000 | P + (U if pass else L) | 0.0444 |
| TP2/MNS64 | 0.5 / 276 / 2.3000 | 0.75 / 391 / 3.2583 | 0.75390625 / 394 / 3.2833 | P + (U if pass else L) | 0.0444 |
| TP4/MNS8 | 0.016055910008 / 301 / 1.2542 | 0.016591107009 / 308 / 1.2833 | 0.017126304009 / 317 / 1.3208 | P + (U if pass else L) | 0.0889 |
| TP4/MNS16 trap | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / **2.4417** | 0.034252608017 / 600 / 2.5000 | **L + P + U** | 0.0889 |
| TP4/MNS32 | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / 2.4417 | 0.034252608017 / 600 / 2.5000 | P + (U if pass else L) | 0.0889 |
| TP4/MNS64 | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / 2.4417 | 0.034252608017 / 600 / 2.5000 | P + (U if pass else L) | 0.0889 |

## Total GPU estimate versus 3.0 cap

The 25 primary anchors contain 8 TP1, 8 TP2, and 9 TP4 replays. At the
80-second per-anchor planning charge they consume `0.1778 + 0.3556 + 0.8000 =
1.3334` H20-hours. Four shared startup/warm-up/cleanup envelopes plus runtime
variance are budgeted at 1.3166 hours, for a **2.65-H20-hour primary plan**.
The remaining **0.35 H20-hours** is reserved for boundary confirmations or
decision-critical outward crawling. The absolute launch cap is **3.0
H20-hours**; projected or actual spend reaching the cap stops all further work.
Expected wall time is 25-35 minutes over four waves. Expected public disk is
0.3-0.6 GB and must remain below 3 GB.

## Decision rules

1. Anchor feasibility is exact-output request pass rate `>=0.95` using raw-length
   stepped TTFT and 50-ms TPOT; absent/early-stopped requests remain failures.
2. Boundary movement is an old peak feasible->infeasible flip or an old adjacent
   infeasible->feasible flip. Material frontier churn is `abs(drift)>5%`.
3. Apply zero-anchored floor buckets with set-wide `tol=max(1e-9,1e-6*max|s|)`
   before every tie/rank decision.
4. ARGMAX MOVED iff TP2/MNS32 is outside the v0.24 top bucket on a bounded
   12-cell subset; censored contenders make survival inconclusive.
5. RANKING SURVIVES only with all 12 cells, argmax survival, tau-b >=0.8, and no
   >5% top-bucket pair reversal. Tau-b <0.8 is RANKING MOVED.
6. TP4/MNS16 trap persists only when it is no worse than adjacent MNS8/MNS32 and
   remains below the global best; an improving neighbor means escape.
7. Boundary `[0.93,0.97]` or verdict-changing anchors require confirmation when
   budget permits. Disagreement without a 2-of-3 resolution is inconclusive.

## Open decisions for orchestrator

1. Approve the **upgrade-path estimand**: resolved default changes are part of
   churn, while dash1->dash0 and co-location remain explicit limitations rather
   than pretending this is a pure engine-version causal effect.
2. Approve the **adaptive 25-anchor subset**, which gives every cell a
   direction-relevant neighbor and the trap both neighbors, instead of spending
   next-infeasible anchors only on a tie-ambiguous top four.
3. Approve **5% material frontier drift**, `tau-b>=0.8`, and the stated
   argmax/trap floor-bucket rules.
4. Approve one primary observation per historical anchor with the 0.35-hour
   reserve for `[0.93,0.97]`/verdict-changing confirmations, rather than reducing
   surface coverage to replicate every anchor.
5. Approve the A-P5-1-class warm-up adaptation from `input>=8192` to
   **raw input >4096**, the longest SLO-relevant tier available reliably in this
   0-8192 C1 workload.

## Sanity block

| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
|---|---:|---:|---:|---:|---|
| C1 cells | 12 | TP=1,MNS=8 | TP=4,MNS=64 | 12 | Exact 3x4 Cartesian surface |
| Historical probe observations | 92 | 7/cell | 8/cell | 2 per-cell counts | `8*8 + 4*7 = 92` |
| Materialized trace rows | 1 file | 32,606 | 32,606 | 1 expected | SHA pinned; prompt content not emitted |
| Raw-length-filtered rows | 1 reconstruction | 17,710 | 17,710 | 1 expected | Raw length 22-8192; non-negative |
| Capped TP1/TP2 source rows | 1 set | 512 | 512 | 1 expected | Stable time sort, then even downsample |
| Reconstructed historical counts | 92 | 66 | 600 | 34 | 92/92 match ground truth |
| C1 peak scores (req/s/GPU) | 12 | 1.283333 | 3.283333 | 8 | Finite, positive, not all identical |
| v0.20 floor tolerance | 1 vector | 3.283333e-6 | 3.283333e-6 | 1 | Applied before display rounding |
| Nominal top-4 effective size | 1 cutoff | 5 | 5 | 1 | TP4 three-way cutoff tie retained |
| Primary peak anchors | 12 | 1/cell | 1/cell | 1 expected | No cell omitted |
| Realized adjacent/trap anchors | 13 | 1/non-trap | 2/trap | 2 | Predeclared direction rule; total 25 |
| Primary anchors by TP | 25 | 8 at TP1/TP2 | 9 at TP4 | 2 counts | `8+8+9=25` |
| Selected requests in anchor candidates | 36 candidates | 121 | 600 | >1 expected | TP1/2 cap-before-threshold; TP4 uncapped |
| Per-anchor planning cost (H20-hour) | 3 TP classes | 0.0222 | 0.0889 | 3 | Proportional to TP for 80 seconds |
| Primary planned H20-hours | 1 plan | 2.65 | 2.65 | 1 | Non-negative; below 3.0 |
| Confirmation/crawl reserve | 1 plan | 0.35 | 0.35 | 1 | Primary plus reserve equals hard cap |
| GPU work in this protocol turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied |

Checked invariants: `12=3*4`; `92=8*8+4*7`; all 92 local selection counts
match; cap precedes threshold; arrivals span the fixed 60-second replay; raw
input length, not retokenized usage, selects TTFT thresholds; completion is
exactly 128; SLO denominator includes all selected requests; surface scores use
60 seconds and divide by TP; floor buckets precede ranking; the top-four cutoff
expands to five; `25=12+11+2`; planned `2.65+0.35=3.0`; ratios and counters have
their declared domains; and no GPU run, helper, manifest, remote artifact, or
results file was created in this protocol-only turn.