aituner/docs/opprof/phase5-protocol.md

# OpProf Phase 5 pre-registered mechanism-decomposition protocol

Status: **ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED**.

Date frozen: 2026-07-12 (Asia/Singapore). This document specifies Phase 5 only.
It does not authorize a GPU launch, helper implementation, trace transfer, or
change to the accepted vLLM patch series. Any change to an estimand, request
set, service order, arrival transform, capture-size set, load, validity gate,
or decision threshold requires a dated amendment before the affected run.

## Approved dispositions (orchestrator; 2026-07-12)

All five open decisions below are approved before execution. These dispositions
are normative and close the protocol review gate without changing any frozen
estimand, command delta, validity threshold, or budget:

1. The recorded-arrival **bridge ledger** is approved. Every machine and human
   output must state that it decomposes the recorded-arrival P5 gap anchored to
   P3 controls, not literally P3's already-uniform P10 gap; A3 supplies the
   explicit bridge back to P3.
2. The dual P03/P04 control ledgers are approved. Both are always reported and
   a dominant-mechanism call must pass the frozen rule under both denominators.
3. A1 is approved exactly as specified: 142 requests, 32-request reorder
   blocks, 16-request analysis cohorts, frozen bins, and a 64-second fairness
   cap.
4. Three P10 replicates per arm, P3 control reuse behind the 3% bridge gate,
   the conditional control reruns, and optional-tier ordering under the 6.0
   H20-hour hard cap are approved.
5. Layer-1-only primary measurement is approved. Routed-expert telemetry is
   analysis-only, private, optional, and never receives a causal ledger share.

This approval authorizes the later execution turn only when its preflight,
echo-before-launch, detached-controller, long-context, privacy, accounting,
cleanup, and budget gates all pass.

## Amendment A-P5-1 — rate-following cold-start gate (orchestrator; 2026-07-12)

The first Phase-5 wave correctly hard-stopped because all four offered-load
arms failed the inherited A-P3-6 throughput-drift gate: recorded base/A4 drift
was approximately 216.5%, A1 was 190.6%, and uniform A3 was 13.23%, versus the
frozen 10% limit. Their 240-second clean windows, output work, offered rates,
Layer-1 accounting, and drains otherwise passed. The gate was semantically
wrong for these arms: a rate-following run's scheduled-token throughput follows
its arrival process by design, so recorded non-stationarity is treatment signal,
not cold-start contamination. The large recorded-versus-uniform drift gap is
retained as direct arrival-mechanism evidence. Throughput-drift stationarity
remains appropriate and unchanged for saturation arms.

For every **rate-following/offered-load** arm, A-P3-6 is replaced by all three
of the following cold-start-artifact gates:

1. Every logged torch.compile or CUDA-graph capture first-occurrence event must
   precede the clean boundary. Match server-log event messages containing
   `torch.compile took`, `Directly load AOT compilation`, `Compiling`, or
   `Capturing CUDA graphs` (case-insensitive). Timestamped events are compared
   against `t0_wall_ns + 60 s`. vLLM's capture progress lines have no timestamp;
   they count as pre-client only when their log-line order precedes the server
   ready/startup-complete marker, which itself precedes client `t0`. Any matching
   event after readiness must have a parseable timestamp and precede clean;
   otherwise the run is invalid.
2. At least 16 requests must complete successfully in `[0,60 s)`, including at
   least one request whose recorded `input_tokens >= 8192`.
3. No first-occurrence capture event may appear inside `[60,300 s)`. Parse the
   configured startup capture-size set and the completed FULL/PIECEWISE startup
   capture passes from the server log. From Layer 1, define a captured replay
   descriptor as `(runtime_mode,bucket_tokens)` for every model-executed
   `cudagraph.hit=true` step. Every clean descriptor must be covered by the
   startup-captured mode and bucket set, and no server-log compile/capture event
   may occur inside clean. Warm and clean descriptor sets are both reported;
   a descriptor's first **replay** in clean is not mislabeled as a first capture
   when startup logs prove it was already captured. An uncovered descriptor or
   clean-window capture/compile event invalidates that run only.

The report records matched server-log events, warm-up completion/long-request
counts, warm-up and clean descriptor sets, and clean-only descriptors per run.
Absence of any required log timestamp, Layer-1 interval, or request record is a
gate failure. The original 10% A-P3-6 drift criterion remains mandatory for
closed-loop saturation arms. This amendment changes no request set, arrival
transform, service order, server configuration, clean interval, metric,
bootstrap, share estimator, dominance rule, control-reuse gate, or GPU budget.

## Goal, system boundary, and success criterion

Phase 3 measured a total useful-token-efficiency gap between irregular patterns
and rectangular controls but did not causally allocate it. Phase 5 asks how much
of the P10 gap is recovered when one treatment at a time removes:

1. intra-cohort input-length raggedness;
2. CUDA-graph decode-batch capture-bucket mismatch;
3. recorded arrival burstiness; or
4. usable natural-prefix structure.

The primary system remains Qwen3-30B-A3B BF16, patched vLLM 0.24.0, C00, TP1,
one H20 per server on dash0. The primary load is the P3 P10 rate
`lambda = 0.60 * 0.7875 = 0.4725 request/s`; it is held fixed across arms so an
ablation changes one treatment, not offered demand. The 240-second clean window
and Layer-1 definition of

```text
E_token = sum(prefill_tokens + decode_tokens) / sum(model-step duration_ms)
```

are inherited unchanged. Layer 2 is not needed for the causal ledger and is not
enabled in primary throughput runs.

Success is a mechanism ledger with an absolute `E_token` for every arm, an
un-normalized share and bootstrap confidence interval for every mechanism, and
an explicit residual/interaction line. A null or negative share is a valid
result. Merely recovering the expected sign is not success.

## Pinned Phase-3 evidence and the arrival-estimand discrepancy

The frozen P3 C00-TP1 moderate values are:

| Cell | `E_token` (tokens/ms) | Role |
|---|---:|---|
| P10 | 2.6191132083 | irregular P3 base |
| P03 | 4.7355997154 | long-input/short-output rectangular control |
| P04 | 3.0547035940 | long-input/long-output rectangular control |

P10 therefore lost 44.693% versus P03 and 14.260% versus P04. P3 used both
controls, so Phase 5 reports two parallel ledgers, one per frozen control. It
never selects the denominator that makes a mechanism look largest. A mechanism
is called control-robust only when its decision agrees under both ledgers.

There is one blocking provenance discrepancy. The private source contains
`timestamp` and `source_timestamp`, but P3's materializer discarded them and
set every P10 row to `arrival=steady`; the P3 client then admitted requests at
exactly `1/lambda`. Thus P3 has no recorded-arrival burstiness to remove.

The recommended resolution, pending orchestrator approval, is a **P5 bridge
ledger**: the P5 base replays the same P10 requests at rate-normalized recorded
timestamps, A3 uniformizes those timestamps, and P03/P04 remain the frozen P3
controls. A3 also acts as a bridge back to the P3 steady workload. This meets
the requested arrival ablation but decomposes a recorded-arrival P5 gap, not
literally the already-uniform P3 gap. The report must show both
`E_A3 - E_P3_P10` and its CI before relating the P5 ledger to P3.

If the orchestrator rejects this rebase, the P3-exact alternative is mandatory:
A3's share is `0 / N/A by construction`, and recorded arrival is reported only
as a stress sensitivity, not as a mechanism share. It is scientifically invalid
to call the reverse, burstiness-injecting treatment an ablation that removes
arrival dynamics. No GPU work may begin before this decision is recorded.

## Common request set and exact arrival transforms

The primary request set is the first **142** rows of the frozen 4,011-row P10
selection in original source order. This is exactly the number of admissions at
`lambda=0.4725` over `[0,300 s)`: scheduled times are
`0, 1/lambda, ..., 141/lambda`. All five arms contain the same 142 request IDs,
prompts, per-request input/output lengths, and aggregate input/output token
totals. There is no wrap, replacement, cancellation, or resampling.

The Phase-5 materializer must preserve `timestamp` as private metadata. Let
`z_i` be its stable source-order timestamp for request `i`, and let `N=142`.
The two arrival vectors are frozen as:

```text
recorded-scaled: a_i = (z_i-z_0) * ((N-1) / (lambda*(z_(N-1)-z_0)))
uniformized:     a_i = i / lambda
```

`z_(N-1)` must exceed `z_0`; timestamps must be finite and nondecreasing.
Ties remain ties. Both vectors start at zero, end at `141/lambda`, have the same
mean rate, and use the same request order except in A1. The client schedules
against `a_i` directly and does not add jitter. This preserves the recorded
inter-arrival shape while preventing mean-rate differences from masquerading as
an arrival mechanism.

The protocol requires a small `scripts/opprof_phase5_client.py` extension in a
later, no-GPU implementation turn. Before execution, CPU-only tests must prove:

- exact 142-row identity and token sums across all manifests;
- timestamp normalization endpoints and nonnegative gaps;
- uniform gaps equal `1/0.4725` within 1 microsecond;
- the A1 fairness bound and deterministic ordering;
- fixed 60+240-second timing, no wrap, exact output work, and text redaction.

Its reviewed SHA-256 and all manifest SHA-256 values are frozen in the detached
controller before GPU use.

## Falsifiable mechanism estimands

For arm `m` and rectangular control `c in {P03,P04}`:

```text
gap_c       = E_control,c - E_base
delta_m     = E_ablated,m - E_base
share_m,c   = delta_m / gap_c
```

The same base and same control are used for all four mechanisms within a
ledger. No share is clipped to `[0,1]`. Shares may be negative, exceed one, or
sum above/below one because single-factor interventions can overlap or interact.

The arithmetic residual/interaction line is

```text
share_residual+interaction,c = 1 - sum_m share_m,c
```

with a joint bootstrap CI. This is bookkeeping, not proof that the remainder is
one separable mechanism. It may contain unmeasured mechanisms, non-additivity,
double-counting, MoE routing, chunked-prefill interference, and measurement
error. Individual shares are never renormalized to total 100%; the residual is
never clipped to make the table visually close.

### A1 — length-binned service order

**Hypothesis.** Length-homogeneous local cohorts reduce ragged-attention/SM
imbalance, so `E_A1 > E_base`. The hypothesis is falsified if the registered
manipulation check fails or the Holm-corrected efficiency contrast is not
positive.

Starting from consecutive **32-request reorder blocks** in original P10 order,
assign each request to the fixed input-length bins

```text
[0,512], [513,1024], [1025,2048], [2049,4096],
[4097,8192], [8193,16384], [16385,32768]
```

and stable-sort each block by `(bin_id, input_tokens, original_index)`. This
creates two more homogeneous 16-request analysis cohorts per complete reorder
block. Assign the sorted requests to the block's unchanged recorded-scaled
arrival slots. If that assignment would delay any request by more than **64
seconds** relative
to its original slot, choose the earliest-deadline request first until all
deadlines are feasible, then resume the length order. Early movement is allowed;
late movement is capped. Ties are stable and no sorting crosses a 32-request
block.

On consecutive complete 16-request cohorts of evaluation-slice service order,
define `R16 = 1 - sum(L_i) / sum(16*max_cohort(L_i))`; the incomplete final
cohort is excluded and reported. Frozen pre-run values are base `R16=0.641744`
and sorted `R16=0.473409`, a 0.168334 absolute (26.23% relative) reduction;
plain sorting's maximum added delay is 62.744 seconds. The manipulation passes
only if regenerated values match these within `1e-6`, `R16` falls by at least
20% relative and 0.15 absolute, and no request violates the 64-second delay
bound.
Arrival-slot timestamps, request/content multiset, per-request output lengths,
total tokens, server config, and prefix-caching setting are identical to base.

A1 estimates the total effect of changing cohort composition. It does not
claim to isolate a particular attention kernel: service order can mediate
decode-batch composition, chunked-prefill mixing, cache locality, and
content-bound MoE routing. If the prefix-query hit ratio changes by more than
one percentage point, or the normalized inter-arrival vector changes at all,
the arm is labeled confounded and has no publishable raggedness share.

### A2 — measured decode-B capture sizes (config-tier deliverable)

**Hypothesis.** Exact capture sizes for P10's observed pure-decode batch support
remove decode-bucket slack, so pure-decode padding falls and `E_A2 > E_base`.
No recovery falsifies the efficiency hypothesis; failure to remove the targeted
padding invalidates the ablation rather than supporting a null mechanism.

The P3 P10/C00/rho=0.60 clean Layer-1 stream has SHA-256
`51ad4be12178da91d2af484d0946a2274afd3bcbbee33f37940cfe0ff2ea7fa7`.
Its 17,941 pure-decode steps have the exact decode-B histogram:

| B | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---:|---:|---:|---:|---:|---:|---:|---:|
| Steps | 13,572 | 2,675 | 792 | 524 | 161 | 202 | 15 |

Defaults already contain 1, 2, 4, and 8. A2 therefore adds exactly
`{3,5,6,7}` and freezes the complete server list as:

```text
1 2 3 4 5 6 7 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128
136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256
272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512
```

This covers 100% of P3's observed P10 pure-decode B support. The A2
manipulation requires at least 99% support coverage in the new clean runs and a
90% reduction in pure-decode padding tokens versus base. It targets decode
capture-bucket mismatch; it does not remove prefill-token padding, eager
overflow, graph launch overhead, or length raggedness itself. Capture startup
time and memory are reported as config cost, not treated as free.

### A3 — recorded arrival to uniform arrival

**Hypothesis.** Uniformizing the same rate and request order reduces burst-driven
decode-batch/queue variance, so `E_A3 > E_base`. The mechanism is falsified if
the clean 5-second decode-B CV and waiting-queue CV do not fall, or if the
Holm-corrected efficiency contrast is not positive.

A3 changes only `recorded-scaled` arrival slots to `i/lambda`. Content, order,
input/output lengths, aggregate tokens, server config, prefix caching, and
capture sizes are unchanged. It estimates the total service effect of arrival
shape, including its legitimate downstream changes to batching and queueing.
It does not isolate a scheduler instruction cost. Under the P3-exact fallback,
this arm is the base-equivalent bridge and its arrival share is N/A as described
above.

### A4 — natural prefix caching disabled

**Hypothesis.** Direction is deliberately two-sided. Natural P10 reuse may
increase `E_token` through KV reuse, in which case disabling it gives a negative
share; alternatively, low-value/fragmented cache structure may impose overhead
or alter batching, giving a positive share. Either direction is publishable.

A4 omits only `--enable-prefix-caching`. Prompts, natural repeated-prefix
structure, recorded arrival slots, request order, token totals, scheduler limits,
and capture sizes remain identical to base. Required manipulation checks are
zero local prefix cache hits/queries in the disabled arm and unchanged prompt
hashes. This estimates the contribution of exploiting C structure, not the
intrinsic content similarity of the prompts.

### Mechanisms without a clean ablation

MoE routing skew cannot be removed while preserving P10 content and model
semantics: changing tokens, router weights, top-k, or expert placement changes
more than routing skew. It receives no causal share.

If budget remains, one **analysis-only**, separately started P10 sample may add
`--enable-return-routed-experts`. It is excluded from every `E_token` clean
window and ledger numerator. Offline analysis reports per-layer expert-count
entropy, Gini coefficient, coefficient of variation, max/mean load, and their
association with same-step token-normalized duration. The arrays remain private.
This telemetry is sampled and perturbing, can consume a large scheduler-side
buffer, and provides correlation rather than causal attribution; it can only
help interpret the residual/interaction line. Phase 3's MoE layer-duration CV
was N/A, so it cannot substitute for these routed-expert counts.

## Exact commands and config deltas

The following interface is normative for the later implementation. Variables:

```bash
P5C='python scripts/opprof_phase5_client.py'
PRIVATE=/home/admin/cpfs/wjh/opprof-phase5-private/manifests
P3PRIVATE=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl
P3SOURCE=/home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/chat_w20260311_1000.jsonl
MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
RATE=0.4725
```

Materialize the five private manifests without printing prompt text:

```bash
$P5C transform --in "$P3PRIVATE" --take-first 142 \
  --timestamp-source "$P3SOURCE" --join-key source_index \
  --timestamp-field timestamp --arrival recorded-scaled --target-rate "$RATE" \
  --service-order original --out "$PRIVATE/P10-base.jsonl"
$P5C transform --in "$P3PRIVATE" --take-first 142 \
  --timestamp-source "$P3SOURCE" --join-key source_index \
  --timestamp-field timestamp --arrival recorded-scaled --target-rate "$RATE" \
  --service-order length-binned --reorder-block-size 32 \
  --analysis-cohort-size 16 \
  --length-bin-edges 512,1024,2048,4096,8192,16384,32768 \
  --max-added-delay-seconds 64 --out "$PRIVATE/P10-A1.jsonl"
$P5C transform --in "$P3PRIVATE" --take-first 142 \
  --timestamp-source "$P3SOURCE" --join-key source_index \
  --timestamp-field timestamp --arrival uniform --target-rate "$RATE" \
  --service-order original --out "$PRIVATE/P10-A3.jsonl"
ln -s P10-base.jsonl "$PRIVATE/P10-A2.jsonl"
ln -s P10-base.jsonl "$PRIVATE/P10-A4.jsonl"
```

The symlinks make unchanged request bytes explicit; the controller hashes the
resolved content and requires A2/A4 hashes to equal base. A1/A3 require equal
sorted request-ID sets and equal input/output token sums.

Common server command (`ARM` is `base`, `A1`, `A2`, `A3`, or `A4`):

```bash
taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU" \
  VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
  vllm serve "$MODEL" --host 127.0.0.1 --port "$PORT" \
  --tensor-parallel-size 1 --enable-chunked-prefill \
  --enable-prefix-caching --shutdown-timeout 600
```

- A2 adds
  `--cudagraph-capture-sizes 1 2 3 4 5 6 7 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512`.
- A4 removes `--enable-prefix-caching`.
- Base, A1, and A3 have no server delta.

Every moderate client command is:

```bash
taskset -c "$CPUSET" $P5C run \
  --manifest "$PRIVATE/P10-$ARM.jsonl" \
  --base-url "http://127.0.0.1:$PORT" --model "$MODEL" \
  --load-point moderate --fixed-request-rate "$RATE" \
  --max-concurrency 256 --ignore-eos --temperature 0 \
  --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 \
  --post-clean-seconds 0 --drain-timeout-seconds 600 \
  --workload-seed 20260712 --server-seed 20260712 \
  --result-dir "$RUN_DIR/client"
```

There are no profiler endpoint calls. Exact commands, effective config, startup
capture list, compile-cache key, hashes, clocks, host load, and GPU process list
are recorded per run.

## Execution and validity discipline

### Placement, order, and detached ownership

A-P3-1 already rejected eight-way and authorized four-way placement. Phase 5
therefore uses at most four simultaneous TP1 servers, GPU0--GPU3, with the
frozen disjoint CPU masks `0-19`, `20-39`, `40-59`, and `60-79`. Topology must
still match P3. A changed host topology, source/patch/runtime hash, or clock
policy invalidates reuse of the placement gate and stops for review.

Three independent replicates are run for each of the five primary arms (15
measured runs). Sort all `(replicate,arm)` assignments by SHA-256 of
`"20260715:<replicate>:<arm>"`, pack them into waves of 4, 4, 4, and 3, and
rotate GPU assignments. The final three-target wave uses the frozen P06/C00
saturation background in the fourth slot so measurements retain the validated
four-way host regime. No wave contains two replicates of the same arm; if the
hash order would do so, stable-swap the later item with the first legal item and
record the resolved order before launch. Background data are not analyzed.

As required by A-P3-2, a dash0-resident `setsid`/`nohup` controller owns every
process, has `--resume`, atomically records state, and skips only fully validated
runs with matching hashes. Interactive SSH never owns a server. Before each
wave it logs one echo line with resolved arms, GPUs/CPUs, manifests, rate,
paths, reserved H20-hours, expected duration, and disk headroom.

Use three unmeasured 60-second burn-ins before measured arms: common C00,
A2 capture-list C00, and prefix-cache-off C00. They warm compile/AOT artifacts
but are not measurements. Per-run-unique OpProf directories must remain ignored
by vLLM compile factors, preserving the accepted Phase-2 fix.

### Long-context-safe warm-up and drain gates

These gates are active from the first run; no short-context default is tried
first.

- Warm-up is exactly 60 seconds and excluded. P10 passes with at least 32
  successful warm-up completions, **or** at least 16 completions plus the exact
  A-P3-6 stabilization test: model-executed steps in `[45,50)`, `[50,55)`, and
  `[55,60)`; at least 16 steps/bin; positive scheduled-token rates `R_j`; and
  OLS drift `abs(slope)*15/mean(R_j) <= 0.10`. Missing bins, discontinuity, or
  accounting failure invalidates the run.
- Drain timeout is 600 seconds. A timeout marks the run drain-quarantined but
  does not invalidate an otherwise valid clean window. More than 20% of primary
  runs quarantined stops Phase 5 for review.
- The clean window is exactly three contiguous 80-second segments. Admission
  and completion accounting follows P3. Offered rate must be within 5% of
  0.4725 request/s, with zero clean failures and exact output tokens.
- Layer-1 JSONL/footer/sidecar accounting, zero drops, contiguous steps, no
  profile leakage, clock/load capture, other-user-process absence, and final
  zero GPU memory are hard gates inherited from P3.

No semantic failure is retried with altered parameters. One exact retry is
allowed for an infrastructure artifact failure, and both attempts remain in
the operational findings.

## Control reuse, secondary scope, and budget

P03/P04 P3 controls may be reused because they already use the same model,
patch, C00-TP1 config, normalized load, 240-second clean metric, placement
regime, and Layer-1 schema. Reuse avoids spending GPU time without changing the
denominator.

Reuse is valid only if source, patch, model, runtime, clocks, and placement
hashes match and fresh A3-uniformized P10 differs from frozen P3 P10 by at most
3% in `E_token` with a bootstrap CI containing zero difference. If this bridge
gate fails, temporal/runtime drift is plausible: rerun P03 and P04 under their
exact P3 manifests and commands, three replicates each, before computing any
share. A failed bridge never licenses rescaling old controls.

After the complete P10 ledger, optional work is ordered as follows and starts
only if the controller's conservative reservation remains below the 6.0
H20-hour hard cap:

1. one saturation run per P10 arm, using the full 4,011-row manifest, for
   descriptive mechanism persistence only;
2. one five-arm moderate ledger each for P09 and P06, reported as exploratory
   within-run-bootstrap evidence, not equal replication to P10; and
3. one routed-expert analysis-only sample.

For secondary A2, the precomputed P3 pure-decode support additions are P09
`{3,5,6,7,9,10,11,12,13,14,15,17,18,19,20,21,22,23,25}` and P06
`{3,9,10,11,12,13,14,15}`; default sizes are retained. P09 A3 is a no-change
steady negative control, while P06 A3 changes its registered burst-8 arrivals
to uniform spacing at the same mean rate. P09/P06 conclusions are always
labeled secondary.

Expected accounting, including server-owned startup/shutdown time:

| Tier | New measured runs | Expected H20-hours | Expected 4-way wall |
|---|---:|---:|---:|
| P10: 5 arms x 3 replicates | 15 | 1.6-1.9 | 35-55 min |
| Three compile burn-ins | 0 | 0.1-0.2 | 3-6 min |
| Conditional P03/P04 reruns | 6 | 0.6-0.8 | 15-25 min |
| Optional P10 saturation | 5 | 0.5-0.8 | 15-25 min |
| Optional P09/P06, 5 arms each | 10 | 0.9-1.2 | 20-35 min |
| Optional routed-expert sample | 1 analysis-only | 0.1-0.2 | 5-10 min |
| **Maximum planned** | **36 ledger + 1 analysis-only** | **3.8-5.1 expected** | **about 1.5-2.5 h** |

The hard cap is **6.0 H20-hours new Phase-5 spend**, not a target. Actual time
while a server owns GPU memory is charged. Optional tiers are skipped rather
than overrunning the cap. The 600-second drain allowance is a watchdog, not an
assumption in the expected estimate; repeated long drains consume the optional
budget first.

With no Kineto traces, primary P10 artifacts are estimated at 0.4-0.6 GB and
all planned public artifacts below 1.5 GB. Stop if public Phase-5 artifacts
exceed 3 GB or CPFS free space falls below 100 GB. Prompt-bearing manifests and
routed-expert arrays remain in mode-0700 private storage and are not counted as
public deliverables.

## Statistical analysis and decision rules

Use 5-second moving-block bootstrap over clean time, 100,000 resamples, seed
20260716. For P10, first resample the three run IDs, then resample 5-second
blocks within each selected run; arms and reused P3 controls are resampled
independently. Every `E`, `delta`, share, share sum, and residual/interaction
gets a percentile 95% CI. Absolute `E` and delta accompany every ratio.

Bootstrap ratio draws are never deleted because their denominator is
inconvenient. If either control gap has a point estimate `<=0`, its CI includes
zero, or more than 5% of bootstrap denominator draws are `<=0`, that control's
ledger is **INCONCLUSIVE (unstable denominator)**.

The confirmatory family is the four two-sided tests of `E_ablated-E_base=0` on
P10. Apply Holm correction at family-wise alpha 0.05 across A1--A4. The same
corrected delta test serves both control ledgers; duplicating denominators does
not create eight tests. A1--A3 only support the expected recovery claim when
their corrected contrast is positive. A4 may be significant in either
direction. Manipulation-check failure makes the corresponding share N/A even
if efficiency changes.

A mechanism is **dominant** only if, under both P03 and P04 ledgers:

- point `share >= 0.30`;
- the 95% share CI excludes 0.15 on the high side (`CI_low > 0.15`); and
- its Holm-corrected efficiency contrast is significant in the expected
  direction (two-sided for A4).

Meeting the rule under only one control is reported as **control-sensitive**,
not dominant.

The primary ledger is **publishable** when all five P10 arms have three valid
replicates, both control denominators are stable, all four manipulation checks
are evaluable, every share/residual has a finite CI, the fresh A3/P3 bridge is
reported, and no privacy/data-sanity red flag exists. It may be publishable
with no dominant mechanism. It is **inconclusive** if any primary arm is
missing, a denominator is unstable, the arrival-rebase decision is unresolved,
or two or more share CIs have width greater than 0.50. One failed mechanism
manipulation yields a publishable partial ledger only if that line is explicitly
N/A and the headline claim excludes it.

No additive causal claim is made from `sum(share_m)`. Pairwise and higher-order
interactions are not identified by this one-factor-at-a-time matrix; a combined
all-off arm would be a new experiment requiring amendment, not an improvised
way to close the ledger.

## Deliverable and artifact contract

Execution, if later approved, must produce:

- `docs/opprof/phase5-results.md`;
- `runs/opprof-phase5/phase5/metrics.json` with schema, arm/run values,
  bootstrap draws' seed and summary, Holm results, dual-control ledgers,
  residual/interaction, gates, and GPU accounting;
- per-run exact commands, environment/provenance, client records, Layer-1
  stream/footer/sidecar, monitor data, and machine-readable sanity JSON; and
- an **Operational findings** section covering warm-up stabilization, drains,
  compile/capture startup cost, contamination, retries, and any mismatch between
  expected and realized mechanism removal.

Every raw-run summary, aggregate table, and final metrics file ends with the P3
sanity schema: `n`, finite/missing, min, max, distinct count, and applicable
invariants. Red flags are reported first and stop inferential analysis. Public
artifacts may contain request IDs, hashes, lengths, counts, and timing only;
prompt, messages, content, generated text, source substrings, and routed-expert
arrays are forbidden.

## Final ablation table

| Mechanism | What changes | What is preserved |
|---|---|---|
| Base | Recorded P10 timestamps are rate-normalized and replayed in source order | Frozen 142 requests, content/tokens, C00-TP1, prefix cache on, default capture list, `lambda=0.4725` |
| A1: length raggedness | Stable length-bin sort within 32-request reorder blocks into 16-request cohorts, 64-second late cap | Requests/content/token totals, arrival-slot vector, output lengths, server config, prefix setting |
| A2: capture mismatch | Add exact P10 decode-B sizes `{3,5,6,7}` to the full default capture list | Manifest/order/arrivals/content/tokens, scheduler limits, prefix setting |
| A3: arrival dynamics | Recorded-scaled slots become uniform `i/0.4725` slots | Request order/content/tokens, mean rate, server/capture/prefix config |
| A4: prefix structure | Remove `--enable-prefix-caching` | Natural prompt structure, request order/arrivals/content/tokens, capture/scheduler config |
| Residual + interactions | No extra run; joint arithmetic remainder `1-sum(shares)` | Raw unnormalized mechanism shares; no clipping or forced 100% allocation |

## Final run count and GPU estimate

Primary commitment: **15 new measured P10 runs plus three unmeasured burn-ins,
1.7-2.1 expected H20-hours, 35-60 minutes four-way wall, and 0.4-0.6 GB public
disk**. Conditional control reruns add 6 runs; all optional tiers bring the
maximum to **36 ledger runs plus one analysis-only run, 3.8-5.1 expected
H20-hours, about 1.5-2.5 hours wall, and less than 1.5 GB public disk**. New
Phase-5 GPU use stops at **6.0 H20-hours** under all circumstances.

## Resolved decisions from the orchestrator

1. **Approved:** use the recommended recorded-arrival P5 bridge ledger, with
   its explicit limitation that it is not a literal decomposition of P3's
   already-uniform P10 gap; otherwise select the P3-exact A3=N/A fallback.
2. **Approved:** dual P03/P04 control ledgers and the requirement that “dominant” hold
   under both, rather than selecting one P3 denominator.
3. **Approved:** the 142-request slice, 32-request reorder blocks, 16-request analysis
   cohorts, fixed bins, and 64-second fairness cap as A1's isolation/latency
   tradeoff.
4. **Approved:** three P10 replicates, reuse of P3 controls behind the 3% bridge gate,
   and the optional-tier order within the 6.0-H20-hour cap.
5. **Approved:** Layer-1-only primary runs and treating routed-expert telemetry as
   private analysis-only evidence with no causal share.

## Protocol sanity block

| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
|---|---:|---:|---:|---:|---|
| P3 control/base `E_token` | 3 | 2.619113 | 4.735600 | 3 | Finite, positive, not identical |
| P3 P10 control gaps | 2 | 0.435590 | 2.116487 | 2 | Positive; dual denominators retained |
| P10 request rows/arm | 5 arms | 142 | 142 | 1 expected | Same IDs and input/output token sums required |
| Selected source timestamps (s) | 142 | 0.014 | 21.445 | 142 | Finite, nondecreasing; gap min/max 0.002/0.851 s, 127 distinct gaps |
| Primary offered rate (req/s) | 5 arms | 0.4725 | 0.4725 | 1 expected | Positive; achieved rate must be within 5% |
| Warm-up / clean / drain gates (s) | 3 values | 60 | 600 | 3 | Clean is exactly `3*80=240`; long-context drain is 600 |
| A1 input-length bins | 7 | 0 | 32768 | 7 intervals | Ordered, contiguous, cover frozen P10 range |
| A1 frozen `R16` values | 2 | 0.473409 | 0.641744 | 2 | Sorted is lower by 0.168334 absolute / 26.23% relative; not identical |
| A1 added-delay values (s) | 142 | 0 | 62.744 | >1 expected | Non-negative and all below 64-second cap |
| P3 P10 pure-decode steps | 17,941 | B=1 | B=7 | 7 B values | Counts sum to 17,941; non-negative; stream SHA pinned |
| P10 A2 added capture sizes | 4 | 3 | 7 | 4 | Exactly missing observed support `{3,5,6,7}`; 100% P3 support covered |
| Primary measured runs | 15 | 3/arm | 3/arm | 1 expected | `5 arms * 3 replicates`; no arm omitted |
| Maximum GPU analysis/ledger runs | 1 plan | 37 | 37 | 1 expected | `15+6+5+10+1=37`; burn-ins excluded |
| Expected H20-hours | 1 plan | 3.8 | 5.1 | 2 bounds | Finite, non-negative, below hard cap 6.0 |
| Share domain | 5 ledger lines/control | Unbounded | Unbounded | N/A | No clipping; ratios may be negative or >1 |
| Bootstrap resamples | 1 setting | 100,000 | 100,000 | 1 expected | Seed 20260716; no denominator-draw deletion |
| Phase-5 GPU runs in this protocol turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied |

Checked invariants: `0.60*0.7875=0.4725`; P3 control gaps are positive;
the seven decode-B counts sum to 17,941; default plus `{3,5,6,7}` covers all
observed P10 pure-decode B values; `5*3=15` primary runs; maximum planned count
is `15+6+5+10+1=37`; expected GPU use remains below the 6.0-hour hard cap;
ratios are not constrained to `[0,1]`; expected constants are labeled; and no
GPU command, helper change, manifest transform, or experiment was executed in
this protocol-only turn. The unresolved P3-arrival/P5-arrival estimand mismatch
is reported first as a blocking decision rather than hidden in the ledger.