# OpProf Phase 3 pre-registered pattern-matrix protocol

Status: **DRAFT FOR ORCHESTRATOR REVIEW — DO NOT EXECUTE**.

Date frozen: 2026-07-12 (Asia/Singapore). This document defines the Phase 3
experiment before any Phase 3 GPU launch. It does not authorize a run, change
the accepted five-patch series, or expose the prompt-bearing trace. Any change
to a pattern, load fraction, configuration, profiler placement, metric,
decision threshold, or exclusion rule requires a dated amendment reviewed
before the affected GPU run.

## Amendments (pre-execution; 2026-07-12)

All six open decisions in this document are approved as proposed. The two
amendments below are normative and take precedence wherever the original
serial-execution wording or estimate conflicts with them. Neither amendment
changes the pattern matrix, load points, profiler placement, metrics, or hard
validity gates.

### A-P3-1 — eight-way parallel execution and co-location gate

The matrix is executed on dash0 with one independent pattern/config cell per
physical GPU, up to eight cells concurrently on GPU0--GPU7. Each GPU has its
own server, client, port, run directory, and `CUDA_VISIBLE_DEVICES` namespace.
No TP1 server spans GPUs. The P10/TP2 counterpoint is scheduled as one cell
using two GPUs and leaves the corresponding second slot empty.

Eight-way execution is authorized only after this pre-registered co-location
validity check:

1. Use P05/C00 on physical GPU0 as the target. First calibrate its saturation
   throughput and run its moderate load (`0.60*T_sat`) with no other Phase 3
   GPU work on the host.
2. Recalibrate P05/C00 saturation and run the same moderate target on GPU0
   while P01, P02, P03, P04, P06, P07, and P08 under C00 run closed-loop
   saturation on GPU1--GPU7, respectively. The seven background loads remain
   active through the target's warm-up, clean interval, profile windows, and
   recovery intervals. Only the target is profiled.
3. Each saturation calibration uses the same placement regime as the
   moderate run it supplies: solo calibration for solo moderate and eight-way
   calibration with all seven backgrounds for eight-way moderate. The target
   manifest, fixed seed, server configuration, fresh-server timeline, 60-second
   warm-up, and 240-second clean interval are otherwise identical.
4. Let `T_solo` and `T_coloc` be target completed-request throughput over the
   clean interval. Let `F3` be the three operator families with greatest
   aggregate device-duration share in the solo target's accepted Layer-2
   validation window. This one-time placement check samples exactly one 2+8
   window per moderate target; it is not a matrix cell and does not change the
   two-window-per-load matrix rule. The check passes only if
   `abs(T_coloc/T_solo - 1) < 0.03` and, for every `f` in `F3`,
   `abs(share_coloc[f] - share_solo[f]) < 0.03`. The latter thresholds are
   absolute share fractions (three percentage points), not relative percent
   changes. Both target validation windows (one solo and one co-located) must
   satisfy the existing trace and
   classifiability gates before this comparison is made.
5. If eight-way fails, repeat the regime-specific saturation and moderate
   comparison at four-way placement: P05/C00 on GPU0 plus P01, P03, and P06
   C00 saturation backgrounds on GPU1--GPU3. Four-way is authorized only if
   the same throughput and `F3` share criteria pass. Failure at four-way stops
   execution for orchestrator review; it does not authorize a smaller regime.

The CPU map is fixed from dash0's observed topology on 2026-07-12. Server and
client descendants inherit the same `taskset` mask, and masks are disjoint:

| Physical GPU | NUMA node | Server/client logical CPUs |
|---:|---:|---|
| 0 | 0 | 0-19 |
| 1 | 0 | 20-39 |
| 2 | 0 | 40-59 |
| 3 | 0 | 60-79 |
| 4 | 1 | 80-99 |
| 5 | 1 | 100-119 |
| 6 | 1 | 120-139 |
| 7 | 1 | 140-159 |

`nvidia-smi topo -m` must still show GPU0--3 local to NUMA0 and GPU4--7 local
to NUMA1 before execution. Every run records host load average and a full GPU
clock snapshot before and after its measured interval, in addition to the
original process-contamination and environment records. Saturation calibration
runs use the same co-location regime as their measurement runs even when an
isolated calibration would be operationally simpler.

The aggregate H20-hour estimate remains approximately 7.5 hours because the
same cells and durations execute; the expected matrix wall time becomes
approximately 1.5--2 hours after burn-ins, dependent pairs, TP2 placement, and
confirmation runs are scheduled. The one-time co-location validation is
reported separately from matrix cost.

### A-P3-2 — detached, resumable execution

All GPU work is launched by a controller resident in the private dash0 work
directory under `setsid`/`nohup`; an interactive Codex or SSH lifetime is never
the owner of a measurement. The controller has a `--resume` mode and an atomic
per-run state file. A run reaches `complete` only after client, Layer-1,
Layer-2, sanity, command, clock/load, and zero-GPU-memory cleanup checks have
all succeeded. Resumption skips only such complete runs, rejects a changed
source/helper/manifest/config hash, and otherwise cleans up controller-owned
PIDs before restarting the incomplete run from its fresh-server boundary.
Completed output directories are idempotent and are never overwritten.

The controller records its PID/session, child process groups, phase,
timestamps, exit status, exact commands, and failure reason; state updates use
write-then-rename. It handles termination by stopping only its recorded child
process groups and preserving resumable state. Immutable copies and SHA-256
hashes of the client, controller, operator mapping, and launch configuration
live in each campaign's `runs/.../provenance/` directory. The orchestrator
polls the state and logs; it does not keep a foreground SSH process alive.

### A-P3-3 — per-class drain budget

The post-admission drain hard stop is class-specific. Output-512 burst cells
receive **240 seconds**; all other cells receive **120 seconds**. Under the
frozen pattern table, the 240-second class is exactly P04, P06, P07, P08, and
P11, including their configuration variants and confirmation runs. P02 has a
512-token output but steady arrivals, so it remains in the 120-second class.
The timer starts when final admission stops after the load point's last
profile/recovery interval. Exceeding the applicable budget invalidates the run
and stops the campaign; it does not authorize cancellation, truncation, or a
larger post-hoc allowance. This amendment supersedes the uniform 120-second
drain wording below but changes no clean interval or throughput denominator.

### A-P3-4 — cumulative Phase-3 GPU budget

The Phase-3 hard stop is **16.0 H20-hours cumulative**, including E-a
co-location validation, the shutdown-fix GPU verification, compile burn-ins,
the 52-run matrix, retries, and cleanup time while a server still owns GPU
memory. E-a consumed 3.964 H20-hours. Before every launch wave, the detached
controller records consumed cost and refuses to launch if the wave's
conservative reservation would exceed 16.0 H20-hours. This amendment
supersedes the 8.0 H20-hour hard stop below; the original 7.5 H20-hour matrix
estimate remains an estimate, not an additional allowance.

### A-P3-5 — crash-resilient Layer-1 accounting

The accepted OpProf series now ends at
`23450fb21ac255b0cf710f4ee965ee694921975d` and contains seven patches. Every
one-second JSONL flush atomically replaces `<stream>.footer.json` through a
same-directory temporary file. This checkpoint carries balanced encoded,
written, and dropped counters through the flush, the last written step index,
its wall-clock timestamp, the configured flush interval, and a `final` flag.

The controller's preferred shutdown remains the official vLLM 0.24.0 path:
start `vllm serve` with a positive `--shutdown-timeout` and signal the API
parent, which selects EngineCore `drain` mode. A clean run must contain an
in-stream footer, and the `final=true` sidecar must agree with its encoded,
written, and dropped counters. Graceful shutdown is not an accounting
dependency, however. If a deliberate or external hard kill leaves no
in-stream footer, the latest atomic sidecar is authoritative. All complete
JSONL data lines must decode; their count must equal sidecar
`written_records`; the final data-line step must equal sidecar
`last_step_index`; `encoded = written + dropped`; and `dropped=0`. The
controller records the signal timestamp and requires the checkpoint age to be
no more than the configured one-second flush interval, with at most 100 ms of
separately reported timestamp/scheduling tolerance. At most one flush interval
after the authoritative checkpoint may be lost. A missing/invalid sidecar,
counter mismatch, older checkpoint, partial JSON line, or nonzero drop count
invalidates the run.

The seventh patch also makes the preregistered two Layer-2 windows executable:
after `/stop_profile`, a stopped scheduled `TorchProfilerWrapper` is discarded
so the next `/start_profile` constructs a fresh 2+8 schedule. A CPU reproducer
on the pinned torch 2.11 showed that reusing the stopped object returned
without error but emitted no second trace. This repair changes neither the
profile schedule nor its placement.

### A-P3-5 — drain quarantine (orchestrator amendment; 2026-07-12)

The orchestrator assigned this amendment the same identifier as the preceding
crash-accounting amendment. This subsection is therefore distinguished by its
title; it supersedes only the drain-budget and drain-stop clauses of A-P3-3 and
leaves crash-resilient accounting unchanged.

Drain is post-measurement and serves as a hang watchdog. P10-class cells using
the long-context real trace receive **600 seconds**. P04/P06-class cells retain
the **240-second** allowance from A-P3-3; this class remains exactly P04, P06,
P07, P08, and P11. All other cells retain **120 seconds**.

Exceeding the applicable allowance marks a run `drain-quarantined` but no
longer stops the matrix. If its 240-second clean window, request/output
correctness, Layer-1 accounting, and Layer-2 artifacts are otherwise valid,
the run remains usable in analysis and is explicitly flagged in every table.
The matrix stops for drain behavior only when more than 20% of measured runs
are drain-quarantined. The existing greater-than-5% stop for clean-window
failures is unchanged.

The preserved P10/C01 saturation run from the first primary wave is
re-adjudicated against 600 seconds. Its observed 288.619107924-second drain is
within that allowance; no GPU rerun is authorized for the clean measurement.

### A-P3-6 — P10 warm-up stabilization gate (orchestrator amendment; 2026-07-12)

The warm-up gate measures whether service has reached steady state; completed
requests are only a proxy for that condition. For P10-class runs only, warm-up
passes if either at least 32 requests complete successfully during the
60-second warm-up, or at least 16 complete successfully and trailing Layer-1
telemetry meets the stabilization test below. All non-P10 warm-up rules remain
unchanged. This amendment changes neither the 240-second clean window nor any
throughput denominator.

The stabilization test is frozen before re-adjudicating retained telemetry:

1. Select `model_executed=true` Layer-1 records by `submit_mono_ns` in the
   trailing warm-up quartile `[t0+45 s,t0+60 s)` and split them into the fixed
   wall-time bins `[45,50)`, `[50,55)`, and `[55,60)` seconds.
2. Each bin must contain at least 16 model-executed steps. For bin `j`, define
   scheduled-token throughput
   `R_j=sum(prefill_tokens+decode_tokens)/5 s`. Non-finite or non-positive
   values, missing bins, discontinuous step indices, or a failed Layer-1
   accounting invariant fail the branch.
3. Fit ordinary least squares to `(47.5,R_1)`, `(52.5,R_2)`, and
   `(57.5,R_3)`. Let `s` be the fitted slope and `R_bar` the mean of the three
   rates. Stabilization passes only if
   `abs(s)*15/R_bar <= 0.10`, i.e. the fitted drift across the complete
   trailing quartile is no more than 10% of its mean.

Every P10 result admitted through the OR branch records the warm-up completion
count, three step counts, three token-throughput values, mean, fitted slope,
normalized drift, and pass/fail result. The final report must render this
post-hoc evidence explicitly; no missing value is imputed. Re-adjudication may
accept an already measured clean window only when this exact test passes.

### A-P3-7 — frozen partial-matrix inference (orchestrator amendment; 2026-07-12)

The Phase-3 matrix is final and frozen at 40/52 accepted measured runs and
20/24 complete pattern/config cells. No missing cell is rerun, replaced, or
imputed. The four incomplete cells, each missing both saturation and moderate
load points, are exactly **P03/C11**, **P05/C00**, **P10/C00-TP2**, and
**P11/C00**. The four Layer-1-only C00-moderate confirmations for P10, P06,
P03, and P01 are also absent; confirmation-run repeatability is therefore not
evaluable and cannot be substituted with within-run resampling.

H1a is evaluated over the completed primary C00 operator-share matrix. Its
formal moderate-load comparison contains exactly P01, P02, P03, P04, P06,
P07, P08, P09, and P10; P05 and P11 are missing. Saturation results for the
same completed patterns remain supporting evidence. A qualifying inversion
among completed patterns confirms H1a because H1a is existential. If none is
found, H1a is **INCONCLUSIVE**, not refuted: the original refutation rule
requires the complete eleven-pattern matrix and simultaneous bounds for every
registered comparison.

Each frozen H1b contrast evaluates if and only if both of its C00-moderate
cells are complete. Contrast status is fixed as follows:

| Frozen H1b contrast | A-P3-7 status |
|---|---|
| P05 vs P01 | NOT EVALUABLE — P05/C00 missing |
| P05 vs P03 | NOT EVALUABLE — P05/C00 missing |
| P06 vs P02 | EVALUABLE |
| P06 vs P04 | EVALUABLE |
| P09 vs P01 | EVALUABLE |
| P09 vs P03 | EVALUABLE |
| P10 vs P03 | EVALUABLE |
| P10 vs P04 | EVALUABLE |

The six evaluable contrasts retain their original waste thresholds, 5% useful
token-efficiency/step-residual association, correction families, seeds, and
decision logic. One qualifying evaluable contrast confirms H1b because H1b is
existential. If none qualifies, H1b is **INCONCLUSIVE**, never refuted, because
two frozen contrasts and four cells are missing. Thus 20/24 coverage permits
confirmation of H1a, H1b, or the compound hypothesis, but it cannot support
the original complete-matrix null/refutation claim. Missing results are
reported as `NOT EVALUABLE` and never assigned zero effect, copied controls,
or synthetic uncertainty.

## Goal and falsifiable hypothesis

The system boundary is Qwen3-30B-A3B BF16 served by patched vLLM 0.24.0 on
dash0 H20 GPUs. Phase 3 asks whether the device-level bottleneck observed under
one serving pattern generalizes to other operationally defined patterns.

The headline hypothesis has two jointly required parts:

- **H1a — composition:** at the same normalized offered load, at least one pair
  of serving patterns has a material inversion in its top GPU operator-family
  ranking. “Material” is defined below as a replicated rank inversion with at
  least a five-percentage-point share gap, not a visual change in a bar chart.
- **H1b — rectangular-benchmark miss:** at least one non-rectangular serving
  pattern exhibits material padding, CUDA-graph miss/bucket mismatch, sequence
  raggedness, or mixed-prefill/decode interference that a rectangular offline
  batch omits, and that waste is associated with at least a 5% loss in useful
  token efficiency or step-time residual relative to its rectangular control.

The **null outcome** is that the same operator family remains top-ranked across
all primary patterns at matched load, all simultaneous 95% bounds on ranking
gaps are below five percentage points, and every pre-declared waste contrast is
below its materiality threshold. If uncertainty or profiler opacity prevents
either conclusion, the result is **INCONCLUSIVE**, not evidence for the null.

The full round hypothesis is confirmed only if both H1a and H1b pass. Passing
only one is reported as a partial finding and does not justify the compound
claim.

## Pinned context and non-goals

| Item | Frozen value |
|---|---|
| vLLM base | v0.24.0, `ee0da84ab9e04ac7610e28580af62c365e898389` |
| Accepted local patch tip | `23450fb21ac255b0cf710f4ee965ee694921975d` |
| Patch series | Seven checksum-recorded patches in `patches/vllm-0.24.0-opprof/` |
| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, community BF16 |
| Hardware | dash0 H20; TP1 primary, one TP2 pattern/config counterpoint |
| Primary backend expectation | TRITON unquantized MoE; every run must record the observed backend |
| Layer-1 overhead evidence | -0.04196%, 95% CI [-0.17443%, 0.04550%] |
| Layer-2 perturbation evidence | 51.30% request-rate reduction during the active P2 collection window |
| Seeds | workload `20260712`; execution order `20260713`; bootstrap/permutation `20260714` |

This protocol does not depend on L-C-A labels, does not tune kernels, does not
change graph capture sizes, and does not claim production-cluster generality.
L-C-A may be computed later as a descriptive annotation but cannot define,
filter, or reorder the Phase 3 cells.

Pinned vLLM source facts used here:

- on an H20-class device (at least 70 GiB, not A100), the OpenAI server defaults
  to `max_num_seqs=1024` and `max_num_batched_tokens=8192`
  (`vllm/engine/arg_utils.py:2375-2456`);
- chunked prefill is supported and defaults on for this model
  (`vllm/engine/arg_utils.py:2458-2498`);
- the random dataset uses integer-uniform length ranges and generates one fixed
  prefix per manifest (`vllm/benchmarks/datasets/datasets.py:557-771`); and
- the bundled finite-rate generator uses Gamma inter-arrivals, but it has no
  fixed-duration mode (`vllm/benchmarks/serve.py:391-499`).

The last point requires a small fixed-duration client before execution. Its
normative interface and behavior are frozen below; the helper is not
implemented in this protocol-only turn.

## Operational pattern axes

### Input-length distribution

All lengths are post-tokenization token counts under the model tokenizer.

- **short-uniform:** discrete integer uniform `U[128,512]`.
- **long-uniform:** discrete integer uniform `U[4096,8192]`.
- **bimodal:** independent 50/50 draw from `U[128,512]` and
  `U[4096,8192]`; exactly half of a 32,768-request manifest comes from each
  mode, followed by a seed-fixed permutation.
- **production-shaped mixed:** exactly 50% `U[128,512]`, 30%
  `U[1024,2048]`, and 20% `U[4096,8192]`, followed by a seed-fixed
  permutation. This is synthetic and is not called a production trace.
- **real-trace anchor:** the approved private subset of
  `chat_w20260311_1000` described below. It preserves prompt text and observed
  input lengths, filters inputs above 32,768 tokens, and caps requested output
  at 256 tokens for bounded execution.

### Output length

- **short:** exactly 64 tokens.
- **long:** exactly 512 tokens.
- **trace-capped:** `min(observed_output_length, 256)`, used only by the real
  anchor.

Every request uses greedy temperature 0 and `ignore_eos`; output work is
therefore exact even if generated content is not bit-identical.

### Arrival shape

Arrival shape applies at the moderate load point. The saturation point uses
`request_rate=inf`, so all patterns intentionally collapse to a continuously
backlogged client; saturation is used to isolate service composition, not to
test inter-arrival shape.

- **steady:** deterministic spacing `1/lambda` seconds, first request at
  `t=0`.
- **bursty:** deterministic bursts of exactly eight simultaneous requests,
  first burst at `t=0`, with burst period `8/lambda` seconds and no within-burst
  delay or jitter.

Here `lambda` is the pattern/config-specific moderate rate defined as 60% of
that pair's measured saturation request throughput. These definitions avoid
conflating the arrival experiment with random arrival noise.

P02 and P11 are the matched steady/bursty arrival pair: both use short-uniform
input, 512-token output, and no shared prefix.

### Prefix sharing

- **none:** no intentionally shared prefix. Synthetic token streams are unique
  per request. The manifest must have no common prefix longer than 16 tokens,
  and the clean Layer-1 prefix-query hit ratio must remain below 1%; otherwise
  the cell is invalid.
- **high:** a pool of eight independently generated 1,024-token prefixes. Each
  request appends a unique 256-token suffix, so 80% of each 1,280-token prompt
  is shared within one-eighth of requests. Requests are seed-shuffled after
  balanced assignment to prefixes.
- **natural:** no prefix is added or removed from the private real prompts. The
  observed prefix-hit ratio is reported rather than constrained.

P07 and P08 are a matched no-prefix/high-prefix pair: both have total input
length 1,280, output length 512, and bursty arrival.

## Required fixed-duration client

Before any GPU use, an execution turn must implement and test
`scripts/opprof_phase3_client.py`. The orchestrator must review its diff and
freeze its SHA-256. The helper may reuse vLLM's tokenizer and endpoint request
function, but must not modify vLLM.

The client contract is:

1. `materialize` creates a token-exact JSONL manifest using the pinned model
   tokenizer and `numpy.random.default_rng(20260712)`. Synthetic manifests
   contain 32,768 requests and no manifest may wrap during a run.
2. `run` supports `--request-rate inf` as a closed-loop saturation stream with
   at most 256 outstanding requests. A finite rate uses the exact steady or
   eight-request periodic-burst schedule above.
3. Admission continues during profiler and recovery intervals. The client
   stops admitting after the final clean segment and drains outstanding
   requests for at most 120 seconds; it never cancels and counts a partial
   response as a failure.
4. The client records request ID, scheduled/admitted/first-token/completion
   monotonic timestamps, prompt tokens, requested and actual output tokens,
   HTTP status, and no prompt or generated text. It emits one result file per
   segment and one manifest SHA-256.
5. `/start_profile` and `/stop_profile` calls are synchronous and timestamped.
   Both profile windows occur after the uninterrupted clean interval. Profile
   and recovery intervals are tagged and excluded from throughput/latency
   summaries.
6. A dry, CPU-only test must prove fixed segment duration, saturation
   concurrency, exact burst timing, no manifest wrap, exact output-token
   requests, and prompt-field redaction.

The exact saturation command interface is:

```bash
python scripts/opprof_phase3_client.py run \
  --manifest "$MANIFEST" --base-url http://127.0.0.1:8000 \
  --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \
  --load-point saturation --request-rate inf \
  --max-concurrency 256 --ignore-eos --temperature 0 \
  --warmup-seconds 60 --clean-segment-seconds 80 \
  --num-clean-segments 3 --profile-after-clean --num-profile-windows 2 \
  --profile-warmup-iterations 2 --profile-active-iterations 8 \
  --recovery-seconds 30 --drain-timeout-seconds 120 \
  --workload-seed 20260712 --result-dir "$PAIR_DIR/saturation/client"
```

The exact moderate command is:

```bash
python scripts/opprof_phase3_client.py run \
  --manifest "$MANIFEST" --base-url http://127.0.0.1:8000 \
  --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \
  --load-point moderate \
  --saturation-result "$PAIR_DIR/saturation/client/result.json" \
  --rate-fraction 0.60 \
  --max-concurrency 256 --ignore-eos --temperature 0 \
  --warmup-seconds 60 --clean-segment-seconds 80 \
  --num-clean-segments 3 --profile-after-clean --num-profile-windows 2 \
  --profile-warmup-iterations 2 --profile-active-iterations 8 \
  --recovery-seconds 30 --drain-timeout-seconds 120 \
  --workload-seed 20260712 --result-dir "$PAIR_DIR/moderate/client"
```

The client derives `lambda=0.60*T_sat` from the accepted saturation result and
refuses a manually supplied finite rate. Arrival class comes from the immutable
manifest. Total throughput-accounted duration is exactly `3*80=240` seconds
per measured run.

## Private real-trace anchor

The local prompt-bearing source is:

```text
/home/gahow/phd/aituner/trace_windows/traces/chat_w20260311_1000.jsonl
```

It is 1,183,031,556 bytes with SHA-256
`f539f38eb0ee0f750e3c23ff47df6eed3faf723a25f1444d55665a85871750b9`.
Metadata records 32,606 requests over 600 seconds. The frozen selection
`sampling_u <= 0.125 && input_length <= 32768` yields 4,011 requests locally;
after the 256-token output cap, input min/median/p95/max is
69/7,009/25,629/32,704 and output min/median/p95/max is 2/256/256/256.
The anchor preserves prompt text, prompt/output-length correlation, and source
order; it deliberately replaces the original 600-second timestamps with the
same controlled saturation/moderate load protocol as every other cell. It is a
real-content anchor, not a native-traffic replay.

If and only if the orchestrator approves this anchor, execution may copy the
source to
`/home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/` on dash0. The
directory must be mode 0700 and files mode 0600. The source and derived
manifest must never enter a run directory, tar archive, Git, client result, or
Kineto/OpProf artifact. Transfer is accepted only after the local and remote
SHA-256 match.

The private materialization command is specified in the final pattern table.
It retains only `prompt`, capped `output_tokens`, and an expected input-token
count. A tokenizer parity precheck must find exact input length for at least
99% of rows and absolute error at most one token for every row; failure stops
the anchor rather than rewriting prompts.

## Server configuration axes

Chunked prefill remains explicitly enabled in every configuration. Disabling
it on this model makes vLLM raise MBT to the model length when MBT is omitted,
or reject MBT below model length when supplied
(`vllm/config/scheduler.py:272-284`). That would conflate chunking with a much
larger token budget. Phase 3 therefore uses an MBT factor instead.

| Config | TP | MNS | MBT | Chunked prefill | Purpose |
|---|---:|---:|---:|---|---|
| C00 | 1 | default = 1024 | default = 8192 | on | Primary vLLM/H20 baseline |
| C10 | 1 | small = 64 | default = 8192 | on | Bound sequence concurrency / decode batch |
| C01 | 1 | default = 1024 | small = 2048 | on | Force smaller prefill chunks and token batches |
| C11 | 1 | small = 64 | small = 2048 | on | Test MNS×MBT interaction |
| C00-TP2 | 2 | default = 1024 | default = 8192 | on | One communication/parallelism counterpoint |

C00 runs for all eleven patterns. The complete 2×2 TP1 factorial C00/C10/C01/C11
runs only for sentinel patterns P01, P03, P06, and P10. C00-TP2 runs only for
P10. This sparse design identifies each scheduler factor on short-decode,
long-prefill, irregular mixed, and real-trace regimes without spending a full
factorial on every pattern.

The common server command is:

```bash
CUDA_VISIBLE_DEVICES="$GPU_IDS" \
VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \
  --tensor-parallel-size "$TP" --enable-chunked-prefill \
  --enable-prefix-caching \
  --profiler-config "{\"profiler\":\"torch\",\"torch_profiler_dir\":\"$TRACE_TMP\",\"torch_profiler_with_stack\":true,\"torch_profiler_record_shapes\":true,\"torch_profiler_use_gzip\":true,\"ignore_frontend\":true,\"wait_iterations\":0,\"warmup_iterations\":2,\"active_iterations\":8}"
```

C00/C00-TP2 omit MNS and MBT flags. C10 adds `--max-num-seqs 64`; C01 adds
`--max-num-batched-tokens 2048`; C11 adds both. Startup logs must show the
frozen effective values. Defaults changing from 1024/8192 is a stop condition,
not permission to edit the protocol.

## Load and execution protocol

### Unit of analysis and load points

A **pattern/config cell** is one pattern row under one server configuration.
An **execution run** is one load point for a pattern/config cell. Each primary
cell has exactly two fresh-server runs:

1. **saturation:** closed-loop `request_rate=inf`, maximum 256 outstanding;
2. **moderate:** the same manifest at `lambda=0.60*T_sat`, where `T_sat` is
   clean-segment completed-request throughput from that cell's accepted
   saturation run.

Thus patterns are matched at normalized load `rho=0.60`, not at equal absolute
requests/s or tokens/s. Saturation is analyzed separately. Moderate admission
is valid only if achieved offered rate is within 5% of `lambda` and no sustained
client lag exceeds one second.

### Warm-up, clean measurement, and profiler placement

Every measured run uses a fresh server and this timeline:

```text
60 s warm-up (excluded; require >=32 completions)
80 s clean segment A
80 s clean segment B
80 s clean segment C
Layer-2 window 1: 2 warm-up + 8 active engine iterations (excluded)
30 s recovery (excluded)
Layer-2 window 2: 2 warm-up + 8 active engine iterations (excluded)
30 s recovery / trace flush (excluded)
stop admission; drain <=120 s (excluded)
```

The clean A/B/C segments are contiguous. The first profile window occurs only
after exactly 240 accumulated clean seconds; the second follows the first
fixed 30-second recovery, never an observed favorable batch. There are exactly
**four Layer-2 windows per pattern/config cell**: two in saturation and two in
moderate. The 240 clean seconds, not server lifetime or profiler interval, form
the throughput/latency denominator. This ordering prevents the 51.30%
active-window perturbation measured in P2 from contaminating throughput.

Offered rate runs continuously from warm-up through the clean interval.
Completed throughput counts completions timestamped inside the 240-second clean
interval; latency summarizes those same completions, including a request
admitted during warm-up if it completes cleanly after the boundary. Admission
rate is counted by admission timestamp. This is a stationary-window rule and
is applied identically to every pattern; no profiler interval precedes clean
measurement.

After each profile window, the 10-second rolling completion rate and
waiting-queue depth must return within 10% of clean-segment-C medians by the end
of the fixed 30-second recovery. This gate protects the second window and
validates post-profile recovery; no clean throughput is measured afterward.
Failure invalidates the run, and recovery is not silently extended.

### Compile warm-up and execution order

Before measured runs, launch one unmeasured burn-in server for each of the five
server configurations C00, C10, C01, C11, and C00-TP2. Each burn-in performs
60 seconds of P06 saturation and exits gracefully. It warms compile/AOT and
Triton artifacts but is never pooled with a pattern result.

After all burn-ins, sort pattern/config IDs by
`SHA256("20260713:" + pattern_id + ":" + config_id)` ascending. Run each
saturation/moderate pair contiguously because moderate depends on saturation.
Four Layer-1-only confirmation runs repeat moderate C00 for P10, P06, P03, and
P01, in that fixed reverse order, after the matrix. They use 60-second warm-up
and 240 clean seconds but no profiler endpoint; they quantify fresh-server
repeatability without changing the pre-declared four Layer-2 windows per cell.

Every GPU launch must first record `nvidia-smi`, selected UUID(s), clock state,
host load, other-user processes, exact command, expected duration, source and
manifest hashes, compile-cache key, and effective vLLM config. One H20 is used
except for the two C00-TP2 runs and its TP2 burn-in. No other user's process may
be touched. The preferred controller path starts the server with a positive
`--shutdown-timeout` and signals the API parent so EngineCore drains before the
process manager exits. Accounting is validated by the A-P3-5 footer/sidecar
rule even when graceful teardown does not occur, and GPU memory must finally
return to zero.

## Measurement plan

### Layer 1: always on

Every measured and confirmation run sets `VLLM_OPPROF_DIR`. Layer 1 is the
source of clean-segment composition and waste counters:

- scheduled requests, prefill/decode batch and token counts;
- chunked-prefill class and chunk-size histogram;
- context-length histogram;
- waiting/running/deferred queues, preemptions, KV usage, and prefix counters;
- step submit/complete timestamps; and
- CUDA-graph hit, runtime mode, unpadded tokens, bucket tokens, and padding.

Each run must produce exactly one schema-v1 JSONL for TP1 or one scheduler-owned
stream for the TP2 engine. Every line must msgspec-decode; step indices must be
unique and contiguous. On clean close, the in-stream footer must satisfy
`encoded = written + dropped`, `dropped=0`, and its step count must equal the
record count, while the final sidecar agrees. Without an in-stream footer, the
atomic sidecar and on-disk records must satisfy the A-P3-5 hard-kill rule.
Profile/recovery records remain in the raw stream but are tagged by monotonic
time and excluded from clean summaries.

Composition is summarized separately for clean A/B/C, each profiler window,
each recovery interval, and the complete clean union. No profile-window record
may leak into throughput, latency, or unperturbed composition statistics.

### Layer 2: sampled Kineto windows

Each profile endpoint invocation uses the server's frozen wait=0, warm-up=2,
active=8 schedule. A logical window is accepted only if:

- `/start_profile` and `/stop_profile` return HTTP 200;
- the trace gzip decompresses and parses as JSON;
- active `ProfilerStep` numbers are exactly 2 through 9;
- exactly eight matching execute annotations are present per TP rank;
- kernel event count is positive; and
- every active execute annotation joins unambiguously to its Layer-1 step.

TP1 yields one worker trace per logical window. TP2 is expected to yield two
rank traces; both are required and analyzed per rank. Rank times are never
summed and called latency.

The client writes a monotonic timestamp immediately before/after each endpoint
call. The analysis joins execute annotations to Layer-1 records by timestamp,
order, and the annotation's context/generation token counts. A join is invalid
if more than one Layer-1 record is compatible or if the annotation and Layer-1
token totals disagree.

Profiler representativeness is checked against the adjacent clean segment for
scheduled tokens, prefill fraction, decode batch size, and graph-mode shares.
For each feature, use standardized mean difference
`SMD=(mean_profile-mean_clean)/pooled_sd`; a constant feature uses exact
equality. If `abs(SMD)>0.25` for any feature in either window, that load point's
operator breakdown is invalid. One complete run may be retried with the same
command; a second failure is reported as Layer-2-inconclusive, not resampled
until it looks representative.

### Kernel-to-operator-family mapping

The primary operator share is summed GPU kernel duration in a family divided
by total GPU kernel/replay activity duration in the eight active steps,
including unmatched/opaque activity in the denominator. It is computed per
window and rank. CPU launch time, CUDA runtime time, memcpy/memset time, and
unioned device-busy time are reported separately. Because streams may overlap,
kernel-duration shares are composition shares, not fractions of wall time.

Mapping is priority-ordered; first match wins. The execution helper must emit a
versioned regex table and SHA-256 before GPU work. The rules below are the
normative v1 mapping:

| Priority | Family | Name/scope rule |
|---:|---|---|
| 1 | `moe_router` | `topkGating`, `moe_align`, `select_expert`, or MoE-scoped routing/top-k kernels; this precedes sampler matching |
| 2 | `collective` | `nccl`, `all_reduce`, `allreduce`, `reduce_scatter`, `all_gather`, or vLLM custom-all-reduce names |
| 3 | `attention` | `flash`, `fmha`, `paged_attention`, `unified_attention`, attention-scoped CUTLASS kernels, or attention KV/cache reshape kernels |
| 4 | `moe_gemm` | `fused_moe`, `moe.*gemm`, `nvjet`, or CUTLASS/Triton GEMM whose parent scope is FusedMoE/MoE |
| 5 | `sampler` | non-MoE `sampling`, `topk`, `topp`, `argmax`, `multinomial`, logits-processor, or penalty kernels |
| 6 | `dense_gemm` | GEMM/matmul/CUTLASS kernels outside attention and MoE scopes |
| 7 | `norm_elementwise` | RMS/layer norm, activation, residual/add, reduction, fill, and vectorized elementwise kernels |
| 8 | `kv_memory` | cache copy/swap kernels and KV movement not already attributed to attention; CUDA memcpy/memset activities remain separately reported |
| 9 | `other` | every unmatched kernel; no post-hoc cell-specific reassignment |

Attention is subdivided by the joined Layer-1 step:

- `attention_prefill`: prefill tokens >0 and decode tokens =0;
- `attention_decode`: decode tokens >0 and prefill tokens =0; and
- `attention_mixed`: both are nonzero.

Mixed attention is not force-split in proportion to tokens. The headline
bottleneck ranking uses `attention_total`; the three subfamilies explain which
serving phase supplied it.

Before inference, report the top 20 unmatched kernel names by duration. If
`other` plus opaque graph replay exceeds 30% of GPU kernel time in any primary
window, that window is not classifiable. No mapping amendment may be derived
from one favored cell: a reviewed global amendment must be applied to every
trace, with old and new results both retained.

### CUDA-graph mode segmentation

Every active step is segmented by its Layer-1 `runtime_mode`:

- `FULL`;
- `PIECEWISE`;
- `NONE`, reported as eager/ungraphed; or
- `ambiguous`, which invalidates the step.

Operator shares and waste are reported both overall and by mode when a mode has
at least eight active steps across the two windows of a load point. If CUPTI
shows only a graph-launch event with no child kernel activity for FULL or
PIECEWISE, its device time is labeled `graph_replay_opaque`; it is never
distributed among operator families. A cell cannot support H1a unless at least
70% of its device kernel time is classifiable.

## Metrics and pre-declared analysis

### End-to-end and composition metrics

For each clean segment and their 240-second union, report:

- completed and failed requests; achieved offered and completed requests/s;
- input, output, and total useful tokens/s;
- TTFT, TPOT, and end-to-end latency: mean, p50, p95, and p99;
- scheduled tokens/step, requests/step, prefill/decode tokens/step, decode batch,
  queue depths, KV usage, preemptions, and prefix hit/query ratios;
- FULL/PIECEWISE/NONE shares and step duration by mode; and
- absolute values plus ratios to C00, never ratios alone.

Primary useful-token efficiency is
`E_token = sum(clean prefill_tokens + clean decode_tokens) /
sum(clean step_duration_ms)` in tokens/ms. End-to-end input/output tokens/s is
reported beside it. H1b's “5% worse efficiency” means
`1-E_irregular/E_control >= 0.05` using `E_token`.

The primary cross-pattern comparison is C00-TP1 at moderate `rho=0.60`.
C00 saturation is secondary. C10/C01/C11 estimate scheduler-factor and
interaction sensitivity on sentinels. C00-TP2 is descriptive parallelism
evidence; a single TP2 pattern does not support a scaling claim.

### Operator shares and bottleneck ranking

For operator family `f` in window `w` and rank `r`:

```text
share[f,w,r] = sum GPU kernel duration assigned to f
               / sum duration of all GPU kernel/replay activity
```

Each load point has two independent wall-separated windows. Rank families by
share within each window; ties within one percentage point receive the same
rank. TP2 reports each rank and their mean/range, not their sum.

The cross-pattern ranking-change test considers every pair of primary C00
patterns and every pair of operator families. An inversion is accepted only
when:

1. family A exceeds B by at least five percentage points in **both** windows of
   pattern p;
2. B exceeds A by at least five percentage points in **both** windows of
   pattern q;
3. a window-stratified permutation test over the 16 active steps per pattern,
   seed 20260714 and 100,000 permutations, rejects equal ordering after Holm
   correction across all tested pattern/family pairs at family-wise alpha 0.05;
4. at least 70% of kernel time is classifiable in all four windows.

Reproduction at saturation or in another server configuration is reported as
strengthening evidence but is not required: saturation deliberately removes
the arrival treatment, and only four patterns receive config variants. The two
fixed, wall-separated windows are the pre-declared replication unit for H1a.

Also report Kendall's tau-b between every pair of family-rank vectors. Tau is
descriptive and cannot replace the inversion rule.

### Waste accounting

All ratios are computed as ratios of sums, not means of per-step percentages.

**CUDA-graph padding.** For graph-hit steps with positive bucket `b_s` and
unpadded tokens `u_s`:

```text
padding_fraction = sum_s (b_s - u_s) / sum_s b_s
```

Also report padded tokens per useful scheduled token and the distribution of
`b_s-u_s`. NONE steps are excluded from padding and counted under graph miss.

**Graph miss and bucket mismatch.** Let `B` be the capture bucket set parsed
from the startup config and `b*(u)=min{b in B: b>=u}`.

```text
graph_miss_rate    = count(model_executed and not hit) / count(model_executed)
eligible_miss_rate = count(model_executed and not hit and b*(u) exists)
                     / count(model_executed)
overflow_rate      = count(model_executed and not hit and b*(u) absent)
                     / count(model_executed)
bucket_slack       = sum(b*(u)-u) / sum(b*(u)) over eligible u
```

Report all four by pattern, config, load, and runtime mode.

**Sequence raggedness.** Partition the immutable arrival-order manifest into
consecutive complete cohorts of 64 requests. For input lengths `L_gi`:

```text
R64 = 1 - sum_g sum_i L_gi / sum_g (64 * max_i L_gi)
```

This is the exact padding fraction an offline rectangular batch-of-64 input
benchmark would incur on the same requests. The incomplete final cohort is
excluded. Report sensitivity at cohort sizes 32 and 128, but R64 is primary.

**Mixed-batch interference.** On clean Layer-1 steps, fit nonnegative robust
splines `F_p(P,N)` on pure-prefill steps and `F_d(D,N)` on pure-decode steps,
where `P`/`D` are prefill/decode tokens and `N` is scheduled requests. Let
`alpha` be median zero-token scheduler-step time. For a mixed step:

```text
t_add = F_p(P,N) + F_d(D,N) - alpha
I_mix = sum_mixed (t_observed - t_add) / sum_mixed t_add
```

Fits are config/load-specific and use all patterns, with leave-one-pattern-out
prediction for the reported cell. A mixed step outside the convex support of
both pure-step fits is excluded and counted. If fewer than 30 supported mixed
steps remain, `I_mix` is N/A rather than extrapolated. This is an interference
association, not a causal kernel ablation.

**MoE/ragged proxy.** Report coefficient of variation of MoE GEMM kernel
durations across layers within each active step:
`CV_moe = sd(duration_l)/mean(duration_l)`, only when layer scopes are
available for at least 80% of MoE GEMM time. Otherwise it is N/A. Exact expert
routes are not enabled because the server-wide routed-expert return path would
contaminate clean throughput.

### Statistical summaries

- Layer-1 and end-to-end intervals use a 5-second moving-block bootstrap over
  clean time blocks, 100,000 resamples, seed 20260714.
- Layer-2 ranking uses the window-stratified permutation rule above. Active
  steps are not presented as independent server replicates.
- The four Layer-1-only sentinel confirmations report run-level absolute and
  relative deltas; no outlier is removed.
- Multiple ranking tests use Holm correction. Waste contrasts use Holm
  correction within each waste family.
- No arithmetic mean of normalized scores is used. Cross-cell normalized
  aggregates, if requested later, use a geometric mean and retain all cells.

### Decision rule

H1a passes if at least one inversion meets all four ranking criteria. H1b
passes if an irregular cell (P05, P06, P09, or P10) versus its declared
rectangular control has a corrected 95% contrast excluding zero and exceeds at
least one threshold:

- padding fraction: 5 percentage points;
- graph-miss or overflow rate: 10 percentage points;
- R64: 0.15 absolute;
- `I_mix`: 0.10; or
- `CV_moe`: 0.15;

and that contrast coincides with at least 5% worse useful-token efficiency or
positive step-time residual. The frozen control contrasts are P05 versus both
P01 and P03, P06 versus both P02 and P04, P09 versus both P01 and P03, and P10
versus both P03 and P04. Every listed contrast enters the same correction
family and is reported; none is selected afterward by the best p-value.

The compound hypothesis is **CONFIRMED** only when H1a and H1b pass. It is
**REFUTED / NULL SUPPORTED** only when the same top family holds in both
windows of every primary pattern, simultaneous 95% upper bounds on all family
share gaps are below five percentage points, and all H1b contrasts are below
threshold. Every other outcome is **INCONCLUSIVE** or **PARTIAL**.

## Artifact, privacy, and reproducibility contract

Each run directory contains only:

```text
manifest.sha256                 # hash and non-sensitive length summary
commands.log                    # exact server/client/profile commands
environment.json                # source, patch, wheel, driver, GPU, clocks
server.log
client/{segments.jsonl,result.json,sanity.json}
opprof/*.jsonl
traces/*.pt.trace.json.gz
analysis/{layer1.json,kernels.json,waste.json,sanity.json}
```

Synthetic manifests may be archived. P10's manifest, prompts, generated text,
and any response body are private inputs and are forbidden from this tree.
Before packaging, a scanner must reject any artifact containing a `prompt`,
`messages`, `content`, or generated-text field, or any source prompt substring.
The private path and manifest hash may appear in the exact command log; file
contents may not. Only counts, length distributions, hashes, and timings may
leave the private directory.

Record source/patched commit, patch checksums, Python/torch/CUDA/vLLM/msgspec
versions, driver, model path and shard hashes, GPU UUID(s), server effective
config, MoE backend, compile cache key, client/helper SHA, workload hash, clock
snapshots, host load, other-user process samples, and all raw analysis seeds.

## Budget and hard stop conditions

The matrix contains 24 pattern/config cells: 23 TP1 and one TP2. Each has two
load runs, for 48 primary measured runs. Four Layer-1-only confirmation runs
bring the measured total to **52**. Five unmeasured compile burn-ins are not
counted as pattern runs but are included in cost.

Using the P2 fixed-cache startup and profile endpoint timings, budget each
primary run as approximately:

```text
70 s startup + 60 s warm-up + 240 s clean
+ 2 * (24 s profiler + 30 s recovery) + 10 s shutdown = 488 s
```

Expected serialized wall time is about 7.1 hours. Accounting for two GPUs in
the two TP2 runs and its burn-in gives **about 7.5 H20-hours**, with a hard
campaign stop at 8.0 H20-hours. The four confirmation runs are budgeted at
380 seconds each. Queueing for a free GPU is not GPU time.

There are 96 logical Layer-2 windows. TP1 contributes 92 trace files; TP2 may
contribute eight rank files, for 100 expected trace files. At the P2 trace size
of 16.4 MB, Kineto uses about 1.6 GB. Layer 1, logs, client data, and analysis
should keep public artifacts below 4 GB. The private source plus derived P10
manifest is separately budgeted below 2 GB. Stop if public artifacts exceed
8 GB or CPFS free space falls below 100 GB.

Stop the matrix immediately and request review if any of these occurs:

1. source/patch/helper/manifest hash mismatch, effective config mismatch, or a
   per-run output directory changes the compile-cache key;
2. selected GPU not idle, another user's GPU process appears, GPU OOM, server
   error, or final GPU cleanup does not return to zero;
3. any clean-segment request fails, an output-token count differs from the
   manifest, the moderate offered rate misses its target by more than 5%, or
   drain exceeds 120 seconds;
4. Layer-1 schema/footer/step accounting fails, any record drops, or any
   profile/recovery step enters a clean summary;
5. either trace is missing/unloadable, has other than 2+8 scheduled iterations,
   lacks kernels, cannot join to Layer 1, or profiler composition has
   `abs(SMD)>0.25` twice;
6. recovery does not meet its fixed 30-second criterion;
7. more than 30% of device kernel time is `other`/opaque in a primary window,
   or fewer than eight active steps are available for a required mode segment;
8. a prompt, message, content, generated-text field, or source prompt substring
   appears outside the private directory;
9. public disk or the 8.0 H20-hour cap is reached; or
10. a data-sanity red flag defined below appears.

No failed cell is silently replaced, retuned, filtered, or omitted. One exact
rerun is allowed only for an infrastructure/profile-artifact failure; both
attempts remain in the report. A semantic workload/config failure requires a
reviewed protocol amendment.

## Data sanity block contract

Every raw-run summary, per-cell analysis, aggregate table, figure source, and
final report must end with a machine-readable and rendered sanity block. For
**every numeric family** it reports `n`, finite count, missing count, minimum,
maximum, and distinct-value count. It also records expected constants rather
than treating them as suspicious.

At minimum, the block checks:

- non-negative request/token/queue/KV/kernel/duration counters;
- ratios in [0,1], except signed residual/interference metrics explicitly
  identified as signed;
- exactly 240 clean seconds, three 80-second segments, two profiler windows per
  load, 2+8 profiler iterations per window, and four logical windows per cell;
- continuous unique Layer-1 steps and balanced footer accounting with zero
  drops;
- exact manifest hash, no wrap, exact requested/actual output work, zero clean
  failures, and moderate rate within 5%;
- bucket >= unpadded, padding identity, graph-mode consistency, and waste
  numerator <= denominator;
- trace loadability, positive kernel count, rank completeness, unambiguous
  Layer-1 join, and at least 70% classifiable device time;
- input/output and arrival distributions match the frozen manifest;
- per-pattern and per-config outputs are not all identical unless the field is
  a declared constant;
- clocks/load/GPU-process contamination and graceful zero-memory cleanup; and
- no private prompt/generated text in public artifacts; private paths are
  allowed only in exact command logs.

Negative non-negative counters, ratios outside bounds, missing footers,
unexpectedly identical pattern/config results, discontinuous step/time series,
manifest drift, profiler leakage, or private-data leakage are red flags. Report
the anomaly first and stop analysis; do not build a conclusion on it.

## Final preregistration summary

The table below is normative. `P3C='python scripts/opprof_phase3_client.py'`,
`PRIVATE=/home/admin/cpfs/wjh/opprof-phase3-private/manifests`, and every
synthetic command uses the pinned model tokenizer. Each command is one logical
line even where Markdown wraps it.

### Pattern-cell table

| ID | Operational cell | Exact manifest-generator command |
|---|---|---|
| P01 | short-uniform input; short output; steady; no prefix | `$P3C materialize --id P01 --kind synthetic --input-uniform 128:512 --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P01.jsonl"` |
| P02 | short-uniform input; long output; steady; no prefix | `$P3C materialize --id P02 --kind synthetic --input-uniform 128:512 --output-fixed 512 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P02.jsonl"` |
| P03 | long-uniform input; short output; steady; no prefix | `$P3C materialize --id P03 --kind synthetic --input-uniform 4096:8192 --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P03.jsonl"` |
| P04 | long-uniform input; long output; bursty 8-at-once; no prefix | `$P3C materialize --id P04 --kind synthetic --input-uniform 4096:8192 --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P04.jsonl"` |
| P05 | 50/50 short/long bimodal input; short output; steady; no prefix | `$P3C materialize --id P05 --kind synthetic --input-mixture '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}' --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P05.jsonl"` |
| P06 | 50/50 short/long bimodal input; long output; bursty 8-at-once; no prefix | `$P3C materialize --id P06 --kind synthetic --input-mixture '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}' --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P06.jsonl"` |
| P07 | fixed 1,280-token input; long output; bursty 8-at-once; no prefix control | `$P3C materialize --id P07 --kind synthetic --input-fixed 1280 --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P07.jsonl"` |
| P08 | fixed 1,280-token input; long output; bursty 8-at-once; high prefix sharing | `$P3C materialize --id P08 --kind prefix-pool --num-prefixes 8 --prefix-len 1024 --suffix-fixed 256 --output-fixed 512 --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P08.jsonl"` |
| P09 | production-shaped 50/30/20 mixed input; short output; steady; no prefix | `$P3C materialize --id P09 --kind synthetic --input-mixture '{"uniform:128:512":0.5,"uniform:1024:2048":0.3,"uniform:4096:8192":0.2}' --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P09.jsonl"` |
| P10 | private real chat prompts; observed input <=32,768; output capped 256; steady; natural prefix | `$P3C materialize-private --id P10 --source /home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/chat_w20260311_1000.jsonl --sampling-u-max 0.125 --max-input-tokens 32768 --output-cap 256 --preserve-prompts --disable-shuffle --arrival steady --out "$PRIVATE/P10.jsonl"` |
| P11 | short-uniform input; long output; bursty 8-at-once; no prefix | `$P3C materialize --id P11 --kind synthetic --input-uniform 128:512 --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P11.jsonl"` |

### Total run count and GPU-hour estimate

- Pattern cells: **11**.
- TP1 pattern/config cells: **23** — C00 for all eleven, plus C10/C01/C11
  for P01/P03/P06/P10.
- TP2 counterpoint cells: **1** — P10/C00-TP2.
- Primary runs: **48** — saturation and moderate for each of 24 cells.
- Layer-1-only confirmation runs: **4**.
- Total measured runs: **52**, plus five unmeasured compile burn-ins.
- Layer-2 windows: **96 logical windows**, 2+8 iterations each; 100 expected
  rank trace files.
- Fixed throughput-accounted time: **240 seconds per measured run**.
- Expected cost: **about 7.1 hours serialized wall time and 7.5 H20-hours**;
  hard stop at 8.0 H20-hours.
- Expected public artifact volume: approximately 2.5–4 GB; hard stop at 8 GB.

### Open decisions for the orchestrator

1. **Blocking:** approve the fixed-duration helper interface/algorithm and a
   no-GPU implementation-and-test turn before execution. The existing bundled
   client cannot enforce this protocol's fixed clean duration and profile masks.
2. **Blocking:** approve copying the sensitive P10 source to the mode-0700
   dash0 private directory, its `sampling_u<=0.125`, input<=32,768 selection,
   and 256-token output cap. Rejection requires a protocol amendment; P10 will
   not be silently replaced.
3. Approve the sparse scheduler design: MNS 64/default 1024 and MBT
   2048/default 8192, full factorial only on P01/P03/P06/P10, chunked prefill
   always on.
4. Approve P10/C00 as the single TP2 counterpoint rather than adding TP4 or a
   second pattern.
5. Approve normalized moderate load `rho=0.60`, deterministic eight-request
   bursts, 240 clean seconds, and exactly four Layer-2 windows per cell.
6. Approve the mapping priority, 70% classifiability gate, materiality
   thresholds, and confirm/refute/inconclusive decision rule.

### Protocol sanity block

| Numeric family | n | Min | Max | Distinct | Invariant/result |
|---|---:|---:|---:|---:|---|
| Pattern rows | 11 | P01 | P11 | 11 | Within required 8–12 range |
| Server configs | 5 | TP1/MNS64/MBT2048 | TP2/MNS1024/MBT8192 | 5 | Four TP1 factorial configs plus one TP2 counterpoint |
| TP size | 5 configs | 1 | 2 | 2 | TP1 primary; exactly one TP2 cell |
| MNS level | 5 configs | 64 | 1024 | 2 | Small/default only |
| MBT level | 5 configs | 2048 | 8192 | 2 | Small/default only; chunking always on |
| Pattern/config cells | 24 | 1 config/pattern | 5 configs for P10 incl. TP2 | 3 counts | Per-pattern config counts are 1, 4, or 5; 23 TP1 + 1 TP2 |
| Primary load runs | 48 | 2/cell | 2/cell | 1 | Saturation plus moderate for every cell |
| Confirmation runs | 4 | 1/sentinel | 1/sentinel | 1 | P10/P06/P03/P01 moderate C00, Layer-1 only |
| Total measured runs | 1 aggregate | 52 | 52 | 1 | `48+4=52` |
| Clean duration/run | 52 | 240 s | 240 s | 1 expected | Three exact 80-second segments |
| Logical Layer-2 windows/cell | 24 | 4 | 4 | 1 expected | Two per load point |
| Logical Layer-2 windows | 1 aggregate | 96 | 96 | 1 | `24*4=96` |
| Active profiler iterations | 1 aggregate | 768 | 768 | 1 | `96*8=768`; every window also has two warm-ups |
| Expected rank trace files | 1 aggregate | 100 | 100 | 1 | 92 TP1 plus 8 TP2 rank files |
| Local real-trace rows | 1 source | 32,606 | 32,606 | 1 | Prompt contents never printed or copied in this turn |
| Frozen P10 selected rows | 1 subset | 4,011 | 4,011 | 1 | Selection reproduced locally without prompt output |
| P10 selected input tokens | 4,011 | 69 | 32,704 | 3,424 | Sum 36,499,001; median 7,009; p95 25,629 |
| P10 capped output tokens | 4,011 | 2 | 256 | 236 | Sum 878,291; median/p95 both 256 |
| Estimated H20-hours | 1 budget | 7.5 | 7.5 | 1 | Positive and below hard cap 8.0 |
| Phase-3 GPU/SSH actions this turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied |

Checked arithmetic invariants: `11+4*3=23` TP1 cells; adding one TP2 cell gives
24; two load points give 48 primary runs; four confirmations give 52 measured
runs; `24*4=96` logical profiler windows; TP2 adds four extra rank files, giving
100. Ratios and thresholds are within their declared domains. Expected constants
are labeled. No Phase 3 GPU run, SSH action, trace transfer, helper
implementation, patch change, or prompt disclosure occurred in this turn.
