Gahow Wang
d5b276180d
Workload-conditioned operator profiling on patched vLLM 0.24.0 + Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64 raggedness, 8-45% token-efficiency loss vs rectangular controls); mechanism decomposition kills the padding narrative and finds the arrival-uniformization artifact (-12.9%); cross-version churn surface shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held. Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer sidecars and metrics are tracked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1093 lines
58 KiB
Markdown
1093 lines
58 KiB
Markdown
# OpProf Phase 3 pre-registered pattern-matrix protocol
|
||
|
||
Status: **DRAFT FOR ORCHESTRATOR REVIEW — DO NOT EXECUTE**.
|
||
|
||
Date frozen: 2026-07-12 (Asia/Singapore). This document defines the Phase 3
|
||
experiment before any Phase 3 GPU launch. It does not authorize a run, change
|
||
the accepted five-patch series, or expose the prompt-bearing trace. Any change
|
||
to a pattern, load fraction, configuration, profiler placement, metric,
|
||
decision threshold, or exclusion rule requires a dated amendment reviewed
|
||
before the affected GPU run.
|
||
|
||
## Amendments (pre-execution; 2026-07-12)
|
||
|
||
All six open decisions in this document are approved as proposed. The two
|
||
amendments below are normative and take precedence wherever the original
|
||
serial-execution wording or estimate conflicts with them. Neither amendment
|
||
changes the pattern matrix, load points, profiler placement, metrics, or hard
|
||
validity gates.
|
||
|
||
### A-P3-1 — eight-way parallel execution and co-location gate
|
||
|
||
The matrix is executed on dash0 with one independent pattern/config cell per
|
||
physical GPU, up to eight cells concurrently on GPU0--GPU7. Each GPU has its
|
||
own server, client, port, run directory, and `CUDA_VISIBLE_DEVICES` namespace.
|
||
No TP1 server spans GPUs. The P10/TP2 counterpoint is scheduled as one cell
|
||
using two GPUs and leaves the corresponding second slot empty.
|
||
|
||
Eight-way execution is authorized only after this pre-registered co-location
|
||
validity check:
|
||
|
||
1. Use P05/C00 on physical GPU0 as the target. First calibrate its saturation
|
||
throughput and run its moderate load (`0.60*T_sat`) with no other Phase 3
|
||
GPU work on the host.
|
||
2. Recalibrate P05/C00 saturation and run the same moderate target on GPU0
|
||
while P01, P02, P03, P04, P06, P07, and P08 under C00 run closed-loop
|
||
saturation on GPU1--GPU7, respectively. The seven background loads remain
|
||
active through the target's warm-up, clean interval, profile windows, and
|
||
recovery intervals. Only the target is profiled.
|
||
3. Each saturation calibration uses the same placement regime as the
|
||
moderate run it supplies: solo calibration for solo moderate and eight-way
|
||
calibration with all seven backgrounds for eight-way moderate. The target
|
||
manifest, fixed seed, server configuration, fresh-server timeline, 60-second
|
||
warm-up, and 240-second clean interval are otherwise identical.
|
||
4. Let `T_solo` and `T_coloc` be target completed-request throughput over the
|
||
clean interval. Let `F3` be the three operator families with greatest
|
||
aggregate device-duration share in the solo target's accepted Layer-2
|
||
validation window. This one-time placement check samples exactly one 2+8
|
||
window per moderate target; it is not a matrix cell and does not change the
|
||
two-window-per-load matrix rule. The check passes only if
|
||
`abs(T_coloc/T_solo - 1) < 0.03` and, for every `f` in `F3`,
|
||
`abs(share_coloc[f] - share_solo[f]) < 0.03`. The latter thresholds are
|
||
absolute share fractions (three percentage points), not relative percent
|
||
changes. Both target validation windows (one solo and one co-located) must
|
||
satisfy the existing trace and
|
||
classifiability gates before this comparison is made.
|
||
5. If eight-way fails, repeat the regime-specific saturation and moderate
|
||
comparison at four-way placement: P05/C00 on GPU0 plus P01, P03, and P06
|
||
C00 saturation backgrounds on GPU1--GPU3. Four-way is authorized only if
|
||
the same throughput and `F3` share criteria pass. Failure at four-way stops
|
||
execution for orchestrator review; it does not authorize a smaller regime.
|
||
|
||
The CPU map is fixed from dash0's observed topology on 2026-07-12. Server and
|
||
client descendants inherit the same `taskset` mask, and masks are disjoint:
|
||
|
||
| Physical GPU | NUMA node | Server/client logical CPUs |
|
||
|---:|---:|---|
|
||
| 0 | 0 | 0-19 |
|
||
| 1 | 0 | 20-39 |
|
||
| 2 | 0 | 40-59 |
|
||
| 3 | 0 | 60-79 |
|
||
| 4 | 1 | 80-99 |
|
||
| 5 | 1 | 100-119 |
|
||
| 6 | 1 | 120-139 |
|
||
| 7 | 1 | 140-159 |
|
||
|
||
`nvidia-smi topo -m` must still show GPU0--3 local to NUMA0 and GPU4--7 local
|
||
to NUMA1 before execution. Every run records host load average and a full GPU
|
||
clock snapshot before and after its measured interval, in addition to the
|
||
original process-contamination and environment records. Saturation calibration
|
||
runs use the same co-location regime as their measurement runs even when an
|
||
isolated calibration would be operationally simpler.
|
||
|
||
The aggregate H20-hour estimate remains approximately 7.5 hours because the
|
||
same cells and durations execute; the expected matrix wall time becomes
|
||
approximately 1.5--2 hours after burn-ins, dependent pairs, TP2 placement, and
|
||
confirmation runs are scheduled. The one-time co-location validation is
|
||
reported separately from matrix cost.
|
||
|
||
### A-P3-2 — detached, resumable execution
|
||
|
||
All GPU work is launched by a controller resident in the private dash0 work
|
||
directory under `setsid`/`nohup`; an interactive Codex or SSH lifetime is never
|
||
the owner of a measurement. The controller has a `--resume` mode and an atomic
|
||
per-run state file. A run reaches `complete` only after client, Layer-1,
|
||
Layer-2, sanity, command, clock/load, and zero-GPU-memory cleanup checks have
|
||
all succeeded. Resumption skips only such complete runs, rejects a changed
|
||
source/helper/manifest/config hash, and otherwise cleans up controller-owned
|
||
PIDs before restarting the incomplete run from its fresh-server boundary.
|
||
Completed output directories are idempotent and are never overwritten.
|
||
|
||
The controller records its PID/session, child process groups, phase,
|
||
timestamps, exit status, exact commands, and failure reason; state updates use
|
||
write-then-rename. It handles termination by stopping only its recorded child
|
||
process groups and preserving resumable state. Immutable copies and SHA-256
|
||
hashes of the client, controller, operator mapping, and launch configuration
|
||
live in each campaign's `runs/.../provenance/` directory. The orchestrator
|
||
polls the state and logs; it does not keep a foreground SSH process alive.
|
||
|
||
### A-P3-3 — per-class drain budget
|
||
|
||
The post-admission drain hard stop is class-specific. Output-512 burst cells
|
||
receive **240 seconds**; all other cells receive **120 seconds**. Under the
|
||
frozen pattern table, the 240-second class is exactly P04, P06, P07, P08, and
|
||
P11, including their configuration variants and confirmation runs. P02 has a
|
||
512-token output but steady arrivals, so it remains in the 120-second class.
|
||
The timer starts when final admission stops after the load point's last
|
||
profile/recovery interval. Exceeding the applicable budget invalidates the run
|
||
and stops the campaign; it does not authorize cancellation, truncation, or a
|
||
larger post-hoc allowance. This amendment supersedes the uniform 120-second
|
||
drain wording below but changes no clean interval or throughput denominator.
|
||
|
||
### A-P3-4 — cumulative Phase-3 GPU budget
|
||
|
||
The Phase-3 hard stop is **16.0 H20-hours cumulative**, including E-a
|
||
co-location validation, the shutdown-fix GPU verification, compile burn-ins,
|
||
the 52-run matrix, retries, and cleanup time while a server still owns GPU
|
||
memory. E-a consumed 3.964 H20-hours. Before every launch wave, the detached
|
||
controller records consumed cost and refuses to launch if the wave's
|
||
conservative reservation would exceed 16.0 H20-hours. This amendment
|
||
supersedes the 8.0 H20-hour hard stop below; the original 7.5 H20-hour matrix
|
||
estimate remains an estimate, not an additional allowance.
|
||
|
||
### A-P3-5 — crash-resilient Layer-1 accounting
|
||
|
||
The accepted OpProf series now ends at
|
||
`23450fb21ac255b0cf710f4ee965ee694921975d` and contains seven patches. Every
|
||
one-second JSONL flush atomically replaces `<stream>.footer.json` through a
|
||
same-directory temporary file. This checkpoint carries balanced encoded,
|
||
written, and dropped counters through the flush, the last written step index,
|
||
its wall-clock timestamp, the configured flush interval, and a `final` flag.
|
||
|
||
The controller's preferred shutdown remains the official vLLM 0.24.0 path:
|
||
start `vllm serve` with a positive `--shutdown-timeout` and signal the API
|
||
parent, which selects EngineCore `drain` mode. A clean run must contain an
|
||
in-stream footer, and the `final=true` sidecar must agree with its encoded,
|
||
written, and dropped counters. Graceful shutdown is not an accounting
|
||
dependency, however. If a deliberate or external hard kill leaves no
|
||
in-stream footer, the latest atomic sidecar is authoritative. All complete
|
||
JSONL data lines must decode; their count must equal sidecar
|
||
`written_records`; the final data-line step must equal sidecar
|
||
`last_step_index`; `encoded = written + dropped`; and `dropped=0`. The
|
||
controller records the signal timestamp and requires the checkpoint age to be
|
||
no more than the configured one-second flush interval, with at most 100 ms of
|
||
separately reported timestamp/scheduling tolerance. At most one flush interval
|
||
after the authoritative checkpoint may be lost. A missing/invalid sidecar,
|
||
counter mismatch, older checkpoint, partial JSON line, or nonzero drop count
|
||
invalidates the run.
|
||
|
||
The seventh patch also makes the preregistered two Layer-2 windows executable:
|
||
after `/stop_profile`, a stopped scheduled `TorchProfilerWrapper` is discarded
|
||
so the next `/start_profile` constructs a fresh 2+8 schedule. A CPU reproducer
|
||
on the pinned torch 2.11 showed that reusing the stopped object returned
|
||
without error but emitted no second trace. This repair changes neither the
|
||
profile schedule nor its placement.
|
||
|
||
### A-P3-5 — drain quarantine (orchestrator amendment; 2026-07-12)
|
||
|
||
The orchestrator assigned this amendment the same identifier as the preceding
|
||
crash-accounting amendment. This subsection is therefore distinguished by its
|
||
title; it supersedes only the drain-budget and drain-stop clauses of A-P3-3 and
|
||
leaves crash-resilient accounting unchanged.
|
||
|
||
Drain is post-measurement and serves as a hang watchdog. P10-class cells using
|
||
the long-context real trace receive **600 seconds**. P04/P06-class cells retain
|
||
the **240-second** allowance from A-P3-3; this class remains exactly P04, P06,
|
||
P07, P08, and P11. All other cells retain **120 seconds**.
|
||
|
||
Exceeding the applicable allowance marks a run `drain-quarantined` but no
|
||
longer stops the matrix. If its 240-second clean window, request/output
|
||
correctness, Layer-1 accounting, and Layer-2 artifacts are otherwise valid,
|
||
the run remains usable in analysis and is explicitly flagged in every table.
|
||
The matrix stops for drain behavior only when more than 20% of measured runs
|
||
are drain-quarantined. The existing greater-than-5% stop for clean-window
|
||
failures is unchanged.
|
||
|
||
The preserved P10/C01 saturation run from the first primary wave is
|
||
re-adjudicated against 600 seconds. Its observed 288.619107924-second drain is
|
||
within that allowance; no GPU rerun is authorized for the clean measurement.
|
||
|
||
### A-P3-6 — P10 warm-up stabilization gate (orchestrator amendment; 2026-07-12)
|
||
|
||
The warm-up gate measures whether service has reached steady state; completed
|
||
requests are only a proxy for that condition. For P10-class runs only, warm-up
|
||
passes if either at least 32 requests complete successfully during the
|
||
60-second warm-up, or at least 16 complete successfully and trailing Layer-1
|
||
telemetry meets the stabilization test below. All non-P10 warm-up rules remain
|
||
unchanged. This amendment changes neither the 240-second clean window nor any
|
||
throughput denominator.
|
||
|
||
The stabilization test is frozen before re-adjudicating retained telemetry:
|
||
|
||
1. Select `model_executed=true` Layer-1 records by `submit_mono_ns` in the
|
||
trailing warm-up quartile `[t0+45 s,t0+60 s)` and split them into the fixed
|
||
wall-time bins `[45,50)`, `[50,55)`, and `[55,60)` seconds.
|
||
2. Each bin must contain at least 16 model-executed steps. For bin `j`, define
|
||
scheduled-token throughput
|
||
`R_j=sum(prefill_tokens+decode_tokens)/5 s`. Non-finite or non-positive
|
||
values, missing bins, discontinuous step indices, or a failed Layer-1
|
||
accounting invariant fail the branch.
|
||
3. Fit ordinary least squares to `(47.5,R_1)`, `(52.5,R_2)`, and
|
||
`(57.5,R_3)`. Let `s` be the fitted slope and `R_bar` the mean of the three
|
||
rates. Stabilization passes only if
|
||
`abs(s)*15/R_bar <= 0.10`, i.e. the fitted drift across the complete
|
||
trailing quartile is no more than 10% of its mean.
|
||
|
||
Every P10 result admitted through the OR branch records the warm-up completion
|
||
count, three step counts, three token-throughput values, mean, fitted slope,
|
||
normalized drift, and pass/fail result. The final report must render this
|
||
post-hoc evidence explicitly; no missing value is imputed. Re-adjudication may
|
||
accept an already measured clean window only when this exact test passes.
|
||
|
||
### A-P3-7 — frozen partial-matrix inference (orchestrator amendment; 2026-07-12)
|
||
|
||
The Phase-3 matrix is final and frozen at 40/52 accepted measured runs and
|
||
20/24 complete pattern/config cells. No missing cell is rerun, replaced, or
|
||
imputed. The four incomplete cells, each missing both saturation and moderate
|
||
load points, are exactly **P03/C11**, **P05/C00**, **P10/C00-TP2**, and
|
||
**P11/C00**. The four Layer-1-only C00-moderate confirmations for P10, P06,
|
||
P03, and P01 are also absent; confirmation-run repeatability is therefore not
|
||
evaluable and cannot be substituted with within-run resampling.
|
||
|
||
H1a is evaluated over the completed primary C00 operator-share matrix. Its
|
||
formal moderate-load comparison contains exactly P01, P02, P03, P04, P06,
|
||
P07, P08, P09, and P10; P05 and P11 are missing. Saturation results for the
|
||
same completed patterns remain supporting evidence. A qualifying inversion
|
||
among completed patterns confirms H1a because H1a is existential. If none is
|
||
found, H1a is **INCONCLUSIVE**, not refuted: the original refutation rule
|
||
requires the complete eleven-pattern matrix and simultaneous bounds for every
|
||
registered comparison.
|
||
|
||
Each frozen H1b contrast evaluates if and only if both of its C00-moderate
|
||
cells are complete. Contrast status is fixed as follows:
|
||
|
||
| Frozen H1b contrast | A-P3-7 status |
|
||
|---|---|
|
||
| P05 vs P01 | NOT EVALUABLE — P05/C00 missing |
|
||
| P05 vs P03 | NOT EVALUABLE — P05/C00 missing |
|
||
| P06 vs P02 | EVALUABLE |
|
||
| P06 vs P04 | EVALUABLE |
|
||
| P09 vs P01 | EVALUABLE |
|
||
| P09 vs P03 | EVALUABLE |
|
||
| P10 vs P03 | EVALUABLE |
|
||
| P10 vs P04 | EVALUABLE |
|
||
|
||
The six evaluable contrasts retain their original waste thresholds, 5% useful
|
||
token-efficiency/step-residual association, correction families, seeds, and
|
||
decision logic. One qualifying evaluable contrast confirms H1b because H1b is
|
||
existential. If none qualifies, H1b is **INCONCLUSIVE**, never refuted, because
|
||
two frozen contrasts and four cells are missing. Thus 20/24 coverage permits
|
||
confirmation of H1a, H1b, or the compound hypothesis, but it cannot support
|
||
the original complete-matrix null/refutation claim. Missing results are
|
||
reported as `NOT EVALUABLE` and never assigned zero effect, copied controls,
|
||
or synthetic uncertainty.
|
||
|
||
## Goal and falsifiable hypothesis
|
||
|
||
The system boundary is Qwen3-30B-A3B BF16 served by patched vLLM 0.24.0 on
|
||
dash0 H20 GPUs. Phase 3 asks whether the device-level bottleneck observed under
|
||
one serving pattern generalizes to other operationally defined patterns.
|
||
|
||
The headline hypothesis has two jointly required parts:
|
||
|
||
- **H1a — composition:** at the same normalized offered load, at least one pair
|
||
of serving patterns has a material inversion in its top GPU operator-family
|
||
ranking. “Material” is defined below as a replicated rank inversion with at
|
||
least a five-percentage-point share gap, not a visual change in a bar chart.
|
||
- **H1b — rectangular-benchmark miss:** at least one non-rectangular serving
|
||
pattern exhibits material padding, CUDA-graph miss/bucket mismatch, sequence
|
||
raggedness, or mixed-prefill/decode interference that a rectangular offline
|
||
batch omits, and that waste is associated with at least a 5% loss in useful
|
||
token efficiency or step-time residual relative to its rectangular control.
|
||
|
||
The **null outcome** is that the same operator family remains top-ranked across
|
||
all primary patterns at matched load, all simultaneous 95% bounds on ranking
|
||
gaps are below five percentage points, and every pre-declared waste contrast is
|
||
below its materiality threshold. If uncertainty or profiler opacity prevents
|
||
either conclusion, the result is **INCONCLUSIVE**, not evidence for the null.
|
||
|
||
The full round hypothesis is confirmed only if both H1a and H1b pass. Passing
|
||
only one is reported as a partial finding and does not justify the compound
|
||
claim.
|
||
|
||
## Pinned context and non-goals
|
||
|
||
| Item | Frozen value |
|
||
|---|---|
|
||
| vLLM base | v0.24.0, `ee0da84ab9e04ac7610e28580af62c365e898389` |
|
||
| Accepted local patch tip | `23450fb21ac255b0cf710f4ee965ee694921975d` |
|
||
| Patch series | Seven checksum-recorded patches in `patches/vllm-0.24.0-opprof/` |
|
||
| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, community BF16 |
|
||
| Hardware | dash0 H20; TP1 primary, one TP2 pattern/config counterpoint |
|
||
| Primary backend expectation | TRITON unquantized MoE; every run must record the observed backend |
|
||
| Layer-1 overhead evidence | -0.04196%, 95% CI [-0.17443%, 0.04550%] |
|
||
| Layer-2 perturbation evidence | 51.30% request-rate reduction during the active P2 collection window |
|
||
| Seeds | workload `20260712`; execution order `20260713`; bootstrap/permutation `20260714` |
|
||
|
||
This protocol does not depend on L-C-A labels, does not tune kernels, does not
|
||
change graph capture sizes, and does not claim production-cluster generality.
|
||
L-C-A may be computed later as a descriptive annotation but cannot define,
|
||
filter, or reorder the Phase 3 cells.
|
||
|
||
Pinned vLLM source facts used here:
|
||
|
||
- on an H20-class device (at least 70 GiB, not A100), the OpenAI server defaults
|
||
to `max_num_seqs=1024` and `max_num_batched_tokens=8192`
|
||
(`vllm/engine/arg_utils.py:2375-2456`);
|
||
- chunked prefill is supported and defaults on for this model
|
||
(`vllm/engine/arg_utils.py:2458-2498`);
|
||
- the random dataset uses integer-uniform length ranges and generates one fixed
|
||
prefix per manifest (`vllm/benchmarks/datasets/datasets.py:557-771`); and
|
||
- the bundled finite-rate generator uses Gamma inter-arrivals, but it has no
|
||
fixed-duration mode (`vllm/benchmarks/serve.py:391-499`).
|
||
|
||
The last point requires a small fixed-duration client before execution. Its
|
||
normative interface and behavior are frozen below; the helper is not
|
||
implemented in this protocol-only turn.
|
||
|
||
## Operational pattern axes
|
||
|
||
### Input-length distribution
|
||
|
||
All lengths are post-tokenization token counts under the model tokenizer.
|
||
|
||
- **short-uniform:** discrete integer uniform `U[128,512]`.
|
||
- **long-uniform:** discrete integer uniform `U[4096,8192]`.
|
||
- **bimodal:** independent 50/50 draw from `U[128,512]` and
|
||
`U[4096,8192]`; exactly half of a 32,768-request manifest comes from each
|
||
mode, followed by a seed-fixed permutation.
|
||
- **production-shaped mixed:** exactly 50% `U[128,512]`, 30%
|
||
`U[1024,2048]`, and 20% `U[4096,8192]`, followed by a seed-fixed
|
||
permutation. This is synthetic and is not called a production trace.
|
||
- **real-trace anchor:** the approved private subset of
|
||
`chat_w20260311_1000` described below. It preserves prompt text and observed
|
||
input lengths, filters inputs above 32,768 tokens, and caps requested output
|
||
at 256 tokens for bounded execution.
|
||
|
||
### Output length
|
||
|
||
- **short:** exactly 64 tokens.
|
||
- **long:** exactly 512 tokens.
|
||
- **trace-capped:** `min(observed_output_length, 256)`, used only by the real
|
||
anchor.
|
||
|
||
Every request uses greedy temperature 0 and `ignore_eos`; output work is
|
||
therefore exact even if generated content is not bit-identical.
|
||
|
||
### Arrival shape
|
||
|
||
Arrival shape applies at the moderate load point. The saturation point uses
|
||
`request_rate=inf`, so all patterns intentionally collapse to a continuously
|
||
backlogged client; saturation is used to isolate service composition, not to
|
||
test inter-arrival shape.
|
||
|
||
- **steady:** deterministic spacing `1/lambda` seconds, first request at
|
||
`t=0`.
|
||
- **bursty:** deterministic bursts of exactly eight simultaneous requests,
|
||
first burst at `t=0`, with burst period `8/lambda` seconds and no within-burst
|
||
delay or jitter.
|
||
|
||
Here `lambda` is the pattern/config-specific moderate rate defined as 60% of
|
||
that pair's measured saturation request throughput. These definitions avoid
|
||
conflating the arrival experiment with random arrival noise.
|
||
|
||
P02 and P11 are the matched steady/bursty arrival pair: both use short-uniform
|
||
input, 512-token output, and no shared prefix.
|
||
|
||
### Prefix sharing
|
||
|
||
- **none:** no intentionally shared prefix. Synthetic token streams are unique
|
||
per request. The manifest must have no common prefix longer than 16 tokens,
|
||
and the clean Layer-1 prefix-query hit ratio must remain below 1%; otherwise
|
||
the cell is invalid.
|
||
- **high:** a pool of eight independently generated 1,024-token prefixes. Each
|
||
request appends a unique 256-token suffix, so 80% of each 1,280-token prompt
|
||
is shared within one-eighth of requests. Requests are seed-shuffled after
|
||
balanced assignment to prefixes.
|
||
- **natural:** no prefix is added or removed from the private real prompts. The
|
||
observed prefix-hit ratio is reported rather than constrained.
|
||
|
||
P07 and P08 are a matched no-prefix/high-prefix pair: both have total input
|
||
length 1,280, output length 512, and bursty arrival.
|
||
|
||
## Required fixed-duration client
|
||
|
||
Before any GPU use, an execution turn must implement and test
|
||
`scripts/opprof_phase3_client.py`. The orchestrator must review its diff and
|
||
freeze its SHA-256. The helper may reuse vLLM's tokenizer and endpoint request
|
||
function, but must not modify vLLM.
|
||
|
||
The client contract is:
|
||
|
||
1. `materialize` creates a token-exact JSONL manifest using the pinned model
|
||
tokenizer and `numpy.random.default_rng(20260712)`. Synthetic manifests
|
||
contain 32,768 requests and no manifest may wrap during a run.
|
||
2. `run` supports `--request-rate inf` as a closed-loop saturation stream with
|
||
at most 256 outstanding requests. A finite rate uses the exact steady or
|
||
eight-request periodic-burst schedule above.
|
||
3. Admission continues during profiler and recovery intervals. The client
|
||
stops admitting after the final clean segment and drains outstanding
|
||
requests for at most 120 seconds; it never cancels and counts a partial
|
||
response as a failure.
|
||
4. The client records request ID, scheduled/admitted/first-token/completion
|
||
monotonic timestamps, prompt tokens, requested and actual output tokens,
|
||
HTTP status, and no prompt or generated text. It emits one result file per
|
||
segment and one manifest SHA-256.
|
||
5. `/start_profile` and `/stop_profile` calls are synchronous and timestamped.
|
||
Both profile windows occur after the uninterrupted clean interval. Profile
|
||
and recovery intervals are tagged and excluded from throughput/latency
|
||
summaries.
|
||
6. A dry, CPU-only test must prove fixed segment duration, saturation
|
||
concurrency, exact burst timing, no manifest wrap, exact output-token
|
||
requests, and prompt-field redaction.
|
||
|
||
The exact saturation command interface is:
|
||
|
||
```bash
|
||
python scripts/opprof_phase3_client.py run \
|
||
--manifest "$MANIFEST" --base-url http://127.0.0.1:8000 \
|
||
--model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \
|
||
--load-point saturation --request-rate inf \
|
||
--max-concurrency 256 --ignore-eos --temperature 0 \
|
||
--warmup-seconds 60 --clean-segment-seconds 80 \
|
||
--num-clean-segments 3 --profile-after-clean --num-profile-windows 2 \
|
||
--profile-warmup-iterations 2 --profile-active-iterations 8 \
|
||
--recovery-seconds 30 --drain-timeout-seconds 120 \
|
||
--workload-seed 20260712 --result-dir "$PAIR_DIR/saturation/client"
|
||
```
|
||
|
||
The exact moderate command is:
|
||
|
||
```bash
|
||
python scripts/opprof_phase3_client.py run \
|
||
--manifest "$MANIFEST" --base-url http://127.0.0.1:8000 \
|
||
--model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \
|
||
--load-point moderate \
|
||
--saturation-result "$PAIR_DIR/saturation/client/result.json" \
|
||
--rate-fraction 0.60 \
|
||
--max-concurrency 256 --ignore-eos --temperature 0 \
|
||
--warmup-seconds 60 --clean-segment-seconds 80 \
|
||
--num-clean-segments 3 --profile-after-clean --num-profile-windows 2 \
|
||
--profile-warmup-iterations 2 --profile-active-iterations 8 \
|
||
--recovery-seconds 30 --drain-timeout-seconds 120 \
|
||
--workload-seed 20260712 --result-dir "$PAIR_DIR/moderate/client"
|
||
```
|
||
|
||
The client derives `lambda=0.60*T_sat` from the accepted saturation result and
|
||
refuses a manually supplied finite rate. Arrival class comes from the immutable
|
||
manifest. Total throughput-accounted duration is exactly `3*80=240` seconds
|
||
per measured run.
|
||
|
||
## Private real-trace anchor
|
||
|
||
The local prompt-bearing source is:
|
||
|
||
```text
|
||
/home/gahow/phd/aituner/trace_windows/traces/chat_w20260311_1000.jsonl
|
||
```
|
||
|
||
It is 1,183,031,556 bytes with SHA-256
|
||
`f539f38eb0ee0f750e3c23ff47df6eed3faf723a25f1444d55665a85871750b9`.
|
||
Metadata records 32,606 requests over 600 seconds. The frozen selection
|
||
`sampling_u <= 0.125 && input_length <= 32768` yields 4,011 requests locally;
|
||
after the 256-token output cap, input min/median/p95/max is
|
||
69/7,009/25,629/32,704 and output min/median/p95/max is 2/256/256/256.
|
||
The anchor preserves prompt text, prompt/output-length correlation, and source
|
||
order; it deliberately replaces the original 600-second timestamps with the
|
||
same controlled saturation/moderate load protocol as every other cell. It is a
|
||
real-content anchor, not a native-traffic replay.
|
||
|
||
If and only if the orchestrator approves this anchor, execution may copy the
|
||
source to
|
||
`/home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/` on dash0. The
|
||
directory must be mode 0700 and files mode 0600. The source and derived
|
||
manifest must never enter a run directory, tar archive, Git, client result, or
|
||
Kineto/OpProf artifact. Transfer is accepted only after the local and remote
|
||
SHA-256 match.
|
||
|
||
The private materialization command is specified in the final pattern table.
|
||
It retains only `prompt`, capped `output_tokens`, and an expected input-token
|
||
count. A tokenizer parity precheck must find exact input length for at least
|
||
99% of rows and absolute error at most one token for every row; failure stops
|
||
the anchor rather than rewriting prompts.
|
||
|
||
## Server configuration axes
|
||
|
||
Chunked prefill remains explicitly enabled in every configuration. Disabling
|
||
it on this model makes vLLM raise MBT to the model length when MBT is omitted,
|
||
or reject MBT below model length when supplied
|
||
(`vllm/config/scheduler.py:272-284`). That would conflate chunking with a much
|
||
larger token budget. Phase 3 therefore uses an MBT factor instead.
|
||
|
||
| Config | TP | MNS | MBT | Chunked prefill | Purpose |
|
||
|---|---:|---:|---:|---|---|
|
||
| C00 | 1 | default = 1024 | default = 8192 | on | Primary vLLM/H20 baseline |
|
||
| C10 | 1 | small = 64 | default = 8192 | on | Bound sequence concurrency / decode batch |
|
||
| C01 | 1 | default = 1024 | small = 2048 | on | Force smaller prefill chunks and token batches |
|
||
| C11 | 1 | small = 64 | small = 2048 | on | Test MNS×MBT interaction |
|
||
| C00-TP2 | 2 | default = 1024 | default = 8192 | on | One communication/parallelism counterpoint |
|
||
|
||
C00 runs for all eleven patterns. The complete 2×2 TP1 factorial C00/C10/C01/C11
|
||
runs only for sentinel patterns P01, P03, P06, and P10. C00-TP2 runs only for
|
||
P10. This sparse design identifies each scheduler factor on short-decode,
|
||
long-prefill, irregular mixed, and real-trace regimes without spending a full
|
||
factorial on every pattern.
|
||
|
||
The common server command is:
|
||
|
||
```bash
|
||
CUDA_VISIBLE_DEVICES="$GPU_IDS" \
|
||
VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
|
||
vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \
|
||
--tensor-parallel-size "$TP" --enable-chunked-prefill \
|
||
--enable-prefix-caching \
|
||
--profiler-config "{\"profiler\":\"torch\",\"torch_profiler_dir\":\"$TRACE_TMP\",\"torch_profiler_with_stack\":true,\"torch_profiler_record_shapes\":true,\"torch_profiler_use_gzip\":true,\"ignore_frontend\":true,\"wait_iterations\":0,\"warmup_iterations\":2,\"active_iterations\":8}"
|
||
```
|
||
|
||
C00/C00-TP2 omit MNS and MBT flags. C10 adds `--max-num-seqs 64`; C01 adds
|
||
`--max-num-batched-tokens 2048`; C11 adds both. Startup logs must show the
|
||
frozen effective values. Defaults changing from 1024/8192 is a stop condition,
|
||
not permission to edit the protocol.
|
||
|
||
## Load and execution protocol
|
||
|
||
### Unit of analysis and load points
|
||
|
||
A **pattern/config cell** is one pattern row under one server configuration.
|
||
An **execution run** is one load point for a pattern/config cell. Each primary
|
||
cell has exactly two fresh-server runs:
|
||
|
||
1. **saturation:** closed-loop `request_rate=inf`, maximum 256 outstanding;
|
||
2. **moderate:** the same manifest at `lambda=0.60*T_sat`, where `T_sat` is
|
||
clean-segment completed-request throughput from that cell's accepted
|
||
saturation run.
|
||
|
||
Thus patterns are matched at normalized load `rho=0.60`, not at equal absolute
|
||
requests/s or tokens/s. Saturation is analyzed separately. Moderate admission
|
||
is valid only if achieved offered rate is within 5% of `lambda` and no sustained
|
||
client lag exceeds one second.
|
||
|
||
### Warm-up, clean measurement, and profiler placement
|
||
|
||
Every measured run uses a fresh server and this timeline:
|
||
|
||
```text
|
||
60 s warm-up (excluded; require >=32 completions)
|
||
80 s clean segment A
|
||
80 s clean segment B
|
||
80 s clean segment C
|
||
Layer-2 window 1: 2 warm-up + 8 active engine iterations (excluded)
|
||
30 s recovery (excluded)
|
||
Layer-2 window 2: 2 warm-up + 8 active engine iterations (excluded)
|
||
30 s recovery / trace flush (excluded)
|
||
stop admission; drain <=120 s (excluded)
|
||
```
|
||
|
||
The clean A/B/C segments are contiguous. The first profile window occurs only
|
||
after exactly 240 accumulated clean seconds; the second follows the first
|
||
fixed 30-second recovery, never an observed favorable batch. There are exactly
|
||
**four Layer-2 windows per pattern/config cell**: two in saturation and two in
|
||
moderate. The 240 clean seconds, not server lifetime or profiler interval, form
|
||
the throughput/latency denominator. This ordering prevents the 51.30%
|
||
active-window perturbation measured in P2 from contaminating throughput.
|
||
|
||
Offered rate runs continuously from warm-up through the clean interval.
|
||
Completed throughput counts completions timestamped inside the 240-second clean
|
||
interval; latency summarizes those same completions, including a request
|
||
admitted during warm-up if it completes cleanly after the boundary. Admission
|
||
rate is counted by admission timestamp. This is a stationary-window rule and
|
||
is applied identically to every pattern; no profiler interval precedes clean
|
||
measurement.
|
||
|
||
After each profile window, the 10-second rolling completion rate and
|
||
waiting-queue depth must return within 10% of clean-segment-C medians by the end
|
||
of the fixed 30-second recovery. This gate protects the second window and
|
||
validates post-profile recovery; no clean throughput is measured afterward.
|
||
Failure invalidates the run, and recovery is not silently extended.
|
||
|
||
### Compile warm-up and execution order
|
||
|
||
Before measured runs, launch one unmeasured burn-in server for each of the five
|
||
server configurations C00, C10, C01, C11, and C00-TP2. Each burn-in performs
|
||
60 seconds of P06 saturation and exits gracefully. It warms compile/AOT and
|
||
Triton artifacts but is never pooled with a pattern result.
|
||
|
||
After all burn-ins, sort pattern/config IDs by
|
||
`SHA256("20260713:" + pattern_id + ":" + config_id)` ascending. Run each
|
||
saturation/moderate pair contiguously because moderate depends on saturation.
|
||
Four Layer-1-only confirmation runs repeat moderate C00 for P10, P06, P03, and
|
||
P01, in that fixed reverse order, after the matrix. They use 60-second warm-up
|
||
and 240 clean seconds but no profiler endpoint; they quantify fresh-server
|
||
repeatability without changing the pre-declared four Layer-2 windows per cell.
|
||
|
||
Every GPU launch must first record `nvidia-smi`, selected UUID(s), clock state,
|
||
host load, other-user processes, exact command, expected duration, source and
|
||
manifest hashes, compile-cache key, and effective vLLM config. One H20 is used
|
||
except for the two C00-TP2 runs and its TP2 burn-in. No other user's process may
|
||
be touched. The preferred controller path starts the server with a positive
|
||
`--shutdown-timeout` and signals the API parent so EngineCore drains before the
|
||
process manager exits. Accounting is validated by the A-P3-5 footer/sidecar
|
||
rule even when graceful teardown does not occur, and GPU memory must finally
|
||
return to zero.
|
||
|
||
## Measurement plan
|
||
|
||
### Layer 1: always on
|
||
|
||
Every measured and confirmation run sets `VLLM_OPPROF_DIR`. Layer 1 is the
|
||
source of clean-segment composition and waste counters:
|
||
|
||
- scheduled requests, prefill/decode batch and token counts;
|
||
- chunked-prefill class and chunk-size histogram;
|
||
- context-length histogram;
|
||
- waiting/running/deferred queues, preemptions, KV usage, and prefix counters;
|
||
- step submit/complete timestamps; and
|
||
- CUDA-graph hit, runtime mode, unpadded tokens, bucket tokens, and padding.
|
||
|
||
Each run must produce exactly one schema-v1 JSONL for TP1 or one scheduler-owned
|
||
stream for the TP2 engine. Every line must msgspec-decode; step indices must be
|
||
unique and contiguous. On clean close, the in-stream footer must satisfy
|
||
`encoded = written + dropped`, `dropped=0`, and its step count must equal the
|
||
record count, while the final sidecar agrees. Without an in-stream footer, the
|
||
atomic sidecar and on-disk records must satisfy the A-P3-5 hard-kill rule.
|
||
Profile/recovery records remain in the raw stream but are tagged by monotonic
|
||
time and excluded from clean summaries.
|
||
|
||
Composition is summarized separately for clean A/B/C, each profiler window,
|
||
each recovery interval, and the complete clean union. No profile-window record
|
||
may leak into throughput, latency, or unperturbed composition statistics.
|
||
|
||
### Layer 2: sampled Kineto windows
|
||
|
||
Each profile endpoint invocation uses the server's frozen wait=0, warm-up=2,
|
||
active=8 schedule. A logical window is accepted only if:
|
||
|
||
- `/start_profile` and `/stop_profile` return HTTP 200;
|
||
- the trace gzip decompresses and parses as JSON;
|
||
- active `ProfilerStep` numbers are exactly 2 through 9;
|
||
- exactly eight matching execute annotations are present per TP rank;
|
||
- kernel event count is positive; and
|
||
- every active execute annotation joins unambiguously to its Layer-1 step.
|
||
|
||
TP1 yields one worker trace per logical window. TP2 is expected to yield two
|
||
rank traces; both are required and analyzed per rank. Rank times are never
|
||
summed and called latency.
|
||
|
||
The client writes a monotonic timestamp immediately before/after each endpoint
|
||
call. The analysis joins execute annotations to Layer-1 records by timestamp,
|
||
order, and the annotation's context/generation token counts. A join is invalid
|
||
if more than one Layer-1 record is compatible or if the annotation and Layer-1
|
||
token totals disagree.
|
||
|
||
Profiler representativeness is checked against the adjacent clean segment for
|
||
scheduled tokens, prefill fraction, decode batch size, and graph-mode shares.
|
||
For each feature, use standardized mean difference
|
||
`SMD=(mean_profile-mean_clean)/pooled_sd`; a constant feature uses exact
|
||
equality. If `abs(SMD)>0.25` for any feature in either window, that load point's
|
||
operator breakdown is invalid. One complete run may be retried with the same
|
||
command; a second failure is reported as Layer-2-inconclusive, not resampled
|
||
until it looks representative.
|
||
|
||
### Kernel-to-operator-family mapping
|
||
|
||
The primary operator share is summed GPU kernel duration in a family divided
|
||
by total GPU kernel/replay activity duration in the eight active steps,
|
||
including unmatched/opaque activity in the denominator. It is computed per
|
||
window and rank. CPU launch time, CUDA runtime time, memcpy/memset time, and
|
||
unioned device-busy time are reported separately. Because streams may overlap,
|
||
kernel-duration shares are composition shares, not fractions of wall time.
|
||
|
||
Mapping is priority-ordered; first match wins. The execution helper must emit a
|
||
versioned regex table and SHA-256 before GPU work. The rules below are the
|
||
normative v1 mapping:
|
||
|
||
| Priority | Family | Name/scope rule |
|
||
|---:|---|---|
|
||
| 1 | `moe_router` | `topkGating`, `moe_align`, `select_expert`, or MoE-scoped routing/top-k kernels; this precedes sampler matching |
|
||
| 2 | `collective` | `nccl`, `all_reduce`, `allreduce`, `reduce_scatter`, `all_gather`, or vLLM custom-all-reduce names |
|
||
| 3 | `attention` | `flash`, `fmha`, `paged_attention`, `unified_attention`, attention-scoped CUTLASS kernels, or attention KV/cache reshape kernels |
|
||
| 4 | `moe_gemm` | `fused_moe`, `moe.*gemm`, `nvjet`, or CUTLASS/Triton GEMM whose parent scope is FusedMoE/MoE |
|
||
| 5 | `sampler` | non-MoE `sampling`, `topk`, `topp`, `argmax`, `multinomial`, logits-processor, or penalty kernels |
|
||
| 6 | `dense_gemm` | GEMM/matmul/CUTLASS kernels outside attention and MoE scopes |
|
||
| 7 | `norm_elementwise` | RMS/layer norm, activation, residual/add, reduction, fill, and vectorized elementwise kernels |
|
||
| 8 | `kv_memory` | cache copy/swap kernels and KV movement not already attributed to attention; CUDA memcpy/memset activities remain separately reported |
|
||
| 9 | `other` | every unmatched kernel; no post-hoc cell-specific reassignment |
|
||
|
||
Attention is subdivided by the joined Layer-1 step:
|
||
|
||
- `attention_prefill`: prefill tokens >0 and decode tokens =0;
|
||
- `attention_decode`: decode tokens >0 and prefill tokens =0; and
|
||
- `attention_mixed`: both are nonzero.
|
||
|
||
Mixed attention is not force-split in proportion to tokens. The headline
|
||
bottleneck ranking uses `attention_total`; the three subfamilies explain which
|
||
serving phase supplied it.
|
||
|
||
Before inference, report the top 20 unmatched kernel names by duration. If
|
||
`other` plus opaque graph replay exceeds 30% of GPU kernel time in any primary
|
||
window, that window is not classifiable. No mapping amendment may be derived
|
||
from one favored cell: a reviewed global amendment must be applied to every
|
||
trace, with old and new results both retained.
|
||
|
||
### CUDA-graph mode segmentation
|
||
|
||
Every active step is segmented by its Layer-1 `runtime_mode`:
|
||
|
||
- `FULL`;
|
||
- `PIECEWISE`;
|
||
- `NONE`, reported as eager/ungraphed; or
|
||
- `ambiguous`, which invalidates the step.
|
||
|
||
Operator shares and waste are reported both overall and by mode when a mode has
|
||
at least eight active steps across the two windows of a load point. If CUPTI
|
||
shows only a graph-launch event with no child kernel activity for FULL or
|
||
PIECEWISE, its device time is labeled `graph_replay_opaque`; it is never
|
||
distributed among operator families. A cell cannot support H1a unless at least
|
||
70% of its device kernel time is classifiable.
|
||
|
||
## Metrics and pre-declared analysis
|
||
|
||
### End-to-end and composition metrics
|
||
|
||
For each clean segment and their 240-second union, report:
|
||
|
||
- completed and failed requests; achieved offered and completed requests/s;
|
||
- input, output, and total useful tokens/s;
|
||
- TTFT, TPOT, and end-to-end latency: mean, p50, p95, and p99;
|
||
- scheduled tokens/step, requests/step, prefill/decode tokens/step, decode batch,
|
||
queue depths, KV usage, preemptions, and prefix hit/query ratios;
|
||
- FULL/PIECEWISE/NONE shares and step duration by mode; and
|
||
- absolute values plus ratios to C00, never ratios alone.
|
||
|
||
Primary useful-token efficiency is
|
||
`E_token = sum(clean prefill_tokens + clean decode_tokens) /
|
||
sum(clean step_duration_ms)` in tokens/ms. End-to-end input/output tokens/s is
|
||
reported beside it. H1b's “5% worse efficiency” means
|
||
`1-E_irregular/E_control >= 0.05` using `E_token`.
|
||
|
||
The primary cross-pattern comparison is C00-TP1 at moderate `rho=0.60`.
|
||
C00 saturation is secondary. C10/C01/C11 estimate scheduler-factor and
|
||
interaction sensitivity on sentinels. C00-TP2 is descriptive parallelism
|
||
evidence; a single TP2 pattern does not support a scaling claim.
|
||
|
||
### Operator shares and bottleneck ranking
|
||
|
||
For operator family `f` in window `w` and rank `r`:
|
||
|
||
```text
|
||
share[f,w,r] = sum GPU kernel duration assigned to f
|
||
/ sum duration of all GPU kernel/replay activity
|
||
```
|
||
|
||
Each load point has two independent wall-separated windows. Rank families by
|
||
share within each window; ties within one percentage point receive the same
|
||
rank. TP2 reports each rank and their mean/range, not their sum.
|
||
|
||
The cross-pattern ranking-change test considers every pair of primary C00
|
||
patterns and every pair of operator families. An inversion is accepted only
|
||
when:
|
||
|
||
1. family A exceeds B by at least five percentage points in **both** windows of
|
||
pattern p;
|
||
2. B exceeds A by at least five percentage points in **both** windows of
|
||
pattern q;
|
||
3. a window-stratified permutation test over the 16 active steps per pattern,
|
||
seed 20260714 and 100,000 permutations, rejects equal ordering after Holm
|
||
correction across all tested pattern/family pairs at family-wise alpha 0.05;
|
||
4. at least 70% of kernel time is classifiable in all four windows.
|
||
|
||
Reproduction at saturation or in another server configuration is reported as
|
||
strengthening evidence but is not required: saturation deliberately removes
|
||
the arrival treatment, and only four patterns receive config variants. The two
|
||
fixed, wall-separated windows are the pre-declared replication unit for H1a.
|
||
|
||
Also report Kendall's tau-b between every pair of family-rank vectors. Tau is
|
||
descriptive and cannot replace the inversion rule.
|
||
|
||
### Waste accounting
|
||
|
||
All ratios are computed as ratios of sums, not means of per-step percentages.
|
||
|
||
**CUDA-graph padding.** For graph-hit steps with positive bucket `b_s` and
|
||
unpadded tokens `u_s`:
|
||
|
||
```text
|
||
padding_fraction = sum_s (b_s - u_s) / sum_s b_s
|
||
```
|
||
|
||
Also report padded tokens per useful scheduled token and the distribution of
|
||
`b_s-u_s`. NONE steps are excluded from padding and counted under graph miss.
|
||
|
||
**Graph miss and bucket mismatch.** Let `B` be the capture bucket set parsed
|
||
from the startup config and `b*(u)=min{b in B: b>=u}`.
|
||
|
||
```text
|
||
graph_miss_rate = count(model_executed and not hit) / count(model_executed)
|
||
eligible_miss_rate = count(model_executed and not hit and b*(u) exists)
|
||
/ count(model_executed)
|
||
overflow_rate = count(model_executed and not hit and b*(u) absent)
|
||
/ count(model_executed)
|
||
bucket_slack = sum(b*(u)-u) / sum(b*(u)) over eligible u
|
||
```
|
||
|
||
Report all four by pattern, config, load, and runtime mode.
|
||
|
||
**Sequence raggedness.** Partition the immutable arrival-order manifest into
|
||
consecutive complete cohorts of 64 requests. For input lengths `L_gi`:
|
||
|
||
```text
|
||
R64 = 1 - sum_g sum_i L_gi / sum_g (64 * max_i L_gi)
|
||
```
|
||
|
||
This is the exact padding fraction an offline rectangular batch-of-64 input
|
||
benchmark would incur on the same requests. The incomplete final cohort is
|
||
excluded. Report sensitivity at cohort sizes 32 and 128, but R64 is primary.
|
||
|
||
**Mixed-batch interference.** On clean Layer-1 steps, fit nonnegative robust
|
||
splines `F_p(P,N)` on pure-prefill steps and `F_d(D,N)` on pure-decode steps,
|
||
where `P`/`D` are prefill/decode tokens and `N` is scheduled requests. Let
|
||
`alpha` be median zero-token scheduler-step time. For a mixed step:
|
||
|
||
```text
|
||
t_add = F_p(P,N) + F_d(D,N) - alpha
|
||
I_mix = sum_mixed (t_observed - t_add) / sum_mixed t_add
|
||
```
|
||
|
||
Fits are config/load-specific and use all patterns, with leave-one-pattern-out
|
||
prediction for the reported cell. A mixed step outside the convex support of
|
||
both pure-step fits is excluded and counted. If fewer than 30 supported mixed
|
||
steps remain, `I_mix` is N/A rather than extrapolated. This is an interference
|
||
association, not a causal kernel ablation.
|
||
|
||
**MoE/ragged proxy.** Report coefficient of variation of MoE GEMM kernel
|
||
durations across layers within each active step:
|
||
`CV_moe = sd(duration_l)/mean(duration_l)`, only when layer scopes are
|
||
available for at least 80% of MoE GEMM time. Otherwise it is N/A. Exact expert
|
||
routes are not enabled because the server-wide routed-expert return path would
|
||
contaminate clean throughput.
|
||
|
||
### Statistical summaries
|
||
|
||
- Layer-1 and end-to-end intervals use a 5-second moving-block bootstrap over
|
||
clean time blocks, 100,000 resamples, seed 20260714.
|
||
- Layer-2 ranking uses the window-stratified permutation rule above. Active
|
||
steps are not presented as independent server replicates.
|
||
- The four Layer-1-only sentinel confirmations report run-level absolute and
|
||
relative deltas; no outlier is removed.
|
||
- Multiple ranking tests use Holm correction. Waste contrasts use Holm
|
||
correction within each waste family.
|
||
- No arithmetic mean of normalized scores is used. Cross-cell normalized
|
||
aggregates, if requested later, use a geometric mean and retain all cells.
|
||
|
||
### Decision rule
|
||
|
||
H1a passes if at least one inversion meets all four ranking criteria. H1b
|
||
passes if an irregular cell (P05, P06, P09, or P10) versus its declared
|
||
rectangular control has a corrected 95% contrast excluding zero and exceeds at
|
||
least one threshold:
|
||
|
||
- padding fraction: 5 percentage points;
|
||
- graph-miss or overflow rate: 10 percentage points;
|
||
- R64: 0.15 absolute;
|
||
- `I_mix`: 0.10; or
|
||
- `CV_moe`: 0.15;
|
||
|
||
and that contrast coincides with at least 5% worse useful-token efficiency or
|
||
positive step-time residual. The frozen control contrasts are P05 versus both
|
||
P01 and P03, P06 versus both P02 and P04, P09 versus both P01 and P03, and P10
|
||
versus both P03 and P04. Every listed contrast enters the same correction
|
||
family and is reported; none is selected afterward by the best p-value.
|
||
|
||
The compound hypothesis is **CONFIRMED** only when H1a and H1b pass. It is
|
||
**REFUTED / NULL SUPPORTED** only when the same top family holds in both
|
||
windows of every primary pattern, simultaneous 95% upper bounds on all family
|
||
share gaps are below five percentage points, and all H1b contrasts are below
|
||
threshold. Every other outcome is **INCONCLUSIVE** or **PARTIAL**.
|
||
|
||
## Artifact, privacy, and reproducibility contract
|
||
|
||
Each run directory contains only:
|
||
|
||
```text
|
||
manifest.sha256 # hash and non-sensitive length summary
|
||
commands.log # exact server/client/profile commands
|
||
environment.json # source, patch, wheel, driver, GPU, clocks
|
||
server.log
|
||
client/{segments.jsonl,result.json,sanity.json}
|
||
opprof/*.jsonl
|
||
traces/*.pt.trace.json.gz
|
||
analysis/{layer1.json,kernels.json,waste.json,sanity.json}
|
||
```
|
||
|
||
Synthetic manifests may be archived. P10's manifest, prompts, generated text,
|
||
and any response body are private inputs and are forbidden from this tree.
|
||
Before packaging, a scanner must reject any artifact containing a `prompt`,
|
||
`messages`, `content`, or generated-text field, or any source prompt substring.
|
||
The private path and manifest hash may appear in the exact command log; file
|
||
contents may not. Only counts, length distributions, hashes, and timings may
|
||
leave the private directory.
|
||
|
||
Record source/patched commit, patch checksums, Python/torch/CUDA/vLLM/msgspec
|
||
versions, driver, model path and shard hashes, GPU UUID(s), server effective
|
||
config, MoE backend, compile cache key, client/helper SHA, workload hash, clock
|
||
snapshots, host load, other-user process samples, and all raw analysis seeds.
|
||
|
||
## Budget and hard stop conditions
|
||
|
||
The matrix contains 24 pattern/config cells: 23 TP1 and one TP2. Each has two
|
||
load runs, for 48 primary measured runs. Four Layer-1-only confirmation runs
|
||
bring the measured total to **52**. Five unmeasured compile burn-ins are not
|
||
counted as pattern runs but are included in cost.
|
||
|
||
Using the P2 fixed-cache startup and profile endpoint timings, budget each
|
||
primary run as approximately:
|
||
|
||
```text
|
||
70 s startup + 60 s warm-up + 240 s clean
|
||
+ 2 * (24 s profiler + 30 s recovery) + 10 s shutdown = 488 s
|
||
```
|
||
|
||
Expected serialized wall time is about 7.1 hours. Accounting for two GPUs in
|
||
the two TP2 runs and its burn-in gives **about 7.5 H20-hours**, with a hard
|
||
campaign stop at 8.0 H20-hours. The four confirmation runs are budgeted at
|
||
380 seconds each. Queueing for a free GPU is not GPU time.
|
||
|
||
There are 96 logical Layer-2 windows. TP1 contributes 92 trace files; TP2 may
|
||
contribute eight rank files, for 100 expected trace files. At the P2 trace size
|
||
of 16.4 MB, Kineto uses about 1.6 GB. Layer 1, logs, client data, and analysis
|
||
should keep public artifacts below 4 GB. The private source plus derived P10
|
||
manifest is separately budgeted below 2 GB. Stop if public artifacts exceed
|
||
8 GB or CPFS free space falls below 100 GB.
|
||
|
||
Stop the matrix immediately and request review if any of these occurs:
|
||
|
||
1. source/patch/helper/manifest hash mismatch, effective config mismatch, or a
|
||
per-run output directory changes the compile-cache key;
|
||
2. selected GPU not idle, another user's GPU process appears, GPU OOM, server
|
||
error, or final GPU cleanup does not return to zero;
|
||
3. any clean-segment request fails, an output-token count differs from the
|
||
manifest, the moderate offered rate misses its target by more than 5%, or
|
||
drain exceeds 120 seconds;
|
||
4. Layer-1 schema/footer/step accounting fails, any record drops, or any
|
||
profile/recovery step enters a clean summary;
|
||
5. either trace is missing/unloadable, has other than 2+8 scheduled iterations,
|
||
lacks kernels, cannot join to Layer 1, or profiler composition has
|
||
`abs(SMD)>0.25` twice;
|
||
6. recovery does not meet its fixed 30-second criterion;
|
||
7. more than 30% of device kernel time is `other`/opaque in a primary window,
|
||
or fewer than eight active steps are available for a required mode segment;
|
||
8. a prompt, message, content, generated-text field, or source prompt substring
|
||
appears outside the private directory;
|
||
9. public disk or the 8.0 H20-hour cap is reached; or
|
||
10. a data-sanity red flag defined below appears.
|
||
|
||
No failed cell is silently replaced, retuned, filtered, or omitted. One exact
|
||
rerun is allowed only for an infrastructure/profile-artifact failure; both
|
||
attempts remain in the report. A semantic workload/config failure requires a
|
||
reviewed protocol amendment.
|
||
|
||
## Data sanity block contract
|
||
|
||
Every raw-run summary, per-cell analysis, aggregate table, figure source, and
|
||
final report must end with a machine-readable and rendered sanity block. For
|
||
**every numeric family** it reports `n`, finite count, missing count, minimum,
|
||
maximum, and distinct-value count. It also records expected constants rather
|
||
than treating them as suspicious.
|
||
|
||
At minimum, the block checks:
|
||
|
||
- non-negative request/token/queue/KV/kernel/duration counters;
|
||
- ratios in [0,1], except signed residual/interference metrics explicitly
|
||
identified as signed;
|
||
- exactly 240 clean seconds, three 80-second segments, two profiler windows per
|
||
load, 2+8 profiler iterations per window, and four logical windows per cell;
|
||
- continuous unique Layer-1 steps and balanced footer accounting with zero
|
||
drops;
|
||
- exact manifest hash, no wrap, exact requested/actual output work, zero clean
|
||
failures, and moderate rate within 5%;
|
||
- bucket >= unpadded, padding identity, graph-mode consistency, and waste
|
||
numerator <= denominator;
|
||
- trace loadability, positive kernel count, rank completeness, unambiguous
|
||
Layer-1 join, and at least 70% classifiable device time;
|
||
- input/output and arrival distributions match the frozen manifest;
|
||
- per-pattern and per-config outputs are not all identical unless the field is
|
||
a declared constant;
|
||
- clocks/load/GPU-process contamination and graceful zero-memory cleanup; and
|
||
- no private prompt/generated text in public artifacts; private paths are
|
||
allowed only in exact command logs.
|
||
|
||
Negative non-negative counters, ratios outside bounds, missing footers,
|
||
unexpectedly identical pattern/config results, discontinuous step/time series,
|
||
manifest drift, profiler leakage, or private-data leakage are red flags. Report
|
||
the anomaly first and stop analysis; do not build a conclusion on it.
|
||
|
||
## Final preregistration summary
|
||
|
||
The table below is normative. `P3C='python scripts/opprof_phase3_client.py'`,
|
||
`PRIVATE=/home/admin/cpfs/wjh/opprof-phase3-private/manifests`, and every
|
||
synthetic command uses the pinned model tokenizer. Each command is one logical
|
||
line even where Markdown wraps it.
|
||
|
||
### Pattern-cell table
|
||
|
||
| ID | Operational cell | Exact manifest-generator command |
|
||
|---|---|---|
|
||
| P01 | short-uniform input; short output; steady; no prefix | `$P3C materialize --id P01 --kind synthetic --input-uniform 128:512 --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P01.jsonl"` |
|
||
| P02 | short-uniform input; long output; steady; no prefix | `$P3C materialize --id P02 --kind synthetic --input-uniform 128:512 --output-fixed 512 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P02.jsonl"` |
|
||
| P03 | long-uniform input; short output; steady; no prefix | `$P3C materialize --id P03 --kind synthetic --input-uniform 4096:8192 --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P03.jsonl"` |
|
||
| P04 | long-uniform input; long output; bursty 8-at-once; no prefix | `$P3C materialize --id P04 --kind synthetic --input-uniform 4096:8192 --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P04.jsonl"` |
|
||
| P05 | 50/50 short/long bimodal input; short output; steady; no prefix | `$P3C materialize --id P05 --kind synthetic --input-mixture '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}' --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P05.jsonl"` |
|
||
| P06 | 50/50 short/long bimodal input; long output; bursty 8-at-once; no prefix | `$P3C materialize --id P06 --kind synthetic --input-mixture '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}' --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P06.jsonl"` |
|
||
| P07 | fixed 1,280-token input; long output; bursty 8-at-once; no prefix control | `$P3C materialize --id P07 --kind synthetic --input-fixed 1280 --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P07.jsonl"` |
|
||
| P08 | fixed 1,280-token input; long output; bursty 8-at-once; high prefix sharing | `$P3C materialize --id P08 --kind prefix-pool --num-prefixes 8 --prefix-len 1024 --suffix-fixed 256 --output-fixed 512 --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P08.jsonl"` |
|
||
| P09 | production-shaped 50/30/20 mixed input; short output; steady; no prefix | `$P3C materialize --id P09 --kind synthetic --input-mixture '{"uniform:128:512":0.5,"uniform:1024:2048":0.3,"uniform:4096:8192":0.2}' --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P09.jsonl"` |
|
||
| P10 | private real chat prompts; observed input <=32,768; output capped 256; steady; natural prefix | `$P3C materialize-private --id P10 --source /home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/chat_w20260311_1000.jsonl --sampling-u-max 0.125 --max-input-tokens 32768 --output-cap 256 --preserve-prompts --disable-shuffle --arrival steady --out "$PRIVATE/P10.jsonl"` |
|
||
| P11 | short-uniform input; long output; bursty 8-at-once; no prefix | `$P3C materialize --id P11 --kind synthetic --input-uniform 128:512 --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P11.jsonl"` |
|
||
|
||
### Total run count and GPU-hour estimate
|
||
|
||
- Pattern cells: **11**.
|
||
- TP1 pattern/config cells: **23** — C00 for all eleven, plus C10/C01/C11
|
||
for P01/P03/P06/P10.
|
||
- TP2 counterpoint cells: **1** — P10/C00-TP2.
|
||
- Primary runs: **48** — saturation and moderate for each of 24 cells.
|
||
- Layer-1-only confirmation runs: **4**.
|
||
- Total measured runs: **52**, plus five unmeasured compile burn-ins.
|
||
- Layer-2 windows: **96 logical windows**, 2+8 iterations each; 100 expected
|
||
rank trace files.
|
||
- Fixed throughput-accounted time: **240 seconds per measured run**.
|
||
- Expected cost: **about 7.1 hours serialized wall time and 7.5 H20-hours**;
|
||
hard stop at 8.0 H20-hours.
|
||
- Expected public artifact volume: approximately 2.5–4 GB; hard stop at 8 GB.
|
||
|
||
### Open decisions for the orchestrator
|
||
|
||
1. **Blocking:** approve the fixed-duration helper interface/algorithm and a
|
||
no-GPU implementation-and-test turn before execution. The existing bundled
|
||
client cannot enforce this protocol's fixed clean duration and profile masks.
|
||
2. **Blocking:** approve copying the sensitive P10 source to the mode-0700
|
||
dash0 private directory, its `sampling_u<=0.125`, input<=32,768 selection,
|
||
and 256-token output cap. Rejection requires a protocol amendment; P10 will
|
||
not be silently replaced.
|
||
3. Approve the sparse scheduler design: MNS 64/default 1024 and MBT
|
||
2048/default 8192, full factorial only on P01/P03/P06/P10, chunked prefill
|
||
always on.
|
||
4. Approve P10/C00 as the single TP2 counterpoint rather than adding TP4 or a
|
||
second pattern.
|
||
5. Approve normalized moderate load `rho=0.60`, deterministic eight-request
|
||
bursts, 240 clean seconds, and exactly four Layer-2 windows per cell.
|
||
6. Approve the mapping priority, 70% classifiability gate, materiality
|
||
thresholds, and confirm/refute/inconclusive decision rule.
|
||
|
||
### Protocol sanity block
|
||
|
||
| Numeric family | n | Min | Max | Distinct | Invariant/result |
|
||
|---|---:|---:|---:|---:|---|
|
||
| Pattern rows | 11 | P01 | P11 | 11 | Within required 8–12 range |
|
||
| Server configs | 5 | TP1/MNS64/MBT2048 | TP2/MNS1024/MBT8192 | 5 | Four TP1 factorial configs plus one TP2 counterpoint |
|
||
| TP size | 5 configs | 1 | 2 | 2 | TP1 primary; exactly one TP2 cell |
|
||
| MNS level | 5 configs | 64 | 1024 | 2 | Small/default only |
|
||
| MBT level | 5 configs | 2048 | 8192 | 2 | Small/default only; chunking always on |
|
||
| Pattern/config cells | 24 | 1 config/pattern | 5 configs for P10 incl. TP2 | 3 counts | Per-pattern config counts are 1, 4, or 5; 23 TP1 + 1 TP2 |
|
||
| Primary load runs | 48 | 2/cell | 2/cell | 1 | Saturation plus moderate for every cell |
|
||
| Confirmation runs | 4 | 1/sentinel | 1/sentinel | 1 | P10/P06/P03/P01 moderate C00, Layer-1 only |
|
||
| Total measured runs | 1 aggregate | 52 | 52 | 1 | `48+4=52` |
|
||
| Clean duration/run | 52 | 240 s | 240 s | 1 expected | Three exact 80-second segments |
|
||
| Logical Layer-2 windows/cell | 24 | 4 | 4 | 1 expected | Two per load point |
|
||
| Logical Layer-2 windows | 1 aggregate | 96 | 96 | 1 | `24*4=96` |
|
||
| Active profiler iterations | 1 aggregate | 768 | 768 | 1 | `96*8=768`; every window also has two warm-ups |
|
||
| Expected rank trace files | 1 aggregate | 100 | 100 | 1 | 92 TP1 plus 8 TP2 rank files |
|
||
| Local real-trace rows | 1 source | 32,606 | 32,606 | 1 | Prompt contents never printed or copied in this turn |
|
||
| Frozen P10 selected rows | 1 subset | 4,011 | 4,011 | 1 | Selection reproduced locally without prompt output |
|
||
| P10 selected input tokens | 4,011 | 69 | 32,704 | 3,424 | Sum 36,499,001; median 7,009; p95 25,629 |
|
||
| P10 capped output tokens | 4,011 | 2 | 256 | 236 | Sum 878,291; median/p95 both 256 |
|
||
| Estimated H20-hours | 1 budget | 7.5 | 7.5 | 1 | Positive and below hard cap 8.0 |
|
||
| Phase-3 GPU/SSH actions this turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied |
|
||
|
||
Checked arithmetic invariants: `11+4*3=23` TP1 cells; adding one TP2 cell gives
|
||
24; two load points give 48 primary runs; four confirmations give 52 measured
|
||
runs; `24*4=96` logical profiler windows; TP2 adds four extra rank files, giving
|
||
100. Ratios and thresholds are within their declared domains. Expected constants
|
||
are labeled. No Phase 3 GPU run, SSH action, trace transfer, helper
|
||
implementation, patch change, or prompt disclosure occurred in this turn.
|