Report held-out active intervention result

Entry-point summary of both closed campaigns: verdicts, corrected
readings (P5 supersedes P3's P10-vs-P04 magnitude), methodological
findings (co-location SLO validity, arrival uniformization, compile-
factor env poisoning), artifact map, and GPU accounting.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.gitignore

@@ -14,3 +14,8 @@ configs/examples/dash0_llm_10min_study_run[0-9]*.json
 configs/examples/dash0_smoke_study_run*.json
 infra/gpu_fleet/config/fleet.toml
 infra/gpu_fleet/config/jobs.toml
+# Raw Layer-1 telemetry streams (507 MB; decision-bearing JSON/log evidence stays tracked)
+runs/**/*.jsonl
+.ruff_cache/
+# Recovered dash1 interaction-run stores (100 MB raw tune logs, kept on disk only)
+recovered-stores/

docs/action-aware-constraint-pilot-v0-protocol-20260714.md

@@ -0,0 +1,179 @@
+# Action-aware constraint pilot v0 protocol
+
+Status: **FROZEN BEFORE NEW GPU RUNS**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Headline question
+
+Can telemetry from one complete initial-config benchmark identify which of two
+competing knob families should be changed, before either target configuration
+is evaluated?
+
+This pilot tests a narrow prerequisite, not an end-to-end tuner claim.  It
+uses fields already present in the per-step OpProf stream to reconstruct exact
+zero-slack conditions for `max_num_seqs` (MNS) and
+`max_num_batched_tokens` (MBBT).  No new vLLM instrumentation is justified
+unless those action-conditioned conditions predict crossed real-system
+intervention responses.
+
+## Hypothesis
+
+I believe config-normalized scheduler constraints provide a stronger tuning
+signal than an aggregate queue symptom because the same waiting queue can be
+blocked by different admission limits.
+
+I will verify it by holding model, hardware, TP, request bands, arrival times,
+and offered load fixed while constructing two source configurations with
+different binding constraints.  From each source run alone, the larger
+exclusive binding fraction predicts the action family.  Both candidate
+actions are then measured on the same requests for the full 300-second replay.
+
+## Frozen platform and workload
+
+- Host: `dash0`, solo placement on GPUs 0-3, four NVIDIA H20 GPUs.
+- Model: Qwen3-30B-A3B BF16.
+- Engine: patched vLLM `0.24.1.dev3+opprof`, TP=4.
+- Workload: the three disjoint `mid` bands from
+  `chat_w20260312_1000`, 2.125 requests/s/GPU, 300-second arrival window,
+  exactly 128 output tokens.
+- SLO: the unchanged study TTFT/TPOT thresholds and 0.95 pass-rate target.
+- Every config starts one fresh server, performs the accepted 16-request
+  warm-up and the existing burn-in, then runs all three disjoint measured
+  bands in its frozen order.
+- SLO early stopping is disabled.  A measured run must drain all selected
+  requests and finish within the 450-second client deadline.
+
+## Frozen configuration and action matrix
+
+| ID | MNS | MBBT | Role |
+|---|---:|---:|---|
+| `b_base` | 64 | 256 | token-budget-bound source; operational gate runs first |
+| `a_base` | 16 | 8192 | MNS-bound source |
+| `shared` | 64 | 8192 | MNS action from A; MBBT action from B |
+| `b_mns` | 128 | 256 | competing MNS action from B |
+| `a_mbbt` | 16 | 16384 | competing MBBT action from A |
+
+The two decisions are therefore:
+
+```text
+Regime A: a_base -> {shared (increase MNS), a_mbbt (increase MBBT)}
+Regime B: b_base -> {b_mns (increase MNS), shared (increase MBBT)}
+```
+
+The candidate magnitudes are intentionally large in this feasibility pilot so
+that a missing crossed response is not explained by an imperceptibly small
+intervention.  This does not establish that these are production step sizes.
+
+Frozen config order is `b_base`, `a_base`, `shared`, `b_mns`, `a_mbbt`.
+Frozen repetition orders are respectively `123`, `231`, `312`, `132`, and
+`213`, reducing band/time alignment without reusing a server across configs.
+
+## Pre-action signal
+
+For each source run, let `waiting` include the normal and deferred waiting
+queues, and let `scheduled_tokens = prefill_tokens + decode_tokens`.
+
+```text
+mns_exclusive = waiting > 0
+                and running == configured MNS
+                and scheduled_tokens < configured MBBT
+
+mbbt_exclusive = waiting > 0
+                 and scheduled_tokens == configured MBBT
+                 and running < configured MNS
+
+both = waiting > 0
+       and running == configured MNS
+       and scheduled_tokens == configured MBBT
+```
+
+Each score is the fraction of all scheduler records in the measured interval
+that satisfies the condition.  The predicted action is the family with the
+larger exclusive fraction.  This uses no target telemetry or target outcome.
+KV usage and preemptions are reported as possible alternative constraints but
+are not silently reassigned to either score.
+
+These conditions reproduce two scheduler loop boundaries, but they are still
+a Level-0 proxy: they do not expose the exact request rejected at the boundary
+or run a shadow schedule.  The pilot explicitly tests whether that additional
+engine patch is warranted.
+
+## Outcomes and baselines
+
+Primary intervention outcome:
+
+```text
+SLO-goodput = full-run SLO pass count / 300-second arrival window
+```
+
+Also report pass rate, TTFT p50/p95/p99, TPOT p50/p95/p99, drain elapsed time,
+KV usage, preemptions, queue area, and CUDA-graph padding.
+
+Required decision baselines:
+
+1. always choose the MNS family;
+2. always choose the MBBT family;
+3. queue-pressure-only, which has no candidate-specific score and therefore
+   must use one frozen family for both regimes;
+4. the pre-action exclusive-binding prediction.
+
+This is a mechanism ablation.  It does not compare against a trained black-box
+tuner because two regimes are not a valid training surface.
+
+## Gates and failure meanings
+
+Data validity requires 15 uncensored measured runs, exact request/arrival/input
+hashes across each repetition, full request accounting, one continuous OpProf
+stream per config, zero dropped records, monotonic timestamps and step indices,
+nonnegative counters, bounded ratios, clean GPU placement, and config values in
+the result matching the server command.
+
+The crossed-response gate passes only if, in all three repetitions:
+
+- the MNS target has higher SLO-goodput than the MBBT target in Regime A;
+- the MBBT target has higher SLO-goodput than the MNS target in Regime B;
+- each winning target exceeds its competing target by at least 10% of the
+  source SLO-goodput.  A source with zero goodput makes the run invalid for
+  this relative gate rather than changing the denominator.
+
+The binding gate passes only if, in both regimes:
+
+- the predicted family matches the measured winning family in all three
+  repetitions;
+- the median winning-family exclusive fraction is at least 0.10;
+- it is at least 5x the median competing-family exclusive fraction;
+- the direction is unchanged under cumulative 25%, 50%, 75%, and 100%
+  checkpoints after the 25% checkpoint.
+
+Decision meanings:
+
+- `STOP_WORKLOAD_NOT_CROSSED`: candidate outcomes do not have different
+  winners; the experiment cannot test action selection.
+- `STOP_BINDING_NOT_PREDICTIVE`: outcomes cross but source-only constraint
+  scores do not select them; do not implement shadow scheduling from this
+  hypothesis.
+- `STOP_NO_NEW_INSTRUMENTATION_NEEDED`: the signal works but every required
+  field was already present; keep it as an analysis/tuner feature and do not
+  claim a new engine-instrumentation contribution.
+- `OPEN_EXACT_ATTRIBUTION_ABLATION`: the signal works but unresolved/both/KV
+  cases are material enough that exact rejection reasons could change a
+  decision.  Only this result authorizes a minimal vLLM attribution patch.
+
+Ambiguity is material only when, in either regime, the median
+`both + waiting_unresolved` fraction is at least the median absolute gap
+between the two exclusive fractions, or when any source run records a
+preemption or median source KV maximum is at least 0.90.  Otherwise all fields
+needed for the observed decision were already present and the result is
+`STOP_NO_NEW_INSTRUMENTATION_NEEDED`.
+
+No result from this development pilot is a paper-level E2E tuning claim.
+
+## Cost and stopping discipline
+
+- Hard cap: 8.0 H20-hours, including failed sessions.
+- Expected: 6.0-7.2 H20-hours and 90-110 minutes wall time.
+- `b_base` runs first.  If its first measured band cannot drain by 450 seconds,
+  the controller stops before any comparative analysis; MBBT=256 is then an
+  operationally invalid source, not negative evidence.
+- Any data red flag stops analysis before computing a tuning conclusion.

docs/action-aware-constraint-pilot-v1-protocol-20260714.md

@@ -0,0 +1,44 @@
+# Action-aware constraint pilot v1 amendment
+
+Status: **FROZEN AFTER V0 OPERATIONAL STOP AND BEFORE V1 GPU RUNS**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+The complete claim, workload, baselines, metrics, action matrix, analysis gates,
+and data-validity requirements remain those in the v0 protocol.  This amendment
+changes only the token-bound source severity and adds an operational burn-in
+gate.
+
+## Why v0 produced no comparative evidence
+
+The first v0 session used MNS=64 and MBBT=256.  During the 510-request,
+60-second burn-in, the client had run for 197 seconds and the engine still held
+64 requests: 13 running and 51 waiting.  The last step scheduled exactly 256
+tokens, KV usage was 0.01151, and there were zero preemptions.  No measured run
+or target configuration had started.
+
+The session was stopped and cleanly released all GPUs after consuming
+0.3859868995 H20-hours.  This is evidence that MBBT=256 is a real token-budget
+bottleneck, but it is not an admissible source for the 2.125 requests/s/GPU
+comparison because it cannot sustain the offered load.  V0 contributes no
+tuning label and none of its runtime data is reused by V1.
+
+Authoritative failure artifact:
+`/home/admin/cpfs/wjh/action-aware-constraint-v0-20260714/operational-stop-v0.json`.
+
+## V1 changes
+
+- `b_base`: MNS=64, MBBT **2048** instead of 256.
+- `b_mns`: MNS=128, MBBT **2048** instead of 256.
+- The B-family MBBT action is therefore 2048 -> 8192.
+- All five configurations and all three repetitions run fresh under a new V1
+  run root.
+- Before any measured run, every config's 510-request/60-second burn-in must
+  drain in at most **90 seconds**.  A slower config is an operational failure;
+  the controller stops before comparative analysis.
+- V1 incremental hard cap is 7.6140131005 H20-hours so that V0 plus V1 remains
+  within the original global 8.0 H20-hour cap.
+
+The V0 protocol's crossed-response and source-only binding gates are unchanged.
+In particular, V1 still requires three-of-three action-family predictions in
+both regimes and a different real winner in Regime A versus Regime B.

docs/action-aware-constraint-pilot-v2-protocol-20260714.md

@@ -0,0 +1,35 @@
+# Action-aware constraint pilot v2 amendment
+
+Status: **FROZEN AFTER V1 CONTROLLER STOP AND BEFORE V2 GPU RUNS**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+The V1 configuration matrix and every scientific gate remain unchanged.  V2
+fixes one controller bug and reruns every configuration and repetition fresh.
+
+## V1 controller failure
+
+The MNS64/MBBT2048 burn-in completed all 510 requests in 61.259 seconds, below
+the frozen 90-second operational limit.  However, the controller accidentally
+assigned the preceding 16-request warm-up result to `burnin_result`; its state
+therefore recorded 4.376 seconds and evaluated the wrong object.
+
+The first measured replay was terminated after 75 seconds, before it produced
+a result.  No target configuration had started.  V1 consumed
+0.3184109431 H20-hours and contributes no action label or telemetry to V2.
+
+## V2 correction and regression gate
+
+- The completed burn-in `run_client()` return value is assigned to
+  `burnin_result`.
+- A dedicated `burnin_gate()` rejects any non-anchor object, any request count
+  other than 510, and elapsed time above 90 seconds.
+- Unit tests explicitly pass a warm-up object and require rejection, then test
+  accepted and over-limit burn-ins.
+- All five configs and 15 measured runs use a new run root; no V0/V1 runtime
+  artifact is reused.
+
+V0 and V1 together consumed 0.7043978426 H20-hours.  V2's incremental hard cap
+is therefore 7.2956021574 H20-hours, preserving the original global 8.0
+H20-hour cap.  The authoritative accounting file is
+`runs/action-aware-v0/prior-attempts-v2.json`.

docs/action-aware-constraint-pilot-v2-results-20260714.md

@@ -0,0 +1,260 @@
+# Action-aware constraint pilot v2 results
+
+Date: 2026-07-14 (Asia/Singapore).
+
+Decision: **`STOP_WORKLOAD_NOT_CROSSED`**.
+
+The pilot produced one valid positive regime and one invalid-for-effect-size
+regime.  It supports continuing a narrower action-response investigation, but
+it does not justify an end-to-end telemetry-guided tuner claim or new engine
+instrumentation yet.
+
+## Question tested
+
+Given only a completed source run, can existing engine telemetry distinguish
+which of two one-knob interventions will improve SLO-goodput more?
+
+The frozen score counted scheduler steps with backlog where either MNS or MBBT
+was exclusively at its configured limit.  It made two pre-intervention
+predictions on the same workload and offered load:
+
+- Regime A: source `(MNS=16, MBBT=8192)` predicts increasing MNS to 64 over
+  increasing MBBT to 16384.
+- Regime B: source `(MNS=64, MBBT=2048)` predicts increasing MBBT to 8192 over
+  increasing MNS to 128.
+
+The primary outcome was 300-second SLO-goodput.  A predicted action had to beat
+the alternative on every paired request band by at least 10% of that band's
+source goodput.  Telemetry direction also had to remain stable at 25%, 50%,
+75%, and 100% of the replay.
+
+## Setup
+
+- Host: `dash0`, GPU 0-3 used exclusively; four NVIDIA H20 GPUs; TP=4.  GPU
+  4-7 remained idle to avoid co-location effects.
+- Model: Qwen3-30B-A3B BF16 at
+  `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`.
+- Runtime: patched vLLM `0.24.1.dev3+g668cfb7e2`, source commit `4b253fd`, with
+  OpProf Layer-1 telemetry.
+- Workload: `chat_w20260312_1000`, 2.125 requests/s/GPU, 300-second arrival
+  window, 128 output tokens, three disjoint request bands.
+- Five fresh-server configurations, three measured runs each, counter-rotated
+  repetition order, 16-request warm-up, and a 510-request/60-second burn-in.
+  SLO early stopping was disabled.
+- Exact request-id, arrival-order, and input-length hashes matched for every
+  paired comparison.
+
+The authoritative run root is
+`/home/admin/cpfs/wjh/action-aware-constraint-v2-20260714`.  The final audit is
+`pilot-audit-final.json`, SHA256
+`7ebe080fcc4970bef423bc587253d157e75aed1ea8b410bd37770c17708135ab`.
+
+## End-to-end result
+
+### Regime A: strong MNS constraint
+
+| Rep | Source 16/8192 | MBBT action 16/16384 | MNS action 64/8192 | MNS-only source steps | MBBT-only source steps | `(MNS action - MBBT action) / source` |
+|---:|---:|---:|---:|---:|---:|---:|
+| 1 | 4.710 | 7.687 | 8.500 | 79.85% | 0.038% | 17.27% |
+| 2 | 2.803 | 4.150 | 8.500 | 94.61% | 0.005% | 155.17% |
+| 3 | 2.227 | 3.943 | 8.500 | 79.87% | 0.005% | 204.64% |
+
+Units are SLO-goodput requests/s except the percentage columns.  The source
+prediction was stable at every phase checkpoint and correct in all three
+paired bands.  The predicted MNS action cleared the frozen 10% material-margin
+gate in all three bands.
+
+This is a real but limited positive result.  The source was an extreme case:
+MNS was full on nearly every scheduler step that retained backlog, TTFT p50 was
+1.64-5.61 seconds, KV usage remained below 2.45%, and no preemption occurred.
+An expert or a simple rule could identify this case without a learned tuner.
+
+### Regime B: MBBT direction at an outcome ceiling
+
+| Rep | Source 64/2048 | MNS action 128/2048 | MBBT action 64/8192 | MBBT-only source steps | `(MBBT action - MNS action) / source` |
+|---:|---:|---:|---:|---:|
+| 1 | 8.423 | 8.420 | 8.500 | 10.90% | 0.950% |
+| 2 | 8.447 | 8.500 | 8.500 | 12.03% | 0% |
+| 3 | 8.500 | 8.497 | 8.500 | 8.79% | 0.039% |
+
+The telemetry direction was stable in all phase checkpoints.  The predicted
+action won twice and tied once, while the wrong MNS action left the MBBT-only
+state intact.  However, the source already delivered 99.10-100% of the offered
+8.5 requests/s.  Even a perfect action could not reach the preregistered 10%
+margin.  Regime B therefore does not test material weak-signal value; it is a
+workload-selection failure, not evidence that telemetry does or does not help
+near a decision boundary.
+
+The missing preflight condition is mathematical.  For an effect threshold
+`delta` and offered goodput ceiling `G`, the source must satisfy
+`source <= G / (1 + delta)`.  Here `G=8.5` and `delta=0.10`, so any source above
+7.727 requests/s cannot possibly pass before either target is measured.
+
+## Why the exclusive-limit rule is incomplete
+
+The alternative MBBT action improved Regime A by 48.0%, 63.2%, and 77.1% over
+the source even though MBBT was almost never the exclusive backlog constraint.
+This rules out the binary interpretation "a knob that is not exclusively at
+its cap cannot help."
+
+Existing richer telemetry provides a plausible mechanism:
+
+| Rep | Split-prefill requests, source -> MBBT action | Prefill steps, source -> MBBT action | Prefill requests/step, source -> MBBT action | Prefix-hit rate, source -> MBBT action |
+|---:|---:|---:|---:|---:|
+| 1 | 41 -> 2 | 2324 -> 2022 | 1.071 -> 1.249 | 13.851% -> 13.747% |
+| 2 | 7 -> 0 | 2410 -> 2291 | 1.012 -> 1.076 | 13.078% -> 12.988% |
+| 3 | 12 -> 1 | 2377 -> 2270 | 1.018 -> 1.096 | 13.613% -> 13.604% |
+
+Increasing MBBT allows more prefill work to be packed into one iteration and
+nearly eliminates split prefills.  Under MNS=16, this can reduce the number of
+iterations for which long prompts occupy scarce running slots.  Prefix-cache
+hit rates differ by at most 0.104 percentage points, and exact workload hashes
+match, so neither explains the gain.  Step-duration p99 also remains similar;
+one 1.127-second decode-step outlier appears in `a_mbbt/rep2`, but the same
+action direction occurs in all three bands.
+
+This is a mechanism-consistent explanation, not a completed causal
+decomposition.  MBBT simultaneously changes total per-iteration token budget,
+per-request chunk size, and multi-request packing.  Instrumentation observes
+their joint response but cannot separate those effects without another
+intervention.
+
+## Instrumentation decision
+
+Do **not** add a new engine patch for this mechanism yet.  The existing OpProf
+stream already records submit/complete timestamps, prefill/decode composition,
+chunked-prefill categories, prefix hits, queues, KV usage, and CUDA graph mode.
+Those fields are sufficient to identify the interaction missed by the initial
+exclusive-limit rule.
+
+The next narrow mechanism ablation is available in the current vLLM runtime:
+
+1. `(MNS=16, MBBT=8192, long-prefill-threshold=0)` is the current source.
+2. `(MNS=16, MBBT=16384, long-prefill-threshold=8192)` keeps individual long
+   chunks at 8192 while increasing total packing budget.
+3. `(MNS=16, MBBT=16384, long-prefill-threshold=0)` is the current MBBT action.
+
+The runtime exposes `--long-prefill-token-threshold`; with a threshold of 8192,
+the second arm separates total packing headroom from the larger per-request
+chunk allowed by the third arm.  A formal test must rerun all three arms fresh
+with counter-rotated order rather than reuse today's endpoints.
+
+## Correct tuning-research route
+
+The pilot does not support turning the frozen equality checks into a larger
+rule tree.  The supported route is **intervention-calibrated, action-conditioned
+system identification**:
+
+```text
+source event sequence + normalized config delta
+    -> predicted distribution of Delta SLO-goodput and evaluation cost
+    -> uncertainty-aware next-config selection
+```
+
+The policy input should retain continuous distributions and phase evolution:
+queue/running residency, MNS and token slack, prefill/decode composition,
+partial-prefill occupancy, step time, KV state, and graph behavior.  Human
+bottleneck labels and hand-authored `if queue then increase MNS` mappings are
+not policy inputs.  Mechanism summaries remain audit and interpretation tools;
+the action response is learned from paired real interventions.
+
+The harness has a narrower, non-heuristic role:
+
+- define legal configurations and exact paired workloads;
+- reject source points without outcome headroom before a full sweep;
+- randomize/counter-rotate execution order and preserve failures/cost;
+- validate stream coverage, hashes, request accounting, and censoring;
+- expose target outcomes only after a source-only prediction is frozen;
+- evaluate fixed-budget regret and H20-hours, not explanation quality alone.
+
+The next tuning experiment should use non-extreme, non-ceiling source points
+and a local two-dimensional MNS/MBBT neighborhood.  A short run may screen load
+only; every inferential telemetry and outcome result remains a 300-second run.
+At least one held-out workload must be reserved before choosing features or
+thresholds.
+
+Primary evaluation is H20-hours/trials to reach 95% of the real local oracle
+and cost-normalized regret AUC.  Required baselines are random search,
+config/outcome-only sequential search, the current rule heuristic, and the
+same action-response model with telemetry removed.  Action-ranking accuracy is
+supporting evidence only.
+
+## What this pilot establishes and does not establish
+
+Established:
+
+- Long-window engine state can make a correct, phase-stable action-family
+  prediction in an extreme MNS-constrained regime.
+- A naive `queue > 0 -> increase MNS` rule would choose an ineffective action
+  in Regime B; action-conditioned state distinguishes the mechanism direction,
+  although the measured effect is immaterial at the selected load.
+- Binary exclusive-cap attribution misses a substantial MNS/MBBT interaction;
+  existing chunk/step telemetry reveals a plausible explanation.
+- Source outcome headroom must be an explicit experiment admission gate.
+
+Not established:
+
+- telemetry improves an end-to-end tuner over an outcome-only baseline;
+- weak or mixed constraints can be ranked with material gain;
+- the response transfers across workloads, models, TP, or hardware;
+- new engine instrumentation is necessary;
+- the chunking/packing breakdown is causal rather than mechanism-consistent.
+
+## Change and verification
+
+Reproduction:
+
+```bash
+python3 runs/action-aware-v0/test_pilot.py
+
+python3 runs/action-aware-v0/analyze_pilot.py \
+  --run-root /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/runs/pilot \
+  --manifest runs/action-aware-v0/pilot-manifest-v2.json \
+  --output /home/admin/cpfs/wjh/action-aware-constraint-v2-20260714/pilot-audit-final.json
+```
+
+The fresh GPU run used AITuner commit `c5ab073`; asynchronous coverage was
+corrected in `3facb18`; reproducible mechanism summaries were added in
+`2af22db`.  The raw run is unchanged across those analyzer-only commits.
+
+Change: added a crossed real-intervention controller and audit, fixed the
+burn-in result gate, corrected asynchronous per-step coverage accounting, and
+added reproducible step/chunk/prefix mechanism summaries.
+
+Expected effect: distinguish descriptive telemetry from source-only action
+predictions that survive paired real interventions.
+
+Verification: local and remote action-aware test suites pass; all five sessions
+completed; all stream/footer and request-accounting invariants pass; the final
+analyzer was run twice and produced byte-identical output.
+
+Result: Regime A passes; Regime B is invalid for the frozen effect-size test;
+the global decision is `STOP_WORKLOAD_NOT_CROSSED`.
+
+Remaining risk: one model, one TP, one trace family, three bands, two action
+families, and deliberately constructed endpoints are development evidence
+only.  The strong positive regime is too obvious to support a paper claim.
+
+## Data sanity
+
+- Measured runs: n=15; elapsed 300.610-317.350 seconds; 15 distinct.  Pass
+  rate min/max 0.2620/1.0 with 11 distinct values; SLO-goodput min/max
+  2.2267/8.5 requests/s with 11 distinct values.
+- Telemetry intervals: n=15; records min/max 13,621/23,711; 15 distinct.
+  Start gaps min/max 0.0412/0.1227 seconds; end gaps 0.00053/0.0649; uncovered
+  internal gaps 0/0.3231.  One submit gap reached 1.1190 seconds but was fully
+  covered by a 1.1269-second recorded execution; contiguous indices and zero
+  drops were preserved.
+- Sessions: n=5; cost min/max 1.1691/1.2730 H20-hours; 5 distinct.  V2 cost
+  was 6.0862 H20-hours; V0/V1/V2 total was 6.7906, below the 8.0 cap.
+- Regime-A split-prefill observations: n=6; min/max 0/41 requests; 6 distinct.
+  Prefix-hit rates: n=6; min/max 0.12988/0.13851; 6 distinct.
+- Checked invariants: non-negative counters and durations; ratios in `[0,1]`;
+  exact request, arrival, and length hashes; 2550/2550 request accounting per
+  measured run; uncensored outcomes; outcomes across configurations not all
+  identical;
+  five complete streams; monotonic timestamps; contiguous step indices; zero
+  drops; footer/sidecar agreement; chunk-token accounting; bounded prefix hits;
+  no OOM, controller error, or residual GPU allocation.  No unresolved red
+  flag remains.  The three identical shared goodputs are reported as the
+  offered-load ceiling, not treated as independent performance variation.

docs/active-intervention-v0-protocol-20260715.md

@@ -0,0 +1,136 @@
+# Active intervention + measurement v0 protocol
+
+Date: 2026-07-15 (Asia/Singapore)
+
+Status: **FROZEN BEFORE THE `chat_w20260313_1000` GPU RUN**.
+
+## Research question
+
+This experiment asks whether a tuner conditioned on direct engine-state
+trajectories can choose both a measurement horizon and a coupled configuration
+intervention with lower real-GPU cost than the same tuner using only external
+prefix outcomes.
+
+The contribution is not the controller, legality checks, telemetry collection,
+or the ridge model.  The route remains open only if engine state changes an
+actual decision and reduces cost-to-near-oracle on unseen workloads.
+
+## Development result that motivates, but does not pass, the route
+
+The frozen trace-12 dataset contains 72 examples: six source decisions, four
+measurement checkpoints, and `noop/MNS/MBBT` actions.  Features are direct
+continuous Layer-1 state summaries; cap-exclusive and bottleneck labels are
+excluded.  Leave-one-repetition-out sequential replay uses the same model,
+candidate set, confidence rule, and checkpoint set for both modes.
+
+The external-outcome policy and telemetry policy both put all six decisions
+within 2% regret.  Outcome-only selected a mean 262.5-second source measurement
+and cost 3.750 replay H20-hours across the six replayed decisions; telemetry
+selected 275 seconds and cost 3.833 H20-hours.  Telemetry therefore increased
+the replay lower-bound cost by 2.22%, with no regret reduction.  This is a
+negative result.  It does not settle the question because the dataset has only
+two source regimes, one source is at the offered ceiling, and there is no joint
+MNS+MBBT action.
+
+Sanity: n=6 decisions; regret min=0, max=0.009412, distinct=3; source cutoff
+min=150s, max=300s, distinct=3 across the two policies; all costs are
+non-negative, regrets are in `[0,1]`, target results are not all identical, and
+the six decisions are complete exact-workload pairs.
+
+## Frozen prospective setup
+
+- Host: `dash0`, 8 NVIDIA H20 GPUs available; each TP4 server runs alone on
+  GPUs 0-3.  Co-location is prohibited for SLO verdicts.
+- Engine: patched vLLM `0.24.1.dev3+g668cfb7e2` from clean source commit
+  `4b253fd8619764b6971a7f2e3a3aa7545f6ace05` at
+  `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0`, using
+  `/tmp/wjh-opprof-phase2-dash0-20260711/.venv`.
+- Model: `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, BF16.
+- Workload: unseen `chat_w20260313_1000`; input 0-8192; output exactly 128;
+  replay scale 0.5; 300-second arrival window.
+- Three disjoint repetitions: source rows are assigned by a deterministic
+  SHA-256 modulo-3 partition before input filtering.  Each repetition selects
+  approximately 3300 requests, or 2.75 requests/s/GPU at TP4.
+- SLO: at least 95% pass; stepped TTFT 2/4/6 seconds; TPOT at most 50 ms.
+- Checkpoints: 75, 150, 225, and 300 seconds.
+- Full 2x2 surface:
+  - source: `MNS=32, MBBT=4096`;
+  - MNS action: `64,4096`;
+  - MBBT action: `32,8192`;
+  - joint action: `64,8192`;
+  - `noop` retains the source.
+- Four config sessions are serialized.  Each session uses a fresh server,
+  warm-up, burn-in, and counter-rotated repetition order.
+- Expected campaign cost: 4.6-5.5 H20-hours; hard cap: 6.0 H20-hours;
+  expected wall time: 75-100 minutes.
+
+The source is executed first.  The frozen telemetry policy selects the next
+real config session; all remaining cells are then measured only to construct
+the exact finite-surface oracle.  Oracle annotation after the selected action
+is reported separately from tuner cost.
+
+## Frozen policies
+
+Both policies fit the paired treatment effect
+
+```text
+target normalized SLO-goodput - source normalized SLO-goodput
+```
+
+from source config, full config delta, offered load, and external prefix
+outcomes.  The telemetry policy additionally receives fixed direct Layer-1
+summaries and their interactions with `delta_log2(MNS)` and
+`delta_log2(MBBT)`.  It does not receive a bottleneck label or a
+diagnosis-to-knob rule.
+
+At each checkpoint, jackknife models produce an effect distribution for
+`noop`, MNS, MBBT, and joint actions.  Measurement stops at the earliest second
+consecutive checkpoint with the same confident best action; otherwise it uses
+the full 300 seconds.  Confidence requires a predicted margin of at least 0.02
+and the best lower bound to exceed the second-best upper bound.  If the final
+choice is not confident, the next run is the positive-UCB action, explicitly
+marked as a diagnostic intervention.  The exact same rule is used for the
+outcome-only baseline.
+
+## Hypotheses and gates
+
+### H1: action value
+
+Engine state must change the selected intervention or its ranking and reduce
+real action regret.  Prediction error or bottleneck-label accuracy is not a
+success metric.
+
+### H2: measurement value
+
+Engine state must select a shorter stable source measurement without increasing
+action regret.  A shorter reconstructed prefix is only a trigger; it is not an
+actual GPU-cost claim until an early-terminated confirmation run measures
+startup, warm-up, drain, and cleanup.
+
+### H3: end-to-end cost
+
+Primary development metric is H20-hours to first reach a configuration within
+2% of the exact median-goodput oracle.  The outcome-only and telemetry policies
+use the same measured config costs and differ only in source information.
+
+- At least 10% prospective replay cost reduction, telemetry regret at most 2%,
+  and no outcome-only-to-telemetry harm triggers an actual early-stop
+  confirmation.
+- At least 20% measured all-in H20-hour reduction is required for a contribution
+  claim.  This one task can only establish development feasibility; a paper
+  claim additionally requires task-held-out replication.
+- Source median normalized goodput at or above 0.98 stops the surface before
+  target runs because the workload has no material improvement headroom.
+- Any hash mismatch, missing/censored result, telemetry drop, non-monotonic
+  phase, negative cost, ratio outside `[0,1]`, or all-identical config outcomes
+  is a red flag and stops analysis.
+
+If the 10% trigger fails, this route is closed for the current engine-state
+representation.  The experimental control plane is not retained as a fallback
+research contribution.
+
+Pre-run provenance amendment: the first controller dry-run on 2026-07-15
+rejected two stale engine paths before starting a server.  The paths and exact
+runtime version above were recovered from the accepted trace-12 campaign and
+corrected before any trace-13 GPU work.  No scientific treatment or gate was
+changed.

docs/active-intervention-v0-results-20260715.md

@@ -0,0 +1,106 @@
+# Active intervention v0: held-out trace-13 result
+
+Date: 2026-07-15 (Asia/Singapore)
+
+Decision: **close the passive-telemetry treatment-effect route**.  The held-out
+campaign produced no telemetry-induced action change, measurement reduction, or
+GPU-cost reduction.  It did show that the engine state contained the correct
+action-specific mechanism; the current feature model failed to use it.
+
+## Headline result
+
+The outcome-only and telemetry policies both measured the source for 300
+seconds, selected `joint=(MNS64,MBBT8192)`, and produced the same complete
+acquisition order.  Both reached the exact finite-surface oracle after the
+first intervention at a reconstructed all-in lower-bound cost of 2.4284
+H20-hours.  Telemetry GPU-cost reduction was therefore exactly 0%, below the
+10% confirmation trigger and 20% contribution gate.  No actual early-stop
+confirmation was launched.
+
+The complete annotation campaign cost 5.0379 H20-hours, below the 6.0 H20-hour
+hard cap.  It ran 12 uncensored real-GPU outcomes: four configs, three disjoint
+request partitions, and a fresh server per config.
+
+## Exact response surface
+
+Median normalized SLO-goodput was:
+
+| Config | Rep values | Median |
+|---|---|---:|
+| `MNS32,MBBT4096` source | 0.40091 / 0.39788 / 0.42061 | 0.40091 |
+| `MNS64,MBBT4096` | 1.00000 / 0.99970 / 1.00000 | 1.00000 |
+| `MNS32,MBBT8192` | 0.44394 / 0.41515 / 0.42606 | 0.42606 |
+| `MNS64,MBBT8192` joint | 1.00000 / 1.00000 / 1.00000 | 1.00000 |
+
+Increasing MNS alone was sufficient and joint was redundant.  Increasing MBBT
+alone improved the median by only 0.02515, versus 0.59909 for MNS.  This is a
+strong non-additive action response, not a setting where independently tuning
+the knobs and merging their improvements is valid.
+
+## What the telemetry actually said
+
+Across 41,086 source scheduler records, 93.12% of steps had waiting work,
+85.36% were MNS-exclusive binding, 1.11% were MBBT-exclusive, mean running-slot
+utilization was 97.39%, mean token-budget utilization was 15.69%, mean KV usage
+was 2.75%, and there were no preemptions.
+
+The intervention transition agreed with that state:
+
+| Config | Waiting | MNS-exclusive | MBBT-exclusive | Median goodput |
+|---|---:|---:|---:|---:|
+| source | 93.12% | 85.36% | 1.11% | 0.40091 |
+| MNS only | 5.38% | 0% | 5.38% | 1.00000 |
+| MBBT only | 91.19% | 91.09% | 0.04% | 0.42606 |
+| joint | 0.89% | 0% | 0.89% | 1.00000 |
+
+Thus this experiment does **not** support the claim that engine telemetry lacks
+tuning information.  It rejects the narrower claim that adding passive state
+summaries to the current small-data ridge policy converts that information into
+lower tuning cost.
+
+## Why the learned policy failed
+
+At 300 seconds, the telemetry model predicted joint, MNS, and MBBT effects of
+0.35190, 0.26118, and 0.09686.  The actual median effects were 0.59909,
+0.59909, and 0.02515.  Telemetry therefore made the nonexistent joint-over-MNS
+gap larger: 0.09072 predicted versus 0 actual; the outcome-only model predicted
+0.03188.
+
+The failure has three concrete causes:
+
+1. The six training decisions contain no joint intervention.  The
+   `delta_product` feature has no support, so joint ranking is extrapolation.
+2. Passive raw summaries do not represent the counterfactual scheduler work
+   unlocked by each action.  Capacity-normalized MNS pressure was visible, but
+   the model was not structurally required to map it to MNS marginal value.
+3. The policy maximizes predicted effect.  It does not identify the smallest
+   epsilon-optimal intervention or price unsupported action complexity.
+
+## Research implication
+
+Do not retain the harness or the passive telemetry model as a contribution.
+The next defensible route is engine-native, action-conditional counterfactual
+instrumentation: at a real scheduling state, shadow-replay the exact scheduler
+decision under an MNS relaxation, MBBT relaxation, and their joint relaxation,
+then expose the incremental queued work admitted by each action.  Real paired
+interventions calibrate how those one-step shadow effects map to E2E SLO
+goodput.  This is distinct from a hand-written cap-to-knob rule and from a
+full-system simulator: it reuses the exact live queue, scheduler, and cache
+state while simulating only the local decision boundary.
+
+That route should be evaluated against outcome-only search, the present passive
+telemetry model, a cap-hit expert rule, and a full simulator.  The paper-level
+gate remains at least 20% measured H20-hour reduction to a 2%-oracle config on
+task-held-out workloads with at most 2% regret.
+
+## Sanity
+
+Surface outcomes: n=12, min=0.39788, max=1.0, distinct=8.  Session costs: n=4,
+min=1.1702, max=1.3566 H20-hours, distinct=4.  Scheduler-record counts: n=4,
+min=37,001, max=41,348, distinct=4.  All counters and costs were non-negative;
+all ratios were in `[0,1]`; request hashes matched; all 12 runs were uncensored;
+the controller and four sessions completed; and config outcomes were not all
+identical.  No red flags were found.
+
+Machine-readable summary: `runs/active-intervention-v0/trace13-results.json`.
+Raw immutable root: `/home/admin/cpfs/wjh/active-intervention-prospective-20260715`.

docs/collectivespec-p2-gate-20260713.md

@@ -0,0 +1,154 @@
+# CollectiveSpec P2：logical-plan 对照的审计与停止门槛
+
+## 决策
+
+**不启动正式的 P1/P2 SLO-goodput sweep，也不把 `compact-vs-padded` 作为
+CollectiveSpec 的研究主线。**
+
+原因不是这条机制一定没有工程收益，而是它的核心研究主张已经无法排除公开工作的
+覆盖：
+
+- [DSpark](https://arxiv.org/html/2607.05147) 已明确采用每请求的动态 verification
+  length，并把逻辑 sequence tracking 与物理 execution 解耦、flatten variable-length
+  token；
+- SGLang 的 [DSpark 集成说明](https://www.lmsys.org/blog/2026-07-06-dspark-sglang/)
+  已公开 `static`、`compact` 与 `cap-accept` 三种 verify mode。其中 `cap-accept`
+  执行完整 block、但只提交 compact window，且说明其输出与 `compact` 相同。这正是
+  “同一语义下 full/padded 与 compact”的 counterfactual；
+- 该实现还公开了 DP attention 下各 rank 使用最大 graph tier 的处理。因此，仅在
+  vLLM/H20 上再复现 compact 比 padded 快，只是环境复现，不是新的系统贡献。
+
+P0 还独立否定了原来的 liveness 动机：目标 runtime 已经以 scalar DP metadata 协调
+不同 DP replica 的物理 shape；异构 verifier candidate 没有引起运行期 collective
+错误。故不能再把“必须新增 canonical header 才能避免死锁”作为论文 premise。
+
+## 术语：什么必须相同
+
+### Logical plan（也称 semantic plan）
+
+logical plan 是一次 speculative verification **应当计算和提交什么**的不可变记录；它
+不包含 padding、CUDA graph tier、物理 rank 行数、worker PID 或耗时。每一个 verifier
+epoch 的最小条目为：
+
+```text
+(global_epoch, dp_rank, ordered request id,
+ logical_output_offset_before, scheduled_seq_len,
+ available_candidate_token_ids, requested_k, effective_k,
+ visible_candidate_token_ids_hash)
+```
+
+请求层还必须固定 `client_request_id`、server request id、prompt/body hash、arrival、DP
+assignment、提交顺序、temperature/seed 与预期 completion length。最终还要逐请求验证
+output token-id hash、completion length、finish reason、usage，以及 endpoint semantic
+transcript hash（content/reasoning/tool-call 的 canonical JSON）。
+
+这里的关键是 `k_i` 的 key 必须是
+`(server_request_id, logical_output_offset_before)`，而不是只有 request id：同一请求会
+经历多个 verification epoch。只有两个 cell 的这些事实都相同，才称为 *same logical
+plan*。
+
+### Compact vs. padded：只是同一 plan 的两个 lowering
+
+给定同一组 non-dummy logical entries：
+
+- **PaddedSync-semantic**：保留这些 entries，但为同步域插入 masked dummy rows，使各
+  DP peer 的 physical shape 对齐；
+- **CompactSync**：保留完全相同的 entries、candidate 和 commit semantics，用 ragged
+  packing / split vector 执行真实 rows，不计算 dummy rows。
+
+因此 `static K=3` 不能当作 padded 对照：它改变了每个请求可见 candidate prefix，改变了
+logical algorithm，而不是只改变 physical lowering。真实 physical-row 公式也不能简单
+写成 `N * (1 + max k_i)`；普通 decode、TP alignment 和 CUDA-graph alignment 都要从
+runner 的 row map 分开计数。
+
+## 当前 P0 对 P2 的限制
+
+P0 heterogeneous policy 按 vLLM **随机生成的 server request id** 哈希，而 client 没有
+发送 `X-Request-Id`。所以即使 trace 和 seed 相同，两个 cell 的每个 request/epoch 的
+`k_i` 也不可保证相同；日志只有 aggregate digest/histogram，也没有 per-row candidate
+token、assignment 或最终 token-id hash。P0 因而不能充当 same-logical-plan 的 P2 A/B。
+
+P0 的 padding 上界也已校正。66 个 target epoch 中 62 个 raw DP counts 不等：
+
+```text
+raw logical rows                 6,276
+local non-DP-aligned rows        6,536
+PaddedSync physical rows         7,024
+DP-global-max attributable rows    488  (= 6.95% of physical rows)
+```
+
+此前的 748 / 10.65% 将 260 行本地 TP/CUDA-graph alignment 混入 DP max padding。即使
+488 行都可回收，它仍只是 **target verifier row-count 的上界**：EAGLE3 仍按 Kmax=3
+完成 drafter 工作，也没有测得 EP bytes、collective critical path 或 E2E SLO-goodput。
+
+## 若未来重新打开，先补齐的测量契约
+
+不应先实现 compact lowering。先添加只用于 audit 的 telemetry：
+
+1. client 对每个请求发送固定 `X-Request-Id`、`X-Data-Parallel-Rank` 和
+   `return_token_ids=true`；记录 response id、prompt/output token-id hash、semantic
+   transcript hash 和 finish reason；
+2. scheduler 作为 semantic ledger 的唯一 writer，记录 per-epoch ordered entries、候选
+   token、requested/effective K、logical cursor 与 sampled token hash；
+3. worker 只记录物理事实：每 rank physical rows、DP/TP/graph padding 的原因、packed row
+   map；若要主张 EP 收益，额外记录 DeepEP all-to-all split vector、bytes、duration 和
+   rank wait；
+4. 汇总器先给出第一个 semantic/output diff，任一 mismatch 即标为 invalid，禁止读取
+   性能数字。
+
+最小 reproducibility smoke（仅在发现 topology gap 后执行）是 fresh engine 上的 16 个
+decode-only requests、每个 64 output tokens、temperature=0、DP0/DP1 各 8 个、显式
+`{0,3}` alternating manifest。先连续跑两次 **同一个** padded cell；只有 ledger 和逐请求
+token hash 全等，才允许运行 padded/compact mechanism probe。`return_token_ids` 会改变 SSE
+负担，故最终 latency cell 必须关掉该字段、改以 scheduler-side hash 审计。
+
+## 唯一尚可证伪的拓扑假设
+
+公开材料没有证明、也没有否定下列特殊情形：**独立 standard-DP scheduler 共享同一个 EP
+all-to-all domain** 时，DP global-max graph tier 之外仍有 EP split-vector / collective
+ordinal 的关键路径浪费。这不能从“论文没有写”推断为新颖性。
+
+只有一次短的 topology reconnaissance 观测到该额外瓶颈，才重新进行文献审计并考虑下列
+顺序严格的 gates：
+
+1. 与 SGLang-style DP global-max tier / current runtime 相比，compact plan 降低实际 EP
+   bytes、split imbalance 或 collective critical-path time；仅少几个 rows 不够；
+2. 在相同 semantic plan、token-exact 输出和无 tail-latency 退化下，至少三次 fresh-engine
+   paired runs 显示 E2E SLO-goodput 增益 >=10%；
+3. topology ablation 支持因果归因：DP=1 或 EP 不跨 DP 时收益消失或显著缩小，而
+   DP×shared-EP 时出现；
+4. 重新完成与 DSpark/SGLang 的逐项差异审计，证明贡献是 topology-aware collective
+   scheduling，而不是已有 ragged packing。
+
+任一 gate 不成立即结束 CollectiveSpec；不以 controller/K/queue knob 调优替代证据。
+
+## 如果 gate 重开时的固定环境与 setup
+
+下列是 P0 实际使用、后续必须 provenance-pin 的环境，而不是当前已启动的实验：
+
+| 项目 | 固定值 |
+|---|---|
+| host / accelerator | `dash0`，8 × NVIDIA H20 |
+| target / draft | Qwen3-235B-A22B FP8；EAGLE3，Kmax=3 |
+| parallelism | TP=4，DP=2，EP=8；`VLLM_MOE_USE_DEEPEP=1` |
+| engine | dash0 live installed vLLM wheel；记录 wheel metadata、import path、launch command 与 commit；不能以本地 checkout API 代替 |
+| execution | `FULL_DECODE_ONLY` CUDA graphs、FP8 KV cache、block size 64、`max-num-batched-tokens=1024`、`max-num-seqs=192`、max model len 262144 |
+| workload | immutable materialized `thinking_w20260327_1000` 的 decode-only window；机制 smoke 使用固定 burst，E2E 使用完整、session-closure 状态明确的 trace |
+| reproducibility | fresh engine per cell、temperature=0、固定 seed、固定 request ids/DP assignment、prefix-cache state 从空开始、ABBA cell order |
+| SLO（若进入 E2E） | 预注册 TPOT <= 40 ms、pass rate >= 0.95；同时报告 completion success、p50/p95/p99、deadline failures 与 output equivalence |
+
+remote source 必须从 Git 同步到
+`/home/admin/cpfs/wjh/collectivespec-pilot/20260713T054328Z/source`，并记录运行时实际
+source revision；任何远端 job 启动前在 artifact 中写明 resolved command、模型/trace path、
+预计 GPU 时间和结果目录。
+
+## 审计数据健全性
+
+- 新增实验数：n=0；本文件不报告任何新的性能数字。
+- 已复核的 P0 target epochs：n=66；两个 DP rank 的 raw row values 共 n=132，
+  min=1、max=77、distinct=35；physical rows 则 n=66，min=4、max=80、distinct=14。
+  原始 JSONL 可复核，不以 aggregate 值伪造每 epoch 分布。
+- 已用 aggregate row totals：n=4，min=488，max=7,024，distinct=4；均为非负。校正后的
+  关系 `6,276 <= 6,536 <= 7,024` 成立，且 `7,024 - 6,536 = 488`。
+- 外部材料覆盖判断区分为“论文明确描述”“官方公开实现明确描述”和“未公开拓扑细节”；
+  未从缺失的 EP 细节推导新颖性或性能收益。

docs/collectivespec-pilot-design-20260713.md

@@ -0,0 +1,319 @@
+# CollectiveSpec：先做机会判定的实验设计
+
+## 结论先行
+
+当前不应先实现一个“按请求动态 K、再做一次 DP collective 同步”的原型。该路径的
+工程修复很薄，而且相邻公开工作已经覆盖了 request-level dynamic speculation 与
+ragged verification 的大量空间。CollectiveSpec 只有在
+一个更强、可证伪的事实成立时才值得继续：**在 wide-EP MoE、DP>1 的生产负载中，
+不同 DP rank / request 所需的投机深度确实不同，并且全局 static K 明显浪费了 SLO
+可行 goodput。**
+
+本文件把第一轮实验定义为一个机会判定（opportunity gate），不是最终性能主张。
+
+## 固定条件
+
+- Host: `dash0`, 8x NVIDIA H20。
+- Model: Qwen3-235B-A22B FP8；draft: EAGLE3。
+- Deployment: TP=4, DP=2, EP=8，`VLLM_MOE_USE_DEEPEP=1`。
+- Trace: `thinking_w20260327_1000`，600 秒 decode-only 窗口。
+- SLO: TPOT <= 40 ms，pass rate >= 0.95。
+- 同一 engine revision、同一模型/trace 路径、同一环境变量；实验串行执行，避免 GPU
+  互相干扰。
+
+这里的 resolved topology 来自远端实际 StudySpec，而不是仓库 README 中可能已过期的
+配置描述。
+
+## 假设与可证伪指标
+
+### G0：static-K 是否有足够可利用的空间？
+
+- H0：在该固定拓扑和负载下，NoSpec/K=1/2/3 的 SLO-goodput 差异很小；最佳固定 K 已经
+  足够好。此时停止 CollectiveSpec。
+- H1：不同 static K 的可行前沿存在实质差异，且最优 K 对负载区间敏感。只有 H1 才
+  说明 dynamic policy 可能有直接性能价值。
+
+主要指标：每个 K 在相同 SLO 下可达到的最大 `sampling_u`，以及对应请求率、TPOT
+pass rate、p50/p95 TPOT、成功/失败原因。`sampling_u` 是现有 replay 使用的一致 trace
+抽样旋钮，因此只能作为此 trace 的 SLO-goodput proxy，不能直接外推为线上 QPS。
+
+本 trace 的 output-length 分布很重尾；replayer 的 drain deadline 可能在长输出尚未完成
+时终止 probe。故每个结果必须同时报告 completion-success count 和 deadline failure；不能
+只因 TPOT pass rate 达标就把截断 request 当作“无成本”。本轮的原始 static screen 仍沿用
+现有 SLO 以便和项目 baseline 可比，但它不能替代完整 completion 的确认实验。
+
+判定门槛（预注册）：
+
+1. 在 K=1/2/3 之间，最优 K 相对次优 K 的最大可行 `sampling_u` 小于 5%，或
+   置信区间/重复实验重叠很大：
+   **停止**。
+2. 最优 speculative K 比次优 speculative K 至少高 10%，且在两个独立重复中方向一致：
+   进入 G1。NoSpec 仅作为“是否值得用 draft model”的部署对照，不能替代这条判据。
+3. 如果 K=3 不受当前 engine 支持、任一配置启动失败，记录为兼容性结果，不把它误作
+   性能差。
+
+## 第一阶段：static-K screening
+
+配置为 `{NoSpec, K=1, K=2, K=3}`。NoSpec 会删除 `--speculative-config`，而不是传
+非法的 `num_speculative_tokens=0`。它释放 draft model 相关资源，因此不等于
+same-stack 的 logical K=0；后者在 EAGLE 类实现中仍可能需要一次 draft forward 来保持
+KV 同步。当前 dash0 binary 的 MLA indexer 明确限制 `num_speculative_tokens <= 3`，
+故这已穷尽该 binary 的合法 static horizon。原计划每个配置：
+
+- 搜索范围 `sampling_u in [0.005, 0.020]`；
+- 最多 3 次 probe、tolerance=0.003；
+- 每个 probe 使用完整 600 秒 trace replay（不会使用 `max_requests_per_probe` 的
+  截断模式）；
+- 启动顺序 `2,1,3,0`，降低冷启动或时间漂移与 K 单调对应的风险；
+- 每个 K 都使用独立 Store、不可变派生 StudySpec、完整 stdout/stderr log。
+
+这是一轮筛选而非 final frontier。它一旦显示值得继续，才对 top-2 K 做交叉顺序的完整
+搜索与至少两次重复。
+
+### 2026-07-13 数据质量修正：controlled screen
+
+原始 trace 第一个 K=2 probe 暴露了一个不应隐藏的 measurement red flag：worker 的
+drain deadline 按 selected set 的 p99 output length 计算，而该 set 含一个 36,034-token
+completion。该 request 因 deadline 被裁掉；尽管 TPOT-only pass rate 仍可达标，censoring
+会随 `sampling_u` 改变 selected set，不能用于比较 static-K frontier。
+
+因此原始 run 只保留为诊断 artifact，停止后改跑一个明确标为 **controlled** 的 screen：
+保持相同 arrival、prompt、sampling seed 和 topology，但把每个 request 的
+`min_tokens=max_tokens=4096`。4096 接近原始 output mean 3,924.6，且使 p99 deadline 覆盖
+每个 request 的完整 completion。它回答的是“在相同输入/到达条件下，static K 是否留下
+可利用空间”，不是 production trace 的最终 goodput 结论。最终论文实验必须同时有：
+
+1. 该受控 curve（机制和可重复性）；
+2. 原始长度 trace 的完整-completion 版本（不能使用 p99 censoring）；
+3. 至少一个 held-out window。
+
+### 2026-07-13 数据质量修正 #2：fresh-engine fixed grid
+
+受控 screen 的第一条 K=2、`u=0.0125` probe 本身通过了完整性检查（152/152 success、
+usage 返回的 completion token 均精确为 4096、无 early stop）。但随后发现原二分搜索会
+在同一 engine 内连续执行多个 probe；第二、三 probe 继承前一 probe 的 prefix/KV cache，
+也继承全局 RNG 的已消耗状态。不同 K 的二分分支/中止路径不同，因此不能把该搜索输出的
+`best_sampling_u` 差异直接归因于 K。该 run 在第二 probe 中主动停止，**不作为 G0 结果**。
+
+替代协议是每个 `(offered-load, K)` 只运行一次、每次均启动一个 fresh engine。两个固定
+负载由原始 immutable trace 的统一 `sampling_u` 阈值物化：`u=0.0125`（152 requests,
+0.2533 req/s）和 `u=0.0200`（263 requests, 0.4383 req/s）。物化后的 request 仍保留原
+prompt、arrival 和 sampling provenance，但强制 `temperature=0`、显式 engine `seed=0`，
+并用统一的 4096 completion override。每个 K 只有一个 probe，所以 accelerator KV/prefix
+cache 为空且 RNG 从相同 seed 开始；每次都从 `probe_details.jsonl` 验证：
+
+1. `early_stopped=false`；
+2. outcome count = selected count，且每个 request success；
+3. TTFT/TPOT 均非空；
+4. completion token 的 source 为 usage，且实际/预期均为 4096；
+5. result 无 partial-probe failure，且只包含一个 primary probe。
+
+运行顺序在两个负载间反向（ABBA），避免 K 与时间漂移完全共线。这个 grid 仍然只回答
+static-K 是否存在足够大的机会；它不估计 production sampling goodput，也不证明 rank-local
+K 的上界。当前 vLLM deployment 的 `reasoning_parser=''`；其 SSE 实现将生成文本放在
+`delta.content`，所以本协议中的 token-time 定义覆盖当前 `<think>` 输出。若以后启用
+reasoning parser，客户端必须同时记录 `reasoning_content` 后才可复用此指标。
+
+## G1：只有在 G0 通过后才做的直接验证
+
+目标不是“不同请求有不同 K”这种已经很常见的说法，而是验证下面的系统命题：
+
+> 在 DP+EP MoE 下，局部独立的 K 决策会让 collective 序列分歧；把它们编译为
+> rank-agreement 的 ragged execution plan，可保留异质请求的计算节省，同时不改变
+> collective order。
+
+当前 dash0 vLLM 已经有一个很好的切入点：每个 DP step 会 all-reduce 一段 metadata，
+并把各 rank 的 total token count padding 到最大值；CUDA graph mode 也会取跨 rank 的
+共同模式。这说明论文的最小机制不应另造一个 scheduler，而应把现有 scalar
+`(num_tokens, num_reqs, graph_mode)` agreement 扩展为 canonical speculative-plan header。
+关键增量是让 header 描述真实 active frontiers，并保证后续 verifier/EP split vector 的
+collective ordinal 相同；若最后仍 padding 到 global max，就没有可主张的性能机制。
+
+需要实现/测量：
+
+1. **oracle trace replayer**：利用 G0 的 per-K service curve，为每个到达时刻选择
+   SLO-feasible K，比较 best-static K 与 oracle 的 upper bound。若 oracle gain <10%，
+   停止，避免把噪声当论文方向。
+2. **collective trace**：按 DP rank 记录每个 decode step 的 collective 序列、token
+   shape、active-sequence mask、MoE all-to-all bytes 和 rank idle time。验证“local K
+   不同”是否真的导致 sequence divergence，而不是仅是一个 API 限制。
+3. **CollectiveSpec prototype**：固定 collective order，用全局 agreement header 和
+   ragged/padded verification plan；对比 `best static K`、global-max-K、oracle 和当前
+   upstream dynamic-spec baseline（包括 DSpark/FASER 能实现的部分）。
+4. **ablation**：去掉 agreement、去掉 ragged packing、去掉 queue/SLO policy；报告
+   goodput、p50/p95/p99 TPOT、acceptance、MoE communication bytes、GPU SM/HBM util、
+   rank skew。
+
+## 主要风险
+
+- 最新 upstream 动态投机对 DP>1 的处理可能本身只需一个 global-K broadcast；那是
+  feature patch，不构成研究贡献。
+- 当前 dash0 runtime 已验证 DP=2 + static EAGLE 可以工作；尚未在这个 binary 上证明
+  “local dynamic K 会 deadlock”。因此研究动机必须写成固定 EAGLE horizon 的执行限制，
+  不能把未运行的 dynamic-K 路径当作既成故障。
+- FASER/DSpark 等相邻工作会把“dynamic K + ragged verify”作为强 baseline；必须在
+  做任何大实现前进行逐项复现/排除。
+- trace 的 `sampling_u` 是 proxy；最终结论必须在固定 arrival trace、真实请求长度和
+  至少一个不同 workload 上复现。
+
+## 2026-07-13 系统重审：收紧主张和后续 gate
+
+本轮查阅 [vLLM Dynamic SD 限制](https://docs.vllm.ai/en/latest/features/speculative_decoding/dynamic_speculative_decoding/)
+与 [DSpark](https://arxiv.org/abs/2607.05147) 后，原先“dynamic K 会使 collective
+diverge，因此做 ragged verifier”这一表述太宽，不能作为实现前提：
+
+- DSpark 已公开主张按请求动态 verification length、跨请求 token flatten 与 ragged
+  physical execution；这些本身不再构成新颖性。
+- dash0 当前实际 import 的 vLLM wheel 与本地 checkout 不同。实验中的运行日志称
+  `v0.11.1`，而 wheel metadata 为 `0.13.0rc2.dev2111+gb44b43f43.d20260309`；后续任何
+  hook 必须对这个 live source 做 provenance pin，不能把本地 v0.24 API 当作证据。
+- 这个 live runtime 已原生使用 per-request `list[list[int]]` draft token IDs，并将真实
+  长度送入 scheduler/metadata；EP all-to-all 也具有 variable-split data path。因此
+  **per-request horizon 不等于 collective divergence**。必须先观测到不同 DP replica
+  的 collective call count、phase 或 branch trace 的真实不一致。
+
+修订后的唯一可能研究命题是更窄的：
+
+> 对共享一个 EP collective domain 的独立 DP scheduler，如何将异构 verification plan
+> 编译为可证明 liveness 的 canonical execution trace，同时在同一逻辑 plan 下回收
+> global physical-max padding 的关键路径成本。
+
+它有四个顺序严格的停止门槛：
+
+1. **P0 / 真实 premise**：以预先给定的 `k_i ∈ {0,1,2,3}` replay 表截断已生成的
+   EAGLE candidates，在每个 DP/TP rank 记录 target、EAGLE 和 EP phase signature。若
+   没有 trace mismatch、强制 global padding 或 liveness 问题，就停止把 plan header 当作
+   研究贡献。
+2. **P1 / 机会量**：即使存在 mismatch，也必须证明 rank-local oracle 相对 best static
+   或 globally synchronized K 有至少 10% 的 SLO-goodput headroom；全局同步若恢复 90%
+   以上 gap，则只值得做 upstream patch。
+3. **P2 / 因果物理对照**：同一个 `k_i` replay plan 必须对比
+   `PaddedSync-semantic`（物理 `N × (1 + max k_i)` rows）与 `CompactSync`
+   （物理 `Σ_i(1+k_i)` rows）。static K=3 不是 PaddedSync，因为二者 logical algorithm
+   不同。若实际 target/EP work 与关键路径未减少，停止。
+4. **P3 / 正确性与部署**：所有 TP peers 的 plan digest 相同、所有 EP ranks 观察到同一
+   header vector、每 epoch 的 target/EAGLE/MoE signature 一致；greedy output token-exact，
+   并通过 empty-rank、`{0,3}` 交替、长时间 stress。只有随后在两个 session-coherent
+   workload 上重现 SLO-goodput 才能讨论论文。
+
+若 P0--P3 任一项失败，合理结论是停止 CollectiveSpec，而不是继续调 controller、K 或
+queue policy。若全部通过，最小原型的边界也只应是 verifier-side compaction：当前
+EAGLE 仍按 Kmax 产生 candidate，短 `k_i` 不会自动消除 draft-side work，不能把 verifier
+节省误报为完整 draft+verify speedup。
+
+### Trace 数据可用性与 window-closure fallback
+
+本轮发现 `prepare_trace_windows.py` 依赖的 2026-03-27 原始格式化 trace span 在 dash0
+已不存在，不能把它的 streaming full-session root 当作已经复验。现有的完整 600 秒
+materialized window 仍含 prompt、`chat_id` 与 `parent_chat_id`，因此新增的 fallback 仅将
+窗口内图的 connected component 作为选择原子：保留 384/384 条窗口内 parent edge，且把
+同一个窗口外 parent 的 sibling 归为同一 component。该窗口有 15,479 requests、15,095
+components；414 条 non-root edge 中 30 条（7.25%）父节点落在窗口外。
+
+这只能称为 **window-session-closed**，不等于 full-session coherent：任何结果都必须报告
+这 30 条 boundary-parent residual，且不能据此声称跨窗口 KV reuse。若原始 span 恢复，必须
+重新从完整 source resolve root 与重新采样，不能沿用 fallback 的 score/threshold。
+
+## 2026-07-13 P0 v2：header/liveness premise 的实际结果
+
+### 先报异常
+
+两个 cell 都在 probe/result 已落盘、64 个请求都已完成后出现 teardown 异常。heterogeneous
+cell 有 `free(): corrupted unsorted chunks` 与共享资源泄漏；control 还出现 SIGTERM/SIG11
+和 TCPStore broken pipe。这些不是运行期 request/collective failure，但也意味着本实验**不
+证明干净退出或部署鲁棒性**。本节只使用完成前的 request、worker phase 与 DP metadata
+作为 P0 evidence；不报告任何 TPOT/QPS 比较。
+
+### 设计、判定与修正后的观测范围
+
+在 dash0 的 Qwen3-235B-A22B FP8 + EAGLE3、TP=4/DP=2/EP=8、DeepEP 配置上，P0 将 EAGLE
+已生成的 Kmax=3 candidates 按预先给定的 request-static 表截断为 `k_i ∈ {0,1,2,3}`。它只
+改变 verifier 可见 candidate，不消除 EAGLE 的 Kmax drafter 工作。
+
+原始 worker hook 还会记录 vLLM 的 profile/DP dummy run；空 `SchedulerOutput` 仍可能有
+physical rows。因此真实 target batch 的判据固定为：
+
+```text
+event == batch_execution_plan
+AND request_count > 0
+AND total_scheduled_rows > 0
+```
+
+最初 summary 将 676/640 条 dummy/profile record 混入 target phase，错误地把 control 的
+145/146 internal-call 差异解释为 rank mismatch。修正后的汇总只比较 target event，并将
+真实 DP pair 识别为 `[0,4]`、`[1,5]`、`[2,6]`、`[3,7]`；同一 logical DP replica 内的 TP
+group 则为 `[0,1,2,3]` 与 `[4,5,6,7]`。
+
+### 结果
+
+远端可复核 artifact：
+`/home/admin/cpfs/wjh/collectivespec-pilot/20260713T054328Z/p0_phase_v2_20260713T0944Z`
+（run source `bb698b5`；`summary.json`、`summary.md`、`driver_result.json` 和原始
+`p0_logs/*.jsonl` 均在该目录）。
+
+| cell | completion | 实际 candidate K | target worker records | target plan / DP coordination |
+|---|---:|---|---:|---|
+| control K=3 | 64/64，usage 均为 64 | `{3}` | 488；DP0 每 TP peer 65，DP1 每 peer 57 | 每个 logical DP replica 内序列完全一致；四个真实 DP pair 的 57 个 shared target epoch 的 scalar coordination signature 一致 |
+| heterogeneous | 64/64，usage 均为 64 | `{0,1,2,3}` | 528；8 个 peer 各 66 | 两个 logical DP replica 内序列完全一致；四个真实 DP pair 的 66/66 shared target epoch signature 一致 |
+
+heterogeneous 的 `candidate_truncate` 直方图为 `{0: 1408, 1: 670, 2: 672, 3: 398}`，而截断前
+全部为 K=3（共 3,148 个 candidate）。所以它不是只改变 log 的“伪异构”实验。两 cell 的
+target record 都满足：
+
+```text
+num_tokens_per_rank[dp_rank] == total_scheduled_rows
+physical_batch_rows == rows_across_dp[dp_rank]
+rows_across_dp[i] >= num_tokens_per_rank[i]
+len(rows_across_dp) == len(num_tokens_per_rank) == 2
+```
+
+这说明两个独立 scheduler 的 logical plan 可以不同，但 live runtime 已用 scalar DP metadata
+协调共同 physical shape，并让共享 EP domain 的真实请求完成；没有观察到运行期 deadlock 或
+collective error。它反驳的是“异构 verifier-side K 必须新增 canonical header 才能先保证
+liveness”的必要性，而不是一般性的形式化证明。
+
+### 留下的物理现象，以及为什么它仍不足以继续造系统
+
+heterogeneous 的 66 个 shared target epoch 中有 62 个的 raw DP token counts 不相等；现有
+runtime 将 `rows_across_dp` 同步为共同 shape。按每个 DP replica 的一个 TP anchor 计，
+target-only raw logical rows 为 6,276；逐 epoch 保留当前 TP/CUDA-graph local alignment 后为
+6,536；最终 physical rows 为 7,024。因此可单独归因给跨 DP global-max padding 的只有
+488 rows（6.95% physical rows）。此前用 7,024-6,276 得到的 748（10.65%）还混入了 260
+行本地 alignment，不能当作 compact 对照可回收的 DP work。control 的 row totals 也不与
+heterogeneous 直接比较，因为 logical plan 和 scheduler trajectory 不同。
+
+这只能看作 **P2 的 row-count upper bound**，绝不能把 control 与 heterogeneous 相减当作
+速度收益：两者 logical plan、scheduler trajectory 都不同。更关键的是 EAGLE drafter 仍完成
+Kmax 工作；即使理想 compact verifier 回收全部 6.95% 的 DP-only target rows，端到端
+SLO-goodput 增益也只会更小。
+
+因此决策为：
+
+1. **停止**把 canonical plan header / deadlock avoidance 当作 CollectiveSpec 的研究主线；
+   P0 已在目标部署上否定其必要前提。
+2. **停止**把“dynamic verification length + flattened ragged execution”本身当贡献；DSpark
+   已覆盖该组合，且它也指出固定长度 drafter 的前置工作不会因 verifier 截断自动消失。
+3. 仅保留一个很窄的、默认 no-go 的机会：同一 logical plan 下的 compact-vs-padded physical
+   execution。只有先以 P1 证明相对 best-static/global-sync 至少 10% E2E SLO-goodput，再以
+   P2 的因果对照证明关键路径 rows/bytes 真正下降，并完成 DSpark topology gap 审计，才值得
+   再投入实现。当前 P0 不满足这些条件。
+
+P0 未验证 greedy token-exact 输出、真实 DeepEP dispatch ordinal/split digest、取消/empty-rank
+stress，或其他模型/后端/更大 K 的泛化；这些都不能从本结果外推。
+
+### P0 data sanity
+
+- **teardown red flag 已单列**：control/heterogeneous 都在 completion 后发生 allocator/资源
+  清理异常；因此没有使用延迟、吞吐或 clean-shutdown 指标作结论。
+- n=2 cells，8 worker/cell，64 usage-verified completions/cell；completion count 的
+  min=max=64，distinct=1。
+- target worker record count：control=488、heterogeneous=528（min=488，max=528，distinct=2）；
+  dummy/profile records 分别为 676/640，已排除。
+- heterogeneous K：min=0，max=3，distinct=4；control K distinct=1。所有计数、rows 和
+  padding 非负；JSON parse errors=0。
+- heterogeneous 的 66 target epochs：两个 DP rank 的 raw row values 共 n=132，min=1、
+  max=77、distinct=35；physical rows n=66，min=4、max=80、distinct=14；分解
+  `6,276 <= 6,536 <= 7,024` 与 `7,024 - 6,536 = 488` 均成立。
+- 修正后的不变量均为 true：probe integrity、8 workers observed、每个 DP replica 内 target
+  phase/sequence 一致、target DP metadata 合法、DP coordination record 存在、四个真实 DP pair
+  的 shared scalar coordination signature 一致。

docs/fidelity-aware-harness-headroom-20260714.md

@@ -0,0 +1,135 @@
+# Fidelity-aware harness headroom audit
+
+Status: **HISTORICAL PREMISE DID NOT PASS PROSPECTIVE P1; NO CONTRIBUTION CLAIM**.
+
+The audit answers whether engine instrumentation has enough incremental signal
+to justify a prospective experiment.  It does not establish generalization.
+
+Post-run update: the exact-timestamp held-out P1 completed and failed the
+registered gate.  Under the stronger simulator-aware `k=2` end-to-end replay,
+telemetry preserved zero regret but saved only 1.426% online H20-hours versus
+sim top-k + real final.  The current route is closed; see
+`docs/fidelity-aware-harness-p1-report-20260714.md`.
+
+## Simulator shortlist lower bound
+
+On the frozen 12-cell SimFid task, the strongest calibrated SLO simulator
+reading places TP2/MNS32 and TP2/MNS64 in the same first tie bucket.  Real-final
+evaluation of that two-cell bucket selects TP2/MNS32 and has zero real regret.
+A method requiring a real calibration probe plus final verification cannot beat
+two real cell evaluations on this task.  Therefore “better initial selection”
+is not a viable claim here; the remaining headroom is shorter real verification
+inside the same shortlist.
+
+## Five-second prefix result
+
+The retrospective Phase-6 dataset contains 37 primary anchors across 12 cells.
+Stable labels use the frozen same-placement 2-of-3 adjudication: 28 feasible and
+9 infeasible.  Three TP4 primary measurements disagree with their repeated
+labels, so single-run feasibility is not treated as ground truth.
+
+Using leave-one-cell-out folds, identical L2 logistic models, and a 5-second
+prefix:
+
+| Metric | Outcome-only | Instrumentation-aware | Delta |
+|---|---:|---:|---:|
+| Accuracy | 78.38% | 89.19% | +10.81 pp |
+| Balanced accuracy | 70.63% | 81.55% | +10.92 pp |
+| Brier score | 0.1297 | 0.0901 | -0.0396 |
+| Correct only in this model | 0 | 4 | +4 |
+| McNemar exact two-sided p | — | 0.125 | not significant |
+
+At the frozen conservative threshold 0.95, both policies make zero false
+accepts and zero false rejects on this retrospective set.  Outcome-only safely
+cuts 36.35% of measured primary-trial cost; instrumentation-aware safely cuts
+61.10%, an additional 24.75 percentage points.  Regularization sensitivity for
+accuracy delta is `[0.00, +10.81]` percentage points, so the sign is
+non-negative but the magnitude is not stable.
+
+Longer prefixes do not strengthen the case monotonically.  At 10 seconds,
+headline accuracy is 91.89% outcome-only versus 89.19% instrumentation-aware;
+at 15 seconds it is 88.89% versus 91.67%; at 20 seconds it is 86.11% versus
+91.67%, but both 0.95 policies make one false reject.  Five seconds is therefore
+a training-selected operating point, not a test result.
+
+## Strong simulator-aware calibration baseline
+
+The original nested comparison used the same simulator shortlist but did not
+put Frontier's per-anchor prediction in either model.  A stronger retrospective
+audit now gives both models frozen-calibrated simulated throughput, simulated
+SLO pass rate, and simulated feasibility.  Under the same leave-one-cell-out
+folds, 5-second cutoff, L2 logistic family, regularization 1.0, and threshold
+0.95:
+
+| Metric | Sim + outcome | Sim + outcome + instrumentation | Delta |
+|---|---:|---:|---:|
+| Accuracy | 81.08% | 89.19% | +8.11 pp |
+| Balanced accuracy | 72.42% | 81.55% | +9.13 pp |
+| Brier score | 0.1058 | 0.0957 | -0.0101 |
+| Safe early decisions | 20/37 | 25/37 | +5 |
+| Valid full-trial cost reduction | 50.89% | 68.98% | +18.09 pp |
+| Residual verification H20-hours | 0.5240 | 0.3310 | -36.84% |
+
+Both 0.95 policies have zero false accept and zero false reject on this
+retrospective task.  Only three 0.5-threshold classifications differ in favor
+of instrumentation and none in favor of the strong baseline; McNemar's exact
+two-sided p-value is 0.25.  The cell-bootstrap accuracy-delta interval is
+`[0.00,+18.18]` percentage points.  The result is not robust to regularization:
+at 0.1 the strong baseline is more accurate and the instrumentation policy
+makes two unsafe decisions; at 10.0 the strong baseline is also more accurate.
+Thus the stronger comparison still has enough point-estimate headroom for a
+held-out test, but it materially weakens the evidence and makes a prospective
+task-level result mandatory.
+
+## Interpretation
+
+There is enough headroom to run a held-out pilot, but not enough evidence to
+claim the harness contribution:
+
+- the 5-second cost gap is operationally large;
+- only four paired classifications differ, so significance is absent;
+- all examples share one workload/SLO/engine task;
+- completion timestamps are reconstructed from arrival + TTFT + TPOT rather
+  than recorded directly;
+- three adjudication disagreements are concentrated in transient TP4 runs;
+- outcome-only already recovers the simulator shortlist oracle with very few
+  real cells.
+
+The next experiment must therefore freeze the 5-second model and threshold,
+record exact monotonic completions, use a held-out trace, and label each anchor
+with three full repetitions.  The registered protocol is
+`docs/fidelity-aware-harness-protocol-20260714.md`.
+
+## Artifacts
+
+- `runs/fidelity-headroom/analyze_existing.py`
+- `runs/fidelity-headroom/metrics.json`
+- `runs/fidelity-headroom/analyze_prefixes.py`
+- `runs/fidelity-headroom/prefix-metrics.json`
+- `runs/fidelity-headroom/test_analysis.py`
+- `runs/fidelity-headroom/test_prefix_analysis.py`
+- `runs/fidelity-headroom/analyze_strong_baseline.py`
+- `runs/fidelity-headroom/strong-baseline-metrics.json`
+- `runs/fidelity-headroom/test_strong_baseline.py`
+
+## Sanity block
+
+| Family | n | Min | Max | Distinct | Invariant/result |
+|---|---:|---:|---:|---:|---|
+| Real SimFid cell scores | 12 | 1.2833 | 3.2833 | 7 | Non-negative; not identical |
+| Prefix examples at 5 s | 37 | 5 s | 5 s | 1 expected | All 12 cells represented |
+| Adjudicated labels | 37 | 0 | 1 | 2 | 28 positive / 9 negative |
+| Primary/adjudicated disagreement | 37 | 0 | 1 | 2 | 3 TP4 disagreements retained |
+| Full primary elapsed time | 37 | 14.566 s | 62.064 s | 37 | Every 5 s prefix is in range |
+| Outcome probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
+| Instrumentation probability | 37 | in `[0,1]` | in `[0,1]` | >1 | Checked before metrics |
+| Layer-1 streams | 12 | 14,174 records | 58,725 records | 12 | Contiguous, zero drops |
+| Matched frozen simulator anchors | 37 | pass rate 0.0688 | pass rate 1.0 | 12 pass-rate values | Every prefix matched exactly once |
+| Frozen simulator anchor corpus | 92 | positive throughput | positive throughput | >1 | No duplicate cell/anchor run |
+
+Checked invariants: same folds/model family and cutoff; no full verdict in a
+feature; prefix-only Layer-1 slicing; non-negative costs/counters; bounded
+ratios/probabilities; both labels present; per-config results not identical;
+tie expansion before top-k; no imputation of non-monotonic frontiers.  The main
+limitation is reconstructed request completion time, explicitly marked on all
+37 five-second examples.

docs/fidelity-aware-harness-p1-report-20260714.md

@@ -0,0 +1,242 @@
+# Fidelity-aware harness P1 result
+
+Status: **REGISTERED ROUTE REJECTED; DO NOT OPEN P2/P3 FOR THE CURRENT METHOD**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Outcome
+
+The registered five-second instrumentation-aware verifier did not pass P1.
+The stronger simulator-aware comparison also failed the independent
+contribution bar.  On the frozen `k=2` end-to-end replay:
+
+- `sim top-k + real final` selected the real oracle with zero regret;
+- instrumentation-aware also selected the oracle, but reduced online H20-hours
+  by only **1.426%** (1.329% when the prior failed attempt is added to both);
+- the required reduction was 30% versus full real final and 20% versus a safe
+  outcome-only calibrator;
+- the outcome-only calibrator was not safe: it rejected the true best cell, so
+  its apparent cost saving is not a deployable comparison.
+
+This rejects the claim that the **current joint logistic verifier**, trained on
+one historical workload, gives the harness an independent tuning contribution.
+It does not prove that engine telemetry contains no useful signal.  Telemetry
+improved held-out classification and removed unsafe decisions, but did not turn
+that signal into meaningful end-to-end tuning-cost reduction.
+
+## Frozen setup
+
+- Host: `dash0`, 8 NVIDIA H20 GPUs; cells were serialized and used TP1, TP2,
+  or TP4 without co-resident serving jobs.
+- Engine/model: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16.
+- Workload: held-out `chat_w20260312_1000`, seven disjoint repeat bands,
+  60-second replay after 0.1 time scaling, input `[0,8192]`, exactly 128 output
+  tokens.
+- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, request pass rate at least 0.95.
+- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
+- Per cell: burn-in, three low-rate repeats, and three high-rate repeats.  The
+  first repeat supplied the five-second prefix; 2-of-3 supplied its label.
+- Models: the registered pair used config/workload/outcome versus the same
+  vector plus Layer-1 engine telemetry.  The strengthened pair additionally
+  gave both models identical frozen Frontier throughput, SLO pass-rate, and
+  feasibility predictions.
+- Policy: accept at `p>=0.95`, reject at `p<=0.05`, otherwise continue the same
+  trial.  Model, cutoff, threshold, role order, request hashes, and cap were
+  frozen before their applicable evaluation.
+
+The first launch failed its warm-up input-count validation before a measured
+anchor.  It cost 0.020552 H20-hours.  The corrected primary attempt cost
+1.722112 H20-hours, so aggregate campaign cost was **1.742664 H20-hours**, below
+the 3.5 cap.  The fix changed only warm-up validation; formal request counts and
+hash checks were unchanged.
+
+## P1 labels are not an artificial easy split
+
+The 12 adjudicated anchor labels contain 7 feasible and 5 infeasible examples.
+They are not simply “low feasible, high infeasible”:
+
+- TP2/MNS64 high was feasible in all three repeats;
+- TP4/MNS64 low and high were feasible in all six repeats;
+- TP4/MNS16 low and high were infeasible in all six repeats.
+
+That last pair creates a large real MNS interaction under an otherwise matched
+TP4 configuration.  Frontier correctly predicted TP4/MNS64 high as feasible,
+but incorrectly predicted TP4/MNS16 low as feasible.  It also incorrectly
+predicted TP1/MNS64 high as feasible.  Overall simulator-only feasibility was
+10/12 correct: 83.33% accuracy, with two false-feasible predictions and no
+false-infeasible prediction.
+
+The two false-feasible cases expose the intended latent-state problem.  At five
+seconds, all 26 completed TP4/MNS16-low requests and all 9 completed
+TP1/MNS64-high requests still passed their SLO, although both full anchors were
+infeasible.  External outcomes had not yet exposed the future failure; queue,
+running-batch, and scheduler state existed before the tail outcome.  This is
+mechanistic evidence that instrumentation can be useful, not evidence that the
+current learned policy uses it well enough.
+
+## Registered and strengthened prefix results
+
+At the frozen 0.95 policy threshold:
+
+| Comparison | Accuracy | Balanced acc. | Early decisions | False accept | False reject | Valid primary-trial saving |
+|---|---:|---:|---:|---:|---:|---:|
+| Registered outcome-only | 41.67% | 50.00% | 6/12 | 0 | 2 | invalid |
+| Registered + telemetry | 66.67% | 71.43% | 4/12 | 0 | 0 | 11.44% |
+| Strong sim + outcome | 66.67% | 68.57% | 5/12 | 0 | 1 | invalid |
+| Strong sim + outcome + telemetry | 83.33% | 85.71% | 4/12 | 0 | 0 | 11.44% |
+
+For the strong pair, telemetry was correct on two examples where the baseline
+was wrong and lost none; McNemar's exact two-sided p-value is 0.5 at `n=12`.
+This is a safety/classification improvement, not a cost contribution.  The
+registered instrumentation policy made two fewer early decisions than its
+baseline, so it failed the registered `+3 decisions or +15 percentage points`
+incremental gate.
+
+The result is not robust to the frozen regularization sensitivity:
+
+| L2 lambda | Sim+outcome acc. | +telemetry acc. | Base policy errors | Telemetry policy errors | Base saving | Telemetry saving |
+|---:|---:|---:|---:|---:|---:|---:|
+| 0.1 | 41.67% | 75.00% | 4 | 2 | invalid | invalid |
+| 1.0 | 66.67% | 83.33% | 1 | 0 | invalid | 11.44% |
+| 10.0 | 83.33% | 83.33% | 0 | 0 | 0.00% | 5.98% |
+
+Consequently the positive classification delta is neither statistically nor
+hyperparameter robust.
+
+## End-to-end shortlist result
+
+Frontier's simulator-feasible ranking on the tested P1 surface was:
+
+| Rank | Cell / anchor | Sim throughput/GPU | Real feasible | Real offered goodput/GPU |
+|---:|---|---:|---:|---:|
+| 1 | TP4/MNS64 high | 3.0718 | yes | 3.1250 |
+| 2 | TP1/MNS64 high | 2.8823 | no | 2.9833 |
+| 3 | TP2/MNS64 high | 2.8096 | yes | 2.8750 |
+| 4 | TP4/MNS16 low | 2.0866 | no | 2.1250 |
+| 5 | TP1/MNS8 low | 1.9806 | yes | 2.0333 |
+| 6 | TP2/MNS8 low | 1.8637 | yes | 1.9083 |
+
+The real oracle was TP4/MNS64 high at 3.125 req/s/GPU.  Cost includes an
+inferred per-cell startup/warm-up/burn-in/cleanup component and the selected
+anchor; benchmark-only 2-of-3 annotation intervals are removed.  Gaps around
+annotation runs remain in the shared setup term, making this a conservative
+method-cost estimate.
+
+| Frozen policy (`k=2`) | Online H20-hours | + prior failure | Real regret | Safety | Saving vs full |
+|---|---:|---:|---:|---|---:|
+| Sim top-2 + real final | 0.281383 | 0.301935 | 0.00% | valid | — |
+| Sim + outcome prefix | 0.214664 | 0.235216 | no selected cell | 1 false reject | invalid |
+| Sim + outcome + telemetry | 0.277370 | 0.297922 | 0.00% | valid | **1.426%** |
+
+For `k=1`, the simulator already picked the oracle and telemetry abstained, so
+its saving was exactly 0%.  For `k=3`, telemetry remained correct but saved
+1.007%.  The result therefore does not depend on an unfavorable choice of `k`.
+
+Across all six cells, a generous replay that removes benchmark-only repeat
+intervals estimates 0.944244 H20-hours for the full online workflow.  The
+instrumentation policy saved 0.043467 H20-hours: 11.44% of primary trial time,
+but only **4.603%** of end-to-end online cost.  Even a post-hoc oracle symmetric
+threshold cannot make the current telemetry model reach the contribution bar;
+its best zero-error envelope saves at most 5.69% of inferred online cost.  A
+strong outcome-only model at another post-hoc regularization/threshold can save
+16.13%, which further prevents attributing a unique advantage to telemetry.
+These oracle-threshold numbers are diagnostics only and are not test evidence.
+
+## Why the learned verifier did not generalize
+
+The training corpus has only 37 anchors from one workload/SLO task.  P1 shows
+large covariate shift:
+
+- sim+outcome: 12/192 feature values exceed 3 training standard deviations and
+  4 exceed 5; maximum absolute z-score is 10.36;
+- sim+outcome+telemetry: 19/396 exceed 3 and 9 exceed 5;
+- the largest shifts include admitted input-length mean (10.36), waiting state
+  (7.77), running maximum (6.38), and decode-batch maximum (6.08).
+
+Coefficient attribution shows that the input-length feature dominates several
+wrong feasible-anchor logits.  Because all training examples share one task,
+the joint classifier can learn incidental within-task correlation and override
+a correct simulator prior on TP2/MNS64-high and TP4/MNS64-high.  This is a
+supported diagnosis of model/data insufficiency; it is not a causal proof that
+one feature alone caused the P1 failure.
+
+More importantly, retuning lambda, threshold, features, or cutoff on P1 and
+then calling P1 a held-out result would violate calibration/evaluation
+separation.  P1 may now be used only as development data.
+
+## Decision and the only defensible reopening condition
+
+Do not run registered P2/P3 with the current model.  It failed the predeclared
+gate on the favorable primary-trial denominator and is even farther from the
+bar under end-to-end cost.  Spending six-task headline GPU budget on the same
+method would be metric shopping, not replication.
+
+A new route may be opened only as a new hypothesis:
+
+1. Replace the joint classifier with a **simulator-residual verifier**.  The
+   simulator prediction remains an explicit prior; nested outcome-only and
+   telemetry models learn when that prior is wrong, rather than freely
+   relearning feasibility and overriding it under workload shift.
+2. Train on multiple complete workload/SLO tasks.  SLO thresholds and target
+   pass rate must be explicit inputs; splits are by complete task.
+3. Calibrate abstention with task-level risk control.  No threshold is selected
+   on a headline task, and “never early decide” is included as the safe
+   outcome-only baseline.
+4. Treat Phase 6 and P1 as development only, freeze the residual architecture,
+   features, cutoff, threshold, simulator reading, and `k`, then use entirely
+   new trace windows for a new gate.
+
+This reopening is justified only if development data show both (a) the
+simulator's errors are predictable from pre-outcome engine state and (b) a
+simulator-preserving residual model does not corrupt correct simulator
+predictions.  It is a new project decision, not a continuation automatically
+authorized by P1.
+
+## Benchmark audit
+
+| Audit item | Verdict | Severity | Evidence / disposition |
+|---|---|---|---|
+| Calibration set separate from P1 | PASS | — | Phase 6/0311 trained; P1/0312 tested |
+| Strong simulator-aware baseline | PASS | — | Identical Frontier features in both nested models |
+| Sim top-k + real-final E2E baseline | PASS | — | Frozen `k=2`, tie expansion, measured setup/continuation cost |
+| Multiple independent headline tasks | NEEDS EVIDENCE | Blocking for a positive claim | P1 gate failed; P2 correctly not opened |
+| Statistical significance | NEEDS EVIDENCE | Blocking for a positive claim | n=12 anchors from one task; McNemar p=0.5 |
+| Hyperparameter robustness | FAIL | Blocking | Lambda sensitivity changes safety and relative result |
+| Full resource accounting | PASS for P1 | — | Failures, startup/warm-up/burn-in, continuation and annotation separated |
+| Avoid post-test retuning | PASS only if route stops | Blocking if violated | P1 is now development-only |
+| Selective winning-workload reporting | PASS | — | Negative P1 and TP/MNS losing cases retained |
+
+Overall recommendation: **Block the current independent harness contribution
+claim.**
+
+## Artifacts
+
+- Registered protocol: `docs/fidelity-aware-harness-protocol-20260714.md`
+- Historical headroom: `docs/fidelity-aware-harness-headroom-20260714.md`
+- Registered P1 analysis: `runs/fidelity-headroom/analyze_pilot.py`
+- Strong P1 analysis: `runs/fidelity-headroom/analyze_strong_pilot.py`
+- E2E shortlist replay: `runs/fidelity-headroom/analyze_pilot_e2e.py`
+- External immutable result root:
+  `/home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714`
+
+## Data sanity block
+
+| Data | n | Min | Max | Distinct | Invariant |
+|---|---:|---:|---:|---:|---|
+| P1 labels | 12 | 0 | 1 | 2 | 7 feasible / 5 infeasible |
+| Primary elapsed seconds | 12 | 19.448 | 61.435 | 12 | Every five-second prefix is in range |
+| Prefix Layer-1 records | 12 | 332 | 557 | 12 | Contiguous; zero drops |
+| Exact timestamped outcomes | 12 anchors | 54 | 750 | 11 | Monotonic completion timestamps |
+| Simulator pass rate | 12 | 0.1548 | 1.0 | 7 | Ratios in `[0,1]` |
+| Strong nested probabilities | 24 | 0.000208 | 0.809422 | 24 | Ratios in `[0,1]` |
+| E2E cost components | 36 | 0.001389 | 0.169653 H20-h | 21 | Non-negative |
+| GPU attempts | 2 | 0.020552 | 1.722112 H20-h | 2 | Aggregate 1.742664 < 3.5 |
+| Copied raw files | 191 | — | 153,093,348 bytes total | — | Remote/local aggregate SHA identical |
+
+Checked invariants: six cells and twelve anchors; exact request count and
+request-ID/arrival/length hashes; all cell validation flags true; both labels
+present; probabilities bounded; costs and counters non-negative; simulator
+results not all identical; committed simulator rerun 12/12 numerically
+identical to the exploratory run; no prompt text in public simulator fixtures;
+no co-resident serving process; final eight GPUs at 0 MiB and 0% utilization.
+No red flag remains.

docs/fidelity-aware-harness-protocol-20260714.md

@@ -0,0 +1,231 @@
+# Fidelity-aware real-verification harness protocol
+
+Status: **P1 FAILED; P2/P3 CLOSED FOR THE REGISTERED METHOD; CONTRIBUTION NOT ESTABLISHED**.
+
+Date frozen: 2026-07-14 (Asia/Singapore).
+
+Post-run disposition (2026-07-14): P1 completed with valid data but failed its
+registered incremental gate.  The strengthened simulator-aware comparison and
+end-to-end `k=2` replay also failed: instrumentation was safe and retained zero
+regret, but reduced online H20-hours by only 1.426% versus sim top-k + real
+final, against the 30% bar.  Outcome-only was unsafe.  P2/P3 are therefore not
+opened for this model.  Full results and the permitted reopening condition are
+in `docs/fidelity-aware-harness-p1-report-20260714.md`; the protocol below is
+retained unchanged as the pre-run record.
+
+## Research question and contribution bar
+
+The harness has an independent systems contribution only if engine-internal
+instrumentation improves a tuning decision beyond what is already achievable
+with a simulator shortlist and external benchmark outcomes.  The intended
+claim is therefore deliberately stronger than “telemetry explains a run”:
+
+> Given the same simulator ranking, the same candidate order, and the same
+> short real-GPU probe, a learned instrumentation-aware verifier reaches a
+> configuration with at most 5% real SLO-goodput regret using materially fewer
+> H20-hours than both (a) simulator top-k followed by full real evaluation and
+> (b) an outcome-only verifier given exactly the same probe.
+
+The paper-facing gate is:
+
+- at least 20% lower real-verification H20-hours than outcome-only calibration;
+- at least 30% lower real-verification H20-hours than simulator top-k plus full
+  real final evaluation;
+- paired 95% task-bootstrap confidence interval for the outcome-only cost
+  reduction strictly above zero;
+- selected-configuration SLO-goodput regret at most 5% on every headline task;
+- no false-safe early accept in the pilot and at most 1% in the expanded suite;
+- profiling, warm-up, confirmation, instrumentation, and failed-run costs are
+  included rather than amortized away.  An amortized profile-cost view may be
+  reported only as a secondary result.
+
+If these conditions fail, instrumentation remains a debugging facility.  It is
+not an independent tuning-harness contribution.
+
+## What is learned, and what is not a rule
+
+The decision target is a stable, repeated real verdict, not a hand-authored
+diagnosis such as “queue length above N means reject.”  Each anchor receives
+three full real repetitions and a frozen 2-of-3 feasibility label.  A nested
+pair of regularized models predicts that label from a fixed prefix:
+
+- **Outcome-only input X:** configuration, offered rate, admitted/completed
+  progress, observed TTFT/TPOT margins, failures, and known workload lengths.
+- **Instrumentation input Z:** the same X plus generic engine state: running and
+  waiting queues, decode-batch shape, KV usage, graph mode and padding, prefill
+  share, preemptions, and model-step rate.
+
+Both models use the same L2 logistic family, train split, standardization,
+regularization, cutoff, and probability threshold.  The only experimental
+difference is Z.  The initial family is intentionally simple: a positive result
+then demonstrates value in the engine signal rather than capacity in a larger
+learner.  A sequence model is admissible only as a later, paired ablation.
+
+### Amendment A1: strengthen the calibration baseline before P2
+
+Frozen 2026-07-14 13:08 Asia/Singapore, after P1 launch but before P1
+completion or analysis.  A baseline audit found that the first frozen P1
+models use the simulator only to define candidate order; their feature vectors
+do not contain the simulator's per-anchor prediction.  This is insufficient
+for the stronger term **outcome-only calibration**.  P1 therefore remains a
+prospective test of the originally frozen cross-workload predictor, but cannot
+by itself open a contribution claim.
+
+For P2/P3, both nested models must additionally receive the identical frozen
+simulator outputs available at that decision: predicted completed throughput
+per GPU, predicted SLO pass rate, and predicted feasibility.  The comparison
+is consequently `sim + config + workload + real outcome prefix` versus that
+exact vector plus real engine state.  Simulator features, regularization,
+cutoff, and thresholds are frozen before any P2 task.  If telemetry does not
+improve this stronger baseline, the harness has no independent contribution.
+
+The same audit also separates algorithm cost from benchmark-oracle cost.
+Headline method cost includes every action the method would execute online:
+simulator profiling/calibration, model onboarding, server startup, warm-up,
+real prefix, continuation after abstention, method-requested confirmation,
+logging overhead, failures, and cleanup.  Exhaustive real-oracle runs and the
+extra repetitions used only to construct 2-of-3 evaluation labels are common
+benchmark annotation cost; they are reported separately and charged to no
+method.  A second, deliberately conservative table adds that common cost to
+all methods.  This prevents both hiding real method cost and making the
+percentage gate mathematically depend on offline ground-truth annotation.
+
+The frozen first policy uses a 5-second prefix, L2 regularization 1.0, and a
+two-sided abstaining threshold of 0.95: accept at `p(feasible)>=0.95`, reject at
+`p(feasible)<=0.05`, otherwise continue the exact same trial to completion.
+Threshold and cutoff were selected on the historical training task and are
+therefore not evidence; all claims come from subsequent held-out tasks.
+
+## Fair baselines
+
+| Method | Simulator | 5-second real prefix | External outcomes | Engine state | Full real continuation |
+|---|---:|---:|---:|---:|---:|
+| Real-only oracle | no | no | full | optional diagnostic | every candidate/anchor |
+| Sim top-k + real final | yes | included in full run | full | no decision use | every shortlisted candidate/anchor |
+| Outcome-only calibration | yes, including its prediction features | yes | yes | no | only on abstention |
+| Instrumentation-aware | same prediction features | yes | yes | yes | only on abstention |
+
+Tie buckets are expanded before top-k.  `k` is selected on training tasks and
+is fixed on held-out tasks; an oracle per-task k is forbidden.  Outcome-only
+receives all information available outside the engine, including config,
+workload, and frozen simulator-prediction features.  Instrumentation cannot use
+any record submitted after the cutoff.  The full label, confirmation votes,
+realized simulator error, and later requests are never model features.
+
+## Staged experiment
+
+### R0: historical premise and headroom audit
+
+The frozen SimFid surface has 12 cells.  The strongest calibrated SLO simulator
+reading has a top tie bucket `{TP2/MNS32, TP2/MNS64}`; full real evaluation of
+those two cells already finds the oracle with zero regret.  Consequently this
+single task cannot demonstrate a selection-count advantage: any method needing
+one real calibration probe and one real final verification has a lower bound of
+two real cells.
+
+The viable estimand is instead the duration and number of full real frontier
+evaluations inside a fixed shortlist.  Historical Phase-6 prefixes are analyzed
+only as training/premise data.  Their request completion times are reconstructed
+from arrival, TTFT, TPOT, and token count, so they cannot support a final claim.
+
+### P1: exact-timestamp prospective pilot
+
+- Engine/model/hardware: patched vLLM 0.24.1.dev3, Qwen3-30B-A3B, one solo
+  server/client on dash0, NVIDIA H20, `TP in {1,2,4}`.
+- Held-out workload: `chat_w20260312_1000`, 60-second replay after the frozen
+  0.1 time scale, raw input `[0,8192]`, exactly 128 output tokens.
+- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, 95% request pass rate.
+- Cells: TP1/MNS8, TP1/MNS64, TP2/MNS8, TP2/MNS64, TP4/MNS16, TP4/MNS64.
+- Per cell: one attainable low offered rate near 0.85x the historical v0.24
+  frontier and one high rate near 1.25x.  The exact threshold and selected
+  request hashes are frozen by a CPU preflight before launch.
+- Each cell uses a fresh server, the accepted long-request warm-up, one
+  unmeasured full-window burn-in, then three repetitions per rate.  Rate order
+  alternates and reverses across cells to prevent a fixed warm-state/order
+  confound.
+- The first repetition supplies the exact prefix.  All three repetitions supply
+  the 2-of-3 label.  Every request records a monotonic completion timestamp;
+  Layer-1 records are cut at the same monotonic boundary.
+- Placement is serialized.  Co-location is forbidden because Phase 6 observed
+  up to 92.86 percentage-point pass-rate shifts under co-location.
+- Hard cap: 3.5 H20-hours, including startup, warm-up, burn-in, all repetitions,
+  failures, and cleanup.  Projected cap violation stops before the next cell.
+
+P1 opens P2 only if all data invariants pass and instrumentation-aware has zero
+false accept/reject, is no worse than outcome-only, and either makes at least
+three additional correct early decisions or improves total valid trial-cost
+reduction by at least 15 absolute percentage points.  The pilot is a gate, not
+paper evidence.
+
+### P2: held-out task replication
+
+If P1 passes, freeze the model and run at least six independent task groups:
+three trace windows spanning distinct date/slot combinations and two SLO
+regimes.  No task used for threshold/model selection enters the headline test.
+The candidate surface is the full 12-cell `TP={1,2,4} x MNS={8,16,32,64}`
+surface.  Splits are by complete task, never by anchor or request.  A task-level
+paired bootstrap (10,000 repetitions, fixed seed) estimates cost and regret
+intervals.  Non-monotonic or split 2-of-3 anchors remain explicit; no frontier
+is imputed.
+
+### P3: end-to-end shortlist and search replay
+
+For each P2 task, run the same frozen simulator and tie-expanded top-k policy.
+Replay the real binary/frontier search under all three verification policies:
+full real, outcome-only, and instrumentation-aware.  The policy consumes only
+prefixes that would have been available at that decision point.  Report:
+
+- selected cell and real SLO-goodput regret;
+- number of real cells, anchors, and confirmations;
+- measured H20-hours and wall time;
+- false accept, false reject, and abstention counts;
+- profile, startup/warm-up, probe, full-continuation, confirmation, logging, and
+  failure cost breakdowns.
+
+### P4: simulator-rank-error attribution
+
+This phase distinguishes an outdated implementation/profile from a structural
+simulator limitation.  For each held-out task compare:
+
+1. the original simulator/profile;
+2. a version-matched re-profiled simulator;
+3. a trajectory-conditioned run supplied with the realized arrival and request
+   length sequence;
+4. outcome-only residual calibration;
+5. instrumentation-aware residual calibration.
+
+The engine trace is extended only as needed with a worker-level step UID and
+CUDA-event duration, because current async submit-to-complete spans overlap and
+are not GPU step time.  Residuals are decomposed into operator-profile error,
+scheduler/state error, and run-to-run noise.  If re-profiling alone restores the
+ranking, the old 30% loss was an implementation/profile defect.  If exact
+profiles and realized trajectories still mis-rank cells, and the residual is
+systematically explained by queue/KV/graph/batch state unavailable to the
+simulator, that is evidence of a structural state-abstraction gap.  Correlation
+alone is not called causal.
+
+## Failure modes that reject the route
+
+- Outcome-only matches or beats instrumentation-aware under the same cutoff.
+- Instrumentation gains average accuracy but introduces false-safe decisions.
+- Gains disappear under task-level rather than request/anchor-level splitting.
+- Savings come only from excluding startup, warm-up, profiling, confirmations,
+  or failed trials.
+- A different cutoff/threshold must be selected after seeing each test task.
+- The simulator top-k baseline already reaches the target with equal or lower
+  total H20-hours.
+- Exact instrumentation overhead exceeds 1% throughput or materially changes
+  p95/p99 latency.
+- Results depend on TP4 transient/non-monotonic trials and do not replicate on
+  held-out tasks.
+
+## Data sanity contract
+
+Every analysis ends with n, min/max, distinct count, label balance, and these
+invariants: non-negative counters/costs; probabilities and ratios in `[0,1]`;
+per-config results not all identical; timestamps monotonic; every prefix record
+at or before its cutoff; selected request ID/arrival/length hashes stable across
+repetitions; exact 128-token completion or counted failure; no dropped Layer-1
+records; 2-of-3 labels reproducible; no co-resident GPU process; total H20-hours
+below the hard cap; final GPUs idle.  A red flag is reported first and blocks
+the contribution claim.

docs/intervention-response-v0-protocol-20260714.md

@@ -0,0 +1,71 @@
+# Telemetry intervention-response v0 protocol
+
+Status: **FROZEN BEFORE V0 ANALYSIS**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Claim boundary
+
+The closed residual route asked whether one absolute engine-state snapshot can
+predict unmeasured configurations.  V0 asks a different, narrower question:
+
+> Does an adjacent, controlled MNS intervention produce an early engine-state
+> response that is distinguishable from same-config repeat noise?
+
+Passing this gate only authorizes a matched real-GPU pilot.  It does not prove
+that telemetry improves tuning, that any metric is a causal mediator, or that
+the response transfers to a new workload, topology, or knob family.
+
+## Data and estimand
+
+- Source: Phase 6 solo-authoritative Qwen3-30B-A3B/vLLM 0.24 Layer-1 streams.
+- Action pairs: primary runs at identical study hash, TP, sampling anchor, and
+  request-order hash, with adjacent `MNS={8,16,32,64}` values.
+- Noise pairs: primary versus confirmation at the same complete config,
+  anchor, and request-order hash.  Only primary-to-confirmation pairs are used;
+  confirmations are not combined into pseudo-independent all-pairs.
+- Fixed early windows: 5 seconds and 10 seconds from the measured interval
+  start.  All runs exceed 10 seconds, so early-stop censoring cannot change the
+  telemetry window.
+- Full-run pass rate and feasibility are descriptive only because an early
+  stop can make full elapsed durations differ.
+
+The statistical unit is a run pair.  Scheduler steps are summarized within a
+run and are never counted as independent trials.
+
+## Frozen response gate
+
+The directly measured gate features are scheduler-step rate, decode-batch
+mean, prefill-token fraction, waiting/running queue mean, KV-usage mean, and
+CUDA-graph padding fraction.
+
+A feature qualifies at one horizon only if:
+
+1. at least 75% of nonzero action deltas have the same sign;
+2. median absolute action delta is at least 2x the median absolute repeat
+   delta; and
+3. at least 50% of action deltas exceed the repeat-noise absolute p95.
+
+V0 opens a GPU pilot only if:
+
+- there are exactly 17 frozen adjacent-MNS action pairs;
+- there are at least 20 primary/confirmation repeat pairs;
+- all identity, finite-value, counter, and ratio invariants pass; and
+- at least two gate features qualify at both 5 and 10 seconds.
+
+Any data red flag stops the analysis before interpreting the response.
+
+## If V0 passes
+
+Register a dash0 pilot around a known scaling knee.  The pilot must use the
+same request sequence and arrival times, one serving job at a time, one changed
+knob, randomized `A/B` versus `B/A` order, common non-censored measurement
+windows, and trial-level repetitions.  It must compare a response-aware next
+action against an outcome-only policy under complete startup, warm-up, and
+H20-hour accounting.
+
+## If V0 fails
+
+Do not add telemetry fields or train a larger model.  The current Layer-1 state
+does not identify even an MNS intervention above repeat noise on this task, so
+the telemetry-guided tuning route remains diagnostic only.

docs/intervention-response-v0-results-20260714.md

@@ -0,0 +1,157 @@
+# Telemetry intervention-response v0/v1 results
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Decision
+
+**STOP before a new H20 pilot.**  The current Layer-1 aggregate telemetry does
+not identify a sufficiently general early response to an MNS intervention,
+and it does not improve action-efficacy prediction over exact external prefix
+outcomes on the available development tasks.
+
+This is a negative result about the present state representation and
+experiment design.  It does not establish that engine telemetry is useless for
+tuning, and it is not held-out evidence.
+
+## Hypothesis and frozen test
+
+The tested hypothesis was:
+
+> With the workload and all non-MNS settings held fixed, increasing MNS causes
+> a 5--10 second engine-state response that is larger than same-config repeat
+> noise and that predicts whether the action makes the full run feasible.
+
+A response feature had to satisfy all three frozen conditions at both 5 and
+10 seconds: at least 0.75 sign consistency, median absolute action effect at
+least 2x the repeat median, and at least 0.50 of action deltas above the repeat
+absolute p95.  At least two features had to pass.  A telemetry feature was
+decision-relevant only if its leave-one-repeat-out balanced accuracy was at
+least 0.75 and at least 0.15 above the best exact external prefix outcome.
+
+## What was implemented
+
+- A common-window analyzer over the existing per-scheduler-step Layer-1 stream.
+- Exact action pairing with request-order hash, offered load, TP, load role,
+  and repetition held fixed.
+- Same-config repeat-noise estimation without treating scheduler steps as
+  independent samples.
+- Exact 5/10-second request-prefix outcomes using monotonic completion times.
+- A one-feature leave-one-repeat-out efficacy audit; no multivariate model was
+  fitted to the 12 examples.
+- Input hashes, stream hashes, frozen thresholds, pair-level deltas, and sanity
+  invariants in machine-readable audit artifacts.
+- Trial-by-trial validation against the P1 manifest, plus content hashes for
+  every result, request file, and Layer-1 stream.
+
+## Experiment A: Phase-6 retrospective audit
+
+Phase 6 supplied 17 adjacent-MNS actions and 29 same-config
+primary/confirmation pairs.  No feature passed at either horizon, producing
+`STOP_NO_IDENTIFIABLE_RESPONSE`.
+
+The confirmation sample is not a clean replication distribution: confirmations
+were selectively run after disputed primary outcomes.  Several same-config
+pairs consequently followed radically different trajectories.  This result
+therefore remains a valid failure of the frozen v0 gate, but it cannot by itself
+separate normal run variance from confirmation-selection bias.
+
+## Experiment B: prospective-repeat confirmation
+
+P1 supplied three pre-arranged, disjoint request bands for every cell/load.
+Exact matched actions exist for TP1 `MNS 8 -> 64` and TP4 `MNS 16 -> 64`, at
+low/high load and repetitions 1/2/3.  This yields 12 action pairs and 24
+same-config consecutive-repeat pairs.
+
+The 24 adjacent repeat differences share their middle repetition within each
+three-run group.  They define a conservative empirical noise reference; they
+are not used as 24 independent samples in an inferential test.
+
+The result is `STOP_NO_PROSPECTIVE_RESPONSE`: zero features passed the response
+gate at either horizon.
+
+The strongest response was mean waiting-queue occupancy:
+
+| Horizon | Sign consistency | Action/repeat median | Action above repeat p95 | Gate |
+|---|---:|---:|---:|---|
+| 5 s | 1.000 | 1.292x | 0.167 | fail |
+| 10 s | 1.000 | 2.611x | 0.250 | fail |
+
+The direction is real enough to merit diagnosis, but the effect is not broad
+enough to guide a general action.  It is large for TP4/high-load trials and
+small or absent in other regimes.
+
+Full-run transitions contain six beneficial actions (`false -> true`) and six
+non-beneficial actions (three `false -> false`, three `true -> true`).  The
+beneficial label is also perfectly confounded with TP4 in this small dataset,
+so it cannot support a topology-general claim.
+
+| Horizon | Best telemetry delta | Balanced accuracy | Best external prefix delta | Balanced accuracy | Telemetry advantage |
+|---|---|---:|---|---:|---:|
+| 5 s | waiting queue | 0.750 | max TPOT / SLO | 0.833 | -0.083 |
+| 10 s | waiting queue | 0.750 | outstanding / admitted | 0.750 | 0.000 |
+
+No telemetry feature reaches the preregistered `+0.15` incremental threshold.
+
+## What this rules out
+
+It rules out using the current vector of 5/10-second global means as a solid
+mechanism for choosing the next config.  In particular, adding these aggregates
+to an LLM prompt or fitting a larger predictor would currently hide, rather
+than solve, the identifiability problem.
+
+It does not rule out an instrumentation-aware tuner built around a deliberately
+excited local system.  The existing runs were designed for endpoint/fidelity
+evaluation, not system identification: the MNS action is large, efficacy is
+confounded with TP, repeat bands contain different requests, and global means
+erase when queue buildup or service-rate changes occur.
+
+## Required redesign before spending H20-hours
+
+The next admissible experiment is a randomized, local A/B system-identification
+pilot around one fixed TP and one load knee:
+
+1. Replay the exact same request sequence and arrival times for both endpoints.
+2. Use small adjacent actions and randomized `A/B` versus `B/A` order.
+3. Record event-aligned response curves, including queue growth/drain rate,
+   prefill/decode service rate, and per-step service time, rather than only one
+   global mean.
+4. Separate a mechanism gate (repeatable response) from the end-to-end gate:
+   fewer trials or H20-hours to select a feasible near-optimal config than an
+   outcome-only tuner.
+5. Hold out a second load/workload for the final policy comparison.
+
+Until that design is frozen, a wider sweep would only generate more correlated
+observations and is not justified by the evidence above.
+
+## Reproduction
+
+```bash
+python3 runs/intervention-response-v0/test_analysis.py
+python3 runs/intervention-response-v0/test_p1_analysis.py
+
+python3 runs/intervention-response-v0/analyze_phase6.py \
+  --metrics runs/opprof-phase6/phase6/metrics.json \
+  --raw-root runs/opprof-phase6/phase6/solo-authoritative/cells \
+  --output runs/intervention-response-v0/phase6-audit.json
+
+python3 runs/intervention-response-v0/analyze_p1.py \
+  --run-root /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b \
+  --manifest /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b/pilot-manifest.json \
+  --output runs/intervention-response-v0/p1-audit.json
+```
+
+## Data sanity
+
+- Phase 6: action pairs `n=17`, repeat pairs `n=29`, trials `n=66`; MNS
+  action size min/max `8/32`, `3` distinct; action-state vectors `n=17`, `17`
+  distinct; streams `n=12`, bytes min/max `12,745,297/52,957,710`, `12`
+  distinct.
+- P1: action pairs `n=12`, repeat pairs `n=24`, trials `n=36`; MNS action
+  size min/max `48/56`, `2` distinct; efficacy labels `n=12`, min/max `0/1`,
+  `2` distinct; streams `n=6`, bytes min/max `17,449,143/29,431,988`, `6`
+  distinct.
+- Checked invariants: exact action request hashes and offered loads match;
+  all `36/36` P1 trials match the manifest; expected pair counts hold; all
+  deltas are finite; non-negative counters and bounded ratios hold; per-config
+  state vectors are not all identical; both efficacy classes are present.  No
+  red flags were observed.

docs/intervention-response-v1-p1-protocol-20260714.md

@@ -0,0 +1,58 @@
+# Intervention-response v1 prospective-repeat confirmation
+
+Status: **FROZEN AFTER PHASE-6 V0 FAILURE AND BEFORE P1 RESPONSE ANALYSIS**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Why this is a new confirmation, not a relaxed V0
+
+Phase-6 V0 failed its frozen global response gate.  Its 29 same-config
+confirmations were triggered after disputed outcomes, and the resulting noise
+sample contains extreme trajectory divergence by construction.  V0 remains
+failed and its thresholds are unchanged.
+
+The already-completed P1 campaign supplies a distinct test: three
+prospectively scheduled, disjoint repeat bands for every cell/load.  TP1 and
+TP4 use identical offered loads and exact request-order hashes across their MNS
+endpoints.  V1 asks whether an MNS response is identifiable against this
+prospective workload-repeat noise, and whether that response predicts action
+efficacy beyond exact external prefix outcomes.
+
+P1 is now development data.  No result here is held-out or paper-facing.
+
+## Frozen pairs
+
+- Action pairs: TP1 `MNS 8 -> 64` and TP4 `MNS 16 -> 64`, at low/high load and
+  repeat 1/2/3.  Endpoints must have identical TP, offered rate, repeat role,
+  and request-order hash. Expected `n=12`.
+- Repeat-noise pairs: consecutive pre-arranged repeat bands within each of six
+  cells and low/high load: `rep1 -> rep2`, `rep2 -> rep3`. Expected `n=24`.
+  Repeat bands intentionally contain different requests and therefore include
+  workload-sampling noise rather than pretending to be identical trials.
+  Adjacent differences share the middle run; the gate uses their empirical
+  magnitude only and does not treat the 24 differences as independent samples
+  for a p-value or confidence interval.
+- Prefix horizons: 5 and 10 seconds. Exact monotonic request completion times
+  and the same Layer-1 intervals are used.
+
+## Frozen gates
+
+The response-identifiability thresholds are exactly the Phase-6 V0 thresholds:
+75% sign consistency, 2x median effect/repeat noise, and at least 50% of action
+deltas above repeat absolute p95. At least two response features must qualify
+at both horizons.
+
+Action efficacy is one only for an infeasible-to-feasible full-run transition.
+The 12 action pairs must contain at least four examples of each class.
+
+For decision relevance, each individual external-outcome response feature and
+each individual telemetry-response feature is evaluated by leave-one-repeat-
+band-out threshold fitting. This intentionally avoids a multivariate model on
+12 examples. At least one telemetry feature must, at both horizons:
+
+1. reach balanced accuracy at least 0.75; and
+2. exceed the best external-outcome response feature by at least 0.15.
+
+Only if data validity, response identifiability, and incremental decision
+relevance all pass does V1 open a newly registered matched GPU pilot. No
+threshold or feature is changed after observing V1.

docs/intervention-response-v2-phase-aware-protocol-20260714.md

@@ -0,0 +1,94 @@
+# Phase-aware telemetry intervention-response v2 protocol
+
+Status: **INVALID OPERATIONAL ATTEMPT; SUPERSEDED BY V3**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+The first MNS=16 session timed out while draining the 3.125 requests/s/GPU
+workload after its 300-second arrival window.  It produced no high-load result,
+and no MNS=64 endpoint was run.  No comparative conclusion is drawn from this
+attempt; see `intervention-response-v3-two-load-protocol-20260714.md`.
+
+## Correction to v0/v1
+
+The 5/10-second analyses tested an ultra-early verifier.  They did not test
+whether telemetry observed after the engine has developed queue, batch, and KV
+state can guide tuning.  The P1 replay lasts 60 seconds after time scaling, and
+the 5/10-second prefixes contain only a small fraction of its requests.
+
+V2 therefore replaces absolute cutoffs with replay phase.  The old audits and
+their negative decisions remain immutable, but their claim is narrowed to the
+first 5/10 seconds.
+
+## Historical corrective audit
+
+The historical audit is development-only and cannot become confirmatory after
+the horizon concern was observed.
+
+- Infer each trial's intended replay duration as selected requests divided by
+  offered requests per second.  All trials must agree.
+- Find every complete 10% replay decile supported by every trial.  Analyze all
+  such deciles; selecting only the best horizon is forbidden.
+- At each decile report both:
+  - cumulative state from replay start to the checkpoint; and
+  - the non-overlapping 10%-wide state block ending at the checkpoint.
+- Report admitted/completed request coverage, response-versus-repeat statistics,
+  telemetry versus external-outcome efficacy, and per-feature trajectory drift.
+- Reuse the frozen v1 action and repeat pairs and the frozen response and
+  incremental-efficacy thresholds.  These thresholds are descriptive in V2;
+  passing one post-hoc horizon does not open a contribution claim.
+
+If early stopping prevents complete observation of the replay phases, the
+historical decision is `REQUIRES_UNCENSORED_PHASE_AWARE_PILOT`, independent of
+which early decile looks best.
+
+## Uncensored matched pilot
+
+The pilot is a mechanism gate, not paper evidence.
+
+- Hardware/engine/model: solo placement on dash0, 4 NVIDIA H20 GPUs, patched
+  vLLM `0.24.1.dev3+opprof`, Qwen3-30B-A3B, fixed `TP=4`.
+- Action: `MNS 16 -> 64`; topology, model, engine build, workload, arrival
+  sequence, offered load, and all other settings remain fixed.
+- Workload: `chat_w20260312_1000` at replay-time scale `0.5`, hence 300 seconds.
+- Offered loads per GPU: `1.5`, `2.125`, and `3.125` requests/s.  These supply a
+  low control and the two already-observed P1 pressure regimes.
+- Repetitions: three disjoint session bands, exact request sequence matched
+  across action endpoints.  Endpoint order alternates `A/B`, `B/A`, `A/B`;
+  load order is counter-rotated across repetitions.
+- A fresh server receives the accepted long-request warm-up and a bounded
+  burn-in before each measured session.
+- SLO-unrecoverable early stop is disabled.  Every run must observe the full
+  300-second arrival window; a separate 360-second safety deadline may mark a
+  run invalid but cannot manufacture a full-run label.
+- Cumulative checkpoints: 10%, 25%, 50%, 75%, and 100%, or 30/75/150/225/300
+  seconds.  Quarter blocks are analyzed separately from cumulative means.
+- A measured Layer-1 interval is complete only when its start-boundary,
+  end-boundary, and maximum internal record gaps are each at most one second;
+  timestamps must be monotonic.
+- Placement is serialized.  Co-location remains forbidden because Phase 6
+  observed material co-location-induced outcome shifts.
+- Hard cap: 8 H20-hours including startup, warm-up, burn-in, invalid attempts,
+  and cleanup.
+
+## Gates
+
+Data validity requires complete 300-second Layer-1 coverage, zero dropped
+records, exact request/arrival/length hashes across action endpoints, monotonic
+timestamps, full request accounting, idle GPUs before and after each session,
+and no co-resident GPU process.
+
+Mechanism evidence requires at least two telemetry features whose matched action
+response exceeds same-config repeat noise at the same pair of consecutive
+checkpoints under the unchanged v1 response thresholds.  Those features must
+also have a consistent direction in at least two of the three load regimes.
+
+Decision evidence additionally requires both action-efficacy classes and at
+least one of the phase-stable mechanism features to reach
+leave-one-repetition-out balanced accuracy at least 0.75 and exceed the best
+external prefix outcome by at least 0.15 at two adjacent predeclared
+checkpoints from 25% onward.  Without label balance the pilot can adjudicate
+mechanism evidence only.
+
+No H20 run is launched if the local analyzer/tests, manifest preflight, GPU
+probe, command dry-run, projected cost, or cleanup plan fails.

docs/intervention-response-v3-results-20260714.md

@@ -0,0 +1,160 @@
+# Phase-aware telemetry intervention-response v3 results
+
+Date: 2026-07-14 (Asia/Singapore).
+
+Decision: **`STOP_NO_INCREMENTAL_TUNING_SIGNAL`**.
+
+## Claim tested
+
+After the replay has developed queue, batch, and KV state, does increasing MNS
+from 16 to 64 create telemetry responses that exceed workload-repeat noise, and
+does any such response identify whether the action repairs the full-run SLO
+better than external prefix outcomes alone?
+
+The first clause passed.  The second clause failed.  Long-window telemetry is
+mechanistically informative, but this pilot does not support its necessity for
+tuning this action.
+
+## Setup
+
+- dash0 GPU 0-3: four NVIDIA H20 GPUs; Qwen3-30B-A3B; patched vLLM
+  `0.24.1.dev3+opprof`; TP=4.
+- Action: MNS `16 -> 64` with exact request, arrival, and input-length hashes.
+- Trace: `chat_w20260312_1000`, replay-time scale 0.5, 300 seconds.
+- Loads: 1.5 and 2.125 requests/s/GPU; three disjoint bands; endpoint order
+  A/B, B/A, A/B; load order low/mid, mid/low, low/mid.
+- Five cumulative checkpoints: 30, 75, 150, 225, and 300 seconds; four
+  non-overlapping quarter blocks.
+- Six fresh-server sessions, 12 measured runs, six action pairs, and eight
+  same-config repeat pairs.
+
+The prior three-load attempt is not part of the result.  Its MNS=16 workload at
+3.125 requests/s/GPU could not drain by the 450-second client timeout and
+produced no high-load result.  V3 reran every retained point from scratch.
+
+## End-to-end outcome
+
+All three low-load pairs remained feasible (`true->true`, label 0).  All three
+pressure-load pairs changed from infeasible to feasible (`false->true`, label
+1).  MNS=16 pressure pass rates were 0.5604, 0.3145, and 0.2635; all three
+MNS=64 pressure runs reached 1.0.  This yielded a balanced 3/3 action label set.
+
+## Mechanism result
+
+No telemetry feature passed the action-versus-repeat gate at 10% or 25%.  At
+50%, graph padding first passed.  At both 75% and 100%, graph padding and queue
+waiting passed, satisfying the requirement for two features at the same pair
+of adjacent checkpoints and with consistent directions in both load regimes.
+
+| Feature | Direction for MNS 16->64 | 75% effect/repeat median | 75% above repeat p95 | 100% effect/repeat median | 100% above repeat p95 |
+|---|---:|---:|---:|---:|---:|
+| `queue_waiting_mean` | lower | 1346.22 | 3/6 | 898.70 | 3/6 |
+| `graph_padding_fraction` | higher | 5.00 | 4/6 | 5.74 | 5/6 |
+
+The very large queue effect/median-repeat ratios should not be read alone: its
+repeat p95 was much larger than its repeat median, so the independent p95
+coverage criterion remained binding.  Full-window queue-waiting deltas were
+-0.19 to -0.32 at low load and -20.99 to -32.16 at pressure load.  Graph
+padding increased in every pair, by 0.00133-0.00217 at low load and
+0.00705-0.00873 at pressure load.
+
+The mechanism is therefore a real tradeoff: larger MNS reduces queueing,
+especially under pressure, while increasing CUDA-graph padding.
+
+## Tuning-signal result
+
+Leave-one-repetition-out balanced accuracy was evaluated against the best
+external prefix-outcome feature at every predeclared checkpoint.
+
+| Replay phase | Best external BA | Best telemetry BA | Incremental telemetry gate |
+|---:|---:|---:|---|
+| 10% | 0.833 | 0.833 | fail |
+| 25% | 1.000 | 1.000 | fail |
+| 50% | 0.833 | 1.000 | pass at this checkpoint only |
+| 75% | 1.000 | 1.000 | fail |
+| 100% | 1.000 | 1.000 | fail |
+
+At 50%, several telemetry features exceeded the external baseline by at least
+0.15, including the phase-stable mechanism feature
+`graph_padding_fraction`.  The advantage did not hold at either adjacent
+checkpoint.  Consequently no feature passed the frozen two-adjacent-phase
+requirement.
+
+The important ordering is that external TTFT already classified the action
+perfectly at 25%, whereas the robust two-feature mechanism response did not
+emerge until 75%-100%.  In this setup telemetry explains *why* MNS helps, but it
+does not provide earlier or more reliable action selection than direct prefix
+outcomes.
+
+## Research conclusion
+
+The 5/10-second negative result was indeed too narrow.  It only ruled out an
+ultra-early telemetry verifier; it did not rule out engine-state information.
+The 300-second pilot finds a clear and reproducible queueing-versus-padding
+response.
+
+However, this does not rescue the direct telemetry-guided tuning claim.  For
+this action and workload, the external signal is already as good or better
+before the telemetry mechanism becomes stable.  The project should therefore
+not claim that engine instrumentation is necessary for tuning on this evidence,
+and should not open an E2E policy test from this pilot.
+
+The narrower simulator-residual route remains logically open: telemetry may
+explain why a simulator misranks real configurations even when direct online
+outcomes can guide a tuner.  That is a different hypothesis and was not tested
+here.
+
+## Change and verification
+
+Change: absolute 5/10-second prefixes were replaced by phase-aware 30/75/150/
+225/300-second analysis; SLO early stop was disabled; full Layer-1 coverage,
+hash, request-accounting, controller, and stream/footer gates were added.  The
+mechanism gate was corrected to require two features at the same adjacent
+phase pair.
+
+Expected effect: distinguish “telemetry has not developed yet” from “telemetry
+does not identify or improve the action decision.”
+
+Verification: five local analysis/controller test suites passed; remote
+manifest preflight and command dry-run passed; six serialized sessions passed
+all stream invariants; the analyzer was rerun and produced byte-identical
+output.
+
+Result: mechanism evidence passed; incremental tuning evidence failed.  Audit
+SHA256: `45f6f248712f9cbd3ed72036837ff6dc5b5c14c0f2eb6ba5cd5daceb1aa4ddb7`.
+
+Remaining risk: this is a development pilot with one model, one TP, one action,
+two retained loads, three request bands, and six action labels.  It is adequate
+to reject opening the next direct-policy stage, not to establish a universal
+negative claim about telemetry.
+
+## Research-validity audit
+
+| Check | Verdict | Evidence / boundary |
+|---|---|---|
+| Real system and E2E outcome | PASS | Real H20/vLLM replay; full SLO outcome accompanies mechanism telemetry. |
+| Matched action baseline | PASS | Exact request/arrival/length hashes for MNS 16 and 64; external prefix outcome is the decision baseline. |
+| Repeats and order effects | PASS for pilot | Three disjoint bands; A/B, B/A, A/B endpoint order; counter-rotated load order. |
+| Selective load removal | PASS with narrowed claim | The 3.125 load produced no result before any action comparison; the failure and cost are retained, and all kept points were freshly rerun. |
+| Significance/generalization | NEEDS EVIDENCE for a paper claim | Only three bands, one model, one TP, one action, and six labels.  This is explicitly a stage gate. |
+| Calibration versus evaluation | NEEDS EVIDENCE for a positive policy claim | Frozen gates and leave-one-band-out folds reduce leakage, but a new workload/model hold-out is still required. |
+| Platform/reproducibility | PASS | Commit, commands, manifest, controller state, platform fingerprint, raw remote paths, and audit hashes are recorded. |
+
+## Data sanity
+
+- Measured runs: n=12; elapsed 300.346-317.012 seconds; 12 distinct; pass rate
+  0.2635-1.0 with 4 distinct values; selected requests 1800-2550 with 2
+  distinct values.
+- Sessions: n=6; 0.8413-0.8631 H20-hours; 6 distinct; Layer-1 records
+  58,465-64,776; 6 distinct.  V3 cost was 5.0924 H20-hours; total including
+  the invalid attempt was 6.4505, below the 8.0 cap.
+- Labels: n=6; min/max 0/1; 2 distinct.  Action pairs were 6 and repeat pairs
+  were 8 at every checkpoint.
+- Coverage-gap observations: n=60; start gaps 0.0427-0.1247 seconds; end gaps
+  0.00014-0.1705; maximum internal gaps 0.1695-0.6528, all below one second.
+- Checked invariants: exact pair hashes and counts, all runs uncensored, full
+  request accounting, monotonic admitted/completed coverage, monotonic Layer-1
+  timestamps, nonnegative counters, bounded ratios, non-identical per-config
+  states, contiguous step indices, zero drops, footer/sidecar agreement, no
+  controller failures, all sessions complete, GPU idle after completion.  No
+  red flags were found.

docs/intervention-response-v3-two-load-protocol-20260714.md

@@ -0,0 +1,73 @@
+# Phase-aware telemetry intervention-response v3 protocol
+
+Status: **FROZEN AFTER A NON-COMPARATIVE OPERATIONAL FAILURE AND BEFORE V3 RUNS**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Why v2 was invalid
+
+The first v2 session completed the 300-second arrival windows at 1.5 and 2.125
+requests/s/GPU.  At 3.125 requests/s/GPU, MNS=16 could not drain the admitted
+requests before the 450-second client timeout.  The session produced no result
+and no MNS=64 action endpoint was run.  V2 is therefore an invalid operational
+attempt, not evidence for or against the telemetry hypothesis.
+
+This failure was observed before any MNS action comparison.  V3 excludes only
+the unmeasurable overload point and reruns every retained point on fresh
+servers; it does not reuse the completed v2 low/mid results.
+
+## Question and hypothesis
+
+Question: after enough replay time for queue, batch, and KV state to develop,
+does an MNS intervention create telemetry responses that exceed workload-repeat
+noise, and does any such response predict whether the intervention repairs the
+full-run SLO outcome better than external prefix outcomes alone?
+
+Hypothesis: increasing MNS from 16 to 64 has little value at the 1.5
+requests/s/GPU control load but can repair the 2.125 requests/s/GPU pressure
+load.  Queue, running-set, batch, or KV telemetry should expose the difference
+at stable replay phases.  Label balance is an assumption to test, not a fact.
+
+## Frozen setup
+
+- Solo placement on dash0 GPU 0-3: 4 NVIDIA H20 GPUs, Qwen3-30B-A3B, patched
+  vLLM `0.24.1.dev3+opprof`, fixed TP=4.
+- Action: MNS `16 -> 64`; all other engine and workload parameters fixed.
+- Workload: `chat_w20260312_1000`, replay-time scale 0.5, hence 300 seconds.
+- Loads per GPU: 1.5 control and 2.125 pressure requests/s.  The failed 3.125
+  overload point is excluded from V3 and retained only as a failure artifact.
+- Three disjoint request bands.  Each MNS action pair has exact request,
+  arrival, and input-length hashes.  Endpoint order is A/B, B/A, A/B; load
+  order is low/mid, mid/low, low/mid.
+- Every session starts a fresh server, then runs the accepted 16-request long
+  warm-up and bounded burn-in before measured runs.
+- SLO-unrecoverable early stop is disabled.  Measured results must cover the
+  full 300-second arrival window and must not be early-stopped.
+- Cumulative checkpoints are 10%, 25%, 50%, 75%, and 100%; non-overlapping
+  quarter blocks are also reported.
+- A Layer-1 interval is complete only if timestamps are monotonic and its
+  start, end, and maximum internal record gaps are each at most one second.
+- Incremental V3 cap is the unused portion of the original 8 H20-hour cap.
+  The exact prior cost and V3 cap are machine-recorded in the manifest.
+
+## Frozen gates
+
+Data validity requires six uncensored sessions, six action pairs, eight
+same-config repeat pairs, exact action-pair hashes, full request accounting,
+zero Layer-1 drops, continuous coverage, all stream/footer invariants, no
+co-resident compute process, idle GPUs before and after sessions, nonnegative
+counters, bounded ratios, non-identical per-config state, and monotonic request
+coverage across checkpoints.  Any red flag stops analysis.
+
+Mechanism evidence requires at least two telemetry features to exceed the
+unchanged v1 repeat-noise thresholds at the same pair of adjacent checkpoints.
+Those features must have a direction consistent in both retained load regimes.
+
+Decision evidence additionally requires at least two positive and two negative
+full-run action-efficacy labels, valid leave-one-repetition-out folds, and at
+least one phase-stable mechanism feature whose balanced accuracy is at least
+0.75 and at least 0.15 above the best external prefix-outcome feature at two
+adjacent predeclared checkpoints from 25% onward.
+
+V3 remains a development mechanism pilot.  Even `OPEN_E2E_POLICY_TEST` opens a
+held-out tuning-policy experiment; it is not itself a paper performance claim.

docs/opprof-simfid-overview-20260713.md

@@ -0,0 +1,131 @@
+# SimFid + OpProf Campaign Overview
+
+Date: 2026-07-13. Status: both campaigns **CLOSED**. This is the entry-point
+summary; every claim below links to a frozen protocol and a results document.
+Decision history lives in `docs/opprof_campaign_state.md` (this repo) and
+`replayserve/docs/simfid_campaign_state.md`.
+
+## Why these campaigns exist
+
+The paper's motivation requires three evidence-backed claims:
+
+1. **Simulators cannot replace real replay** for config tuning → SimFid.
+2. **The leverage point is engine-knob configuration, not operator
+   implementation**, at the measured regimes → OpProf P3–P5.
+3. **Tuning is genuinely hard**: the surface is workload-conditioned
+   (sign-flips defeat defaults), real evaluation is expensive (measured GPU
+   cost), and the surface churns across engine versions → OpProf P4/P6.
+
+All three now rest on pre-registered protocols with frozen decision rules,
+Holm-corrected contrasts, and no imputation over censored data.
+
+## Campaign 1 — SimFid (replayserve repo)
+
+**Question.** Can a Frontier-class simulator replace real replay for AITuner
+config tuning? **Verdict: NOT ADEQUATE** under the pre-declared decision rule.
+
+- S2-E (3-config TP family, same model/HW): ranking perfect after throughput
+  calibration, but latency is uncalibrated (TTFT p95 sim/real 0.30–0.38, TPOT
+  0.63–0.79) → false-feasible counterexample across a 50 ms TPOT SLO.
+- S2-R-b (12-cell TP×MNS surface, exact C1 workload replayed in Frontier,
+  184 runs, 0 failures): the decision-bearing frozen-calibrated
+  throughput-proxy reading picks TP1/MNS64 → **30.46% real top-1 regret**;
+  trap detection 3/6; LOAO 0/92. All three adequacy components FAIL.
+- Post-hoc diagnostic (NOT decision-bearing): an SLO-gated reading achieves
+  0–0.76% regret, tau-b 0.967, trap 6/6 — anchor-level SLO errors partially
+  cancel at per-cell peaks. This motivates hybrid sim-prune + real-final
+  designs but must always carry the post-hoc label.
+
+Sources: `replayserve/docs/simfid_s2e_report.md`,
+`replayserve/docs/simfid_s2rb_results.md`.
+
+## Campaign 2 — OpProf (this repo, `docs/opprof/`)
+
+**Question.** Do operators face materially different patterns online than in
+offline rectangular benchmarks ("workload-conditioned operator profiling"),
+and if so, where is the recoverable gain? Patched vLLM 0.24.0
+(`patches/vllm-0.24.0-opprof/`), Qwen3-30B-A3B BF16, dash0 H20.
+
+| Phase | Deliverable | Verdict / headline |
+|---|---|---|
+| P0–P2 | Dual-layer instrumentation (always-on Layer-1 telemetry + sampled Layer-2 Kineto) | Overhead **−0.04%** (CI [−0.17, +0.05]) after the compile-factor fix; Layer-2 perturbs 51.3% → sampled-only by design |
+| P3 | 10-pattern × config matrix (`phase3-{protocol,results}.md`) | **H1b PASS** (5/6 evaluable contrasts, Holm p≈0): irregular patterns carry R64 raggedness +23.0 to +44.8 pp over rectangular controls with 8.3–44.7% useful-token efficiency loss. **H1a INCONCLUSIVE** (Layer-2 window representativeness) |
+| P4 | Ranked optimization plan (`phase4-optimization-plan.md`) | #1 prefix-affine routing: **+82.14%** saturation req/s (P08 vs matched P07, 62.3% fewer prefill tokens). #4 MNS is pattern-conditioned: MNS64 gives +3.4/+3.7% on P06/P10 but **−24.27% on P01** — the sign flips by workload |
+| P5 | Mechanism decomposition (`phase5-{protocol,results}.md`) | All four intuitive mechanisms ≈0 at rho=0.60: raggedness 3.8% n.s., capture-size fix +1.6% n.s. (padding 13.74%→2.40% with no E2E gain), prefix ~0. **Real finding:** P3's P10-vs-P04 gap was largely an arrival-uniformization artifact — replaying recorded (bursty) arrival recovers +12.9%; burstiness helps batch formation at low rate |
+| P6 | Cross-version churn, paired 12-cell surface, vLLM 0.20→0.24 (`phase6-{protocol,results}.md`) | Old #2 config TP2/MNS64 **−29.41%** (solo-confirmed, bounded both sides); TP1 plateau **+13.5%** at MNS8; old argmax TP2/MNS32 held (−0.76%). Formal ARGMAX/RANKING/TRAP **INCONCLUSIVE** — 4 cells right-censored/non-monotonic, no imputation |
+
+**P6 mechanism note.** TP2/MNS64's old anchor now fails with decode-batch
+means 11.5–18.1 and 4.6–9.5% of steps executing outside cudagraph coverage;
+all 37 solo primaries had zero preemptions.
+
+**Combined churn claim (paper-usable).** Across one ~2-month engine upgrade
+the surface below the argmax reorganized (rank-2 config −29%, plateau +13%)
+even though the argmax survived → warm-start/transfer from stale tuning
+surfaces is unreliable; every engine upgrade is a retuning trigger.
+
+## Cross-cutting methodological findings (reusable beyond this paper)
+
+1. **Co-location validity is metric-dependent.** 21 exact same-request pairs
+   (co-located vs solo, P6): throughput and operator shares move <3%, but SLO
+   pass rates flip by up to **+92.86 pp** (0.071→1.000) at frontier anchors —
+   feasible/infeasible verdicts invert. Deltas are cell- and anchor-dependent
+   (0 to +92.9 pp), so no fixed correction exists. Rule adopted: SLO-frontier
+   measurements must be solo; mean-type metrics may be co-located behind the
+   pre-registered A-P3-1 validity gate (which rejected 8-way, passed 4-way).
+2. **Uniformizing arrival distorts efficiency** (−12.9% at low rate) in the
+   opposite direction from intuition — offline benchmarks that regularize
+   arrival misestimate real efficiency.
+3. **Env vars hashed into vLLM compile factors** silently cause cold
+   torch.compile caches and ~4% slower artifacts (root cause of our phantom
+   overhead; fixed with a one-line ignore-list entry). Upstream-report
+   candidate.
+4. **Long-context real traces break short-load-calibrated harness
+   assumptions** (drain deadlines, warm-up stabilization); P10/TP2 never
+   stabilizes (36.8% drift).
+
+## Corrections and honest limits — do NOT quote these stale readings
+
+- P3's "P10 is 14.3% worse than P04" is **superseded by P5**: largely a
+  materialization artifact of uniformized arrival. Quote pattern-vs-control
+  raggedness/padding effects (H1b) and the P5-corrected arrival result
+  instead. The remaining P10-vs-P03 gap (~36%) is workload physics, not
+  recoverable waste at this regime.
+- H1a (operator bottleneck-ranking inversions) is **not refuted** — it is
+  inconclusive at the measured regimes; saturation-regime decomposition
+  remains open.
+- P6 formal verdicts are INCONCLUSIVE by right-censoring, not by data
+  invalidity; the −29.41%/+13.5%/−0.76% numbers are bounded and quotable.
+- Only solo-tier SLO numbers are quotable; co-located W1–W3 artifacts are
+  preserved but superseded.
+- The <3% co-location bound is an empirical gate verified at moderate load on
+  specific patterns, not a theorem; re-run the gate before reusing 4-way
+  placement in new regimes.
+- SimFid's SLO-gated 0–0.76% regret reading is post-hoc, not decision-bearing.
+
+## Artifact map
+
+| Path | Content |
+|---|---|
+| `docs/opprof_campaign_state.md` | Full OpProf decision ledger (echoes, amendments, acceptances) |
+| `docs/opprof/phase{0,2}-*.md`, `docs/opprof/patch-design.md` | Recon, patch design, smoke + overhead evidence |
+| `docs/opprof/phase{3,5,6}-protocol.md` | Frozen pre-registered protocols incl. amendments |
+| `docs/opprof/phase{3,5,6}-results.md`, `phase4-optimization-plan.md` | Results; phase6 metrics pinned by SHA-256 `290ba7fc…` |
+| `patches/vllm-0.24.0-opprof/` | 7-patch series + apply.sh + tests (base `ee0da84a`) |
+| `runs/opprof-phase{3,5,6}/` | Decision-bearing evidence (metrics/manifests/validation); raw Layer-1 JSONL streams are git-ignored (507 MB, kept on disk and dash0) |
+| `docs/simulator-fidelity-frontier-20260711.md` | Standalone review: does the data show simulator mis-ranking rigorously? |
+| `replayserve/docs/simfid_*` | SimFid protocols, results, ledger |
+
+## GPU accounting
+
+OpProf total ≈ **22.2 H20-hours** (P0–P5 ≈ 16.5, P6 5.64 of a 6.0 cap).
+SimFid accounting lives in the replayserve ledger. No prompt or generated
+text appears in any committed artifact; the prompt-bearing trace copy stays
+in git-ignored `trace_windows/`.
+
+## Open items (not committed to)
+
+- Bound the 3 right-censored TP4 cells + TP2/MNS16 (~1.6 H20-h, exceeds the
+  P6 cap) to convert ARGMAX/TRAP into formal verdicts.
+- Saturation-regime mechanism decomposition (P5 analogue at high rho).
+- Upstream reports: compile-factor env poisoning; co-location SLO validity.
+- Synthesis of both campaigns into the paper's motivation section.

docs/opprof/oracle-gap-protocol.md

@@ -0,0 +1,240 @@
+# Static-policy oracle-gap protocol
+
+Status: **FROZEN WITH A-OG-1 THROUGH A-OG-4 AMENDMENTS**.
+
+Date frozen: 2026-07-13 (Asia/Singapore). Existing Phase-3 measurements were
+inspected only to choose the workload pair and rate brackets. They are
+exploratory calibration data, not primary observations in this protocol.
+
+### A-OG-4 — close a majority-shifted boundary (before confirmation scores)
+
+After all four primary frontiers and 70 scores were complete, the controller
+was interrupted before the first confirmation produced a score. The partial
+`P01-C10-r26-rep1` attempt contained no result, sanity file, or score and is
+archived separately. This amendment is therefore blind to confirmation
+outcomes.
+
+The original confirmation schedule still runs first. Afterward, each cell's
+majority-vote frontier is recomputed. If either side of the *actual* final
+boundary has fewer than three trials because the provisional boundary moved,
+the controller runs only enough repetitions at that existing rate to reach
+three, high-to-low, on a fresh server for that config. It then recomputes the
+boundary and repeats for at most three closure rounds. No new rate anchor may
+be added. Failure to obtain a monotone, bracketed boundary with three trials
+per side within three rounds or the 6 H20-hour cap makes the experiment
+inconclusive.
+
+This fills a stopping-rule omission: it does not change any existing score,
+majority rule, phase, config, rate grid, SLO, or oracle calculation.
+
+### A-OG-3 — freeze a per-logical-trial transport retry rule (after 62 scores)
+
+The first attempt at `P06-C10-r1.9-rep0` reproduced the same isolated local
+transport signature seen in A-OG-2: one clean request raised
+`ClientOSError` 3.56 ms after admission, with HTTP status 0, no first token,
+and no output. The other 343 requests produced exactly 512 tokens, the server
+stayed healthy, drain took 4.00 s, and every other client invariant passed.
+The controller again rejected the attempt before creating a `score.json`.
+
+For the remainder of this experiment, an attempt may be quarantined and the
+identical logical trial retried once on a fresh server only if all of the
+following hold: the sole failed client invariant is `clean_failures_zero`;
+exactly one clean request failed; its error is `ClientOSError`, HTTP status is
+0, it produced neither a first token nor output, and it completed within 10
+ms of admission; every successful request has exact output; and the server
+has no crash or error. The attempt is never scored. A second invalid attempt
+for the same logical key, more than one failed request, or any other error
+signature is a stop condition and makes the experiment inconclusive.
+
+This rule is independent of config, rate, and observed performance, and
+supersedes the one-key wording in A-OG-2 without changing how that retry was
+executed. Existing scores remain immutable. No client, grid, policy,
+threshold, order, SLO, or oracle calculation changes.
+
+### A-OG-2 — retry one transport-invalid attempt (after 49 scored trials)
+
+The first attempt at `P06-C01-r2.2-rep0` produced one local
+`ClientOSError` 3.27 ms after admission, with HTTP status 0, no first token,
+and no output. The other 399 requests completed successfully, every successful
+request produced exactly 512 tokens, the server stayed healthy, and all other
+client invariants passed. The controller rejected the attempt before creating
+a `score.json`; no TTFT/TPOT result from this attempt was used to choose the
+recovery rule.
+
+This is a measurement-transport failure, not an observed server SLO outcome.
+The entire attempt is content-hashed and moved under `invalid-attempts/`, then
+the identical logical trial (phase, config, rate, repetition, derived manifest,
+seeds, timeline, and SLO) is run once on a fresh C01 server. All 49 existing
+scores remain immutable. A second client transport failure in the retry is a
+stop condition and makes the experiment inconclusive; it must not be retried
+again. No grid, policy, threshold, order, or oracle calculation changes.
+
+### A-OG-1 — extend the P06 upper bracket (after trial 33)
+
+The controller stopped as registered after C00/P06 remained SLO-feasible at
+every original and upward-extension anchor through 2.3 requests/s. At that
+point 33 trials and 1.519475 H20-hours were complete, all GPU memory had been
+released, and no C00/P06 infeasible upper bound existed. No oracle inference
+was performed.
+
+This amendment changes only the P06 upward-extension list from
+`2.1,2.2,2.3` to `2.1,2.2,2.3,2.4,2.5,2.6,2.8,3.0`. The controller stops at
+the first bracket exactly as before. Existing trials are immutable and are
+reused; config order, P01 rates, SLO, timelines, repetitions, placement,
+metrics, and decision threshold do not change. The resumable controller may
+accept the new code/protocol fingerprint only when all immutable runtime,
+client, model, manifest, config, and base-grid fields match the pre-amendment
+state, and it records both fingerprints under `A-OG-1`.
+
+## Question and decision gate
+
+The candidate motivation is:
+
+> A single global static batching policy leaves at least 10% end-to-end
+> SLO-goodput on the table when serving temporally heterogeneous phases; a
+> phase-aware runtime policy can recover that gap without changing hardware,
+> model, precision, or tensor-parallel topology.
+
+This experiment tests a necessary condition in the existing TP1 policy space
+`{C00,C10,C01,C11}`. The optimistic oracle knows the phase and switches with
+zero delay, zero state-transfer cost, and no prediction error. If even this
+oracle cannot beat the best one-config-for-all-phases policy by 10%, an online
+controller over these MNS/MBT choices cannot do so either.
+
+The primary gate uses a conservative capacity bracket:
+
+- `L[p,c]`: highest offered rate accepted as SLO-feasible for phase `p` and
+  config `c`;
+- `U[p,c]`: lowest higher offered rate accepted as SLO-infeasible;
+- oracle upper bound at phase-time weights `w`:
+  `sum_p w[p] * max_c U[p,c]`;
+- best-static lower bound:
+  `max_c sum_p w[p] * L[p,c]`.
+
+We scan every P01/P06 time mixture, including pure endpoints. The current
+motivation is **REFUTED** if the maximum conservative ratio
+`oracle_upper / static_lower - 1` is below 10%. It is **NOT ESTABLISHED** if the
+bound crosses 10% but the observed point estimate does not. A positive result
+requires a point-estimate gap of at least 10% and then a separately
+pre-registered interleaved-trace validation; this frontier experiment alone
+cannot establish a positive E2E contribution.
+
+The conclusion is scoped to the measured MNS/MBT policy family and the chosen
+strongest-conflict phase pair. It does not rule out new scheduling mechanisms,
+KV-state policies, topology changes, or other workload phases.
+
+## Fixed system boundary
+
+| Item | Frozen value |
+|---|---|
+| Host | `dash0`, one run at a time on physical GPU0 |
+| GPU | NVIDIA H20; no other GPU process anywhere on the host |
+| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, BF16 |
+| Runtime | `/tmp/wjh-opprof-phase2-dash0-20260711/.venv`, vLLM `0.24.1.dev3+g668cfb7e2` |
+| vLLM source | `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0` |
+| Topology | TP1, one server, no data/pipeline parallelism |
+| Fixed mechanisms | chunked prefill on; prefix caching on |
+| Client | Phase-5 timestamp/fixed-rate wrapper over the Phase-3 exact-token client |
+| Seeds | workload `20260712`; trial token-domain seed derived only from phase/rate/repetition, never config |
+
+SLO co-location results in Phase 6 showed pass-rate flips despite small
+throughput deltas. Therefore unused H20s remain idle: parallel placement is not
+authoritative for this experiment.
+
+## Workloads and policies
+
+The pair is chosen before new measurements because Phase 3 showed the strongest
+opposing static preference:
+
+- **P01:** input `U[128,512]`, output exactly 64 tokens, deterministic steady
+  arrivals. C10 lost 24.27% saturation throughput relative to C00.
+- **P06:** 50/50 input mixture `U[128,512]`/`U[4096,8192]`, output exactly 512
+  tokens, deterministic bursts of eight. C10 gained 3.37% over C00.
+
+Both reuse the immutable 32,768-row Phase-3 manifests. For every trial a
+derived manifest preserves request order, lengths, outputs, and arrival class,
+but applies a trial-specific token-seed offset. The same derived manifest is
+used for all four configs. This prevents prefix-cache carry-over when a hot
+server executes several anchors without changing the logical workload.
+
+| Config | Effective MNS | Effective MBT | Extra flags |
+|---|---:|---:|---|
+| C00 | 1024 | 8192 | none |
+| C10 | 64 | 8192 | `--max-num-seqs 64` |
+| C01 | 1024 | 2048 | `--max-num-batched-tokens 2048` |
+| C11 | 64 | 2048 | both flags |
+
+Startup logs must confirm these values. A default drift is a stop condition.
+
+## Load grid, order, and repetitions
+
+Primary grids:
+
+- P01: `{26,28,30,32,34,36}` requests/s; execution order
+  `32,26,36,28,34,30`.
+- P06: `{1.4,1.5,1.6,1.7,1.8,1.9,2.0}` requests/s; execution order
+  `1.7,1.4,2.0,1.5,1.9,1.6,1.8`.
+
+Every primary anchor runs once. For each phase/config, the highest primary
+feasible anchor and its next higher primary anchor are then run two more times,
+giving three trials at both sides of the boundary. If all primary anchors are
+feasible, extend upward in the fixed order P01 `38,40,42` or P06
+`2.1,2.2,2.3,2.4,2.5,2.6,2.8,3.0`.
+If all are infeasible, extend downward in the fixed order P01 `24,22,20` or P06
+`1.3,1.2,1.1`. Stop extending at the first bracket.
+
+One primary server is launched per config in order `C11,C00,C01,C10`.
+Confirmation servers are fresh and launch in reverse order
+`C10,C01,C00,C11`; their boundary anchors run high-to-low. This balances
+machine-time drift and makes confirmation independent of the primary server's
+cache/compiler state.
+
+Timelines:
+
+- P01: 60 s warm-up + 60 s clean measurement; drain cap 120 s.
+- P06: 60 s warm-up + 120 s clean measurement; drain cap 240 s.
+- no Kineto profiling; exact greedy output with `ignore_eos`; maximum client
+  concurrency 256.
+
+A trial is SLO-feasible when at least 95% of requests admitted during the clean
+interval eventually finish successfully and individually satisfy both:
+
+- TTFT <= 2 s for input <= 4,096 tokens; <= 4 s for input <= 32,768; <= 6 s
+  otherwise;
+- TPOT <= 50 ms, computed as `(completion - first_token)/(output_tokens - 1)`.
+
+SLO-goodput is the number of those passing clean-admission requests divided by
+clean seconds. Client schedule lag must stay <=1 s and achieved clean offered
+rate must be within 5% of target. Failure of either condition makes the anchor
+infeasible; its admitted-only latency is not used to rescue it.
+
+At a repeated boundary, feasibility is the majority of three trial verdicts.
+All accepted anchor verdicts must be monotone in offered rate. A persistent
+non-monotone result after the registered repeats is a red flag and stops the
+oracle-gap inference.
+
+## Validity and stopping rules
+
+Before every server launch record host, GPU, driver, clocks, runtime package
+versions, git/source hashes, manifest hashes, exact commands, and process
+contamination. Stop on another GPU process, request/output mismatch, manifest
+drift, server crash, non-finite latency, ratio outside `[0,1]`, negative
+counter, or discontinuous/non-monotone accepted frontier.
+
+The controller is detached and resumable. It kills only process groups it
+created, checks zero GPU memory after every server, never overwrites a complete
+trial, and writes state atomically. The hard budget is 6 H20-hours; expected
+cost is 3.0--4.0 H20-hours and approximately the same wall time because runs
+are serialized.
+
+## Required report
+
+The report includes every trial's target/achieved rate, clean cohort size,
+pass count/rate, SLO-goodput, TTFT/TPOT percentiles, schedule lag, failure
+reasons, accepted frontier brackets, per-phase oracle choices, best static
+choice, equal-time gap, worst-mixture conservative gap, and GPU-hours.
+
+The final statistics section ends with a data-sanity block containing `n`,
+min/max, distinct-value counts, and checks for non-negative counters, ratios in
+`[0,1]`, non-identical per-config results, exact output work, monotone
+frontiers, and continuous rate brackets.

docs/opprof/oracle-gap-results.md

@@ -0,0 +1,199 @@
+# Static-policy oracle-gap results
+
+Status: **FINAL — REFUTED WITHIN THE FROZEN TP1 MNS/MBT POLICY SPACE**.
+
+Date: 2026-07-13. The registered experiment completed 104 valid trials on
+`dash0` and returned `REFUTED`. Even a phase-perfect oracle with zero detection,
+switching, and state-transfer cost is bounded below the registered 10%
+SLO-goodput contribution gate.
+
+The machine result is
+`runs/opprof-oracle-gap/metrics.json` (SHA-256
+`250ba4c1657a8830795ee06392eea4e21c62d958fea11701ba60581ef0266543`).
+All 104 trial-level measurements are in
+`runs/opprof-oracle-gap/trials.csv` (SHA-256
+`60cd901f18bbf107eb130f0095e867c3b6d47a78a42e83bcbe57fecf23cc5f9c`).
+The final resumable controller state is
+`runs/opprof-oracle-gap/controller-state.json` (SHA-256
+`27ec5e9a3cd32a871d583ba8eb6d7d3fe2a338a8fd42369233ada57cd4da6436`).
+
+## Decision
+
+The tested motivation was:
+
+> One static batching policy leaves at least 10% end-to-end SLO-goodput on the
+> table across temporally heterogeneous phases, and a phase-aware runtime can
+> recover it by switching MNS/MBT policies.
+
+The registered 10,001-point mixture scan finds a worst conservative gap of
+**8.333219%**, at P01 time weight `0.0244`. Equal phase time gives
+**7.260726%**. An independent implementation that evaluates every exact
+static-policy crossover gives a slightly more conservative exact maximum of
+**8.333333% = 1/12**, at P01 weight `1/41`. This leaves 1.666667 percentage
+points below the 10% gate.
+
+The negative result is stronger than a failed online prototype. The oracle
+already knows the current phase and pays no switching cost. A realizable
+controller over the same four policies cannot exceed this oracle bound.
+
+This is a scoped refutation, not a universal claim about adaptive serving. It
+rules out the current contribution based on phase-aware selection among the
+four MNS/MBT configurations for the frozen P01/P06 pair, Qwen3-30B-A3B, TP1,
+H20, vLLM 0.24, and the registered SLO. It does **not** rule out a new scheduler,
+KV/cache policy, routing-aware mechanism, topology change, or a different
+workload family.
+
+## Fixed setup
+
+| Item | Value |
+|---|---|
+| Placement | `dash0`, serialized on physical GPU0; GPUs 1-7 idle |
+| GPU | NVIDIA H20, driver 580.95.05 |
+| Model | Qwen3-30B-A3B, BF16 |
+| Runtime | vLLM `0.24.1.dev3+g668cfb7e2`, source `4b253fd8619764b6971a7f2e3a3aa7545f6ace05` |
+| Topology | TP1; one server and one client |
+| Fixed mechanisms | chunked prefill on; prefix caching on |
+| P01 | input `U[128,512]`, exactly 64 output tokens, deterministic steady arrivals |
+| P06 | 50/50 input `U[128,512]` / `U[4096,8192]`, exactly 512 output tokens, deterministic bursts of eight |
+| P01 timeline | 60 s warm-up + 60 s clean measurement |
+| P06 timeline | 60 s warm-up + 120 s clean measurement |
+| SLO feasibility | at least 95% of clean-admitted requests pass both TTFT and TPOT |
+| TTFT SLO | <=2 s for input <=4096; <=4 s for input <=32768; otherwise <=6 s |
+| TPOT SLO | <=50 ms/token |
+
+The four policies were:
+
+| Config | MNS | MBT |
+|---|---:|---:|
+| C00 | 1024 | 8192 |
+| C10 | 64 | 8192 |
+| C01 | 1024 | 2048 |
+| C11 | 64 | 2048 |
+
+## Final SLO frontiers
+
+`L` is the highest majority-feasible offered rate. `U` is the next higher
+majority-infeasible rate. Both sides below have at least three trials, and all
+accepted rate sequences are monotone.
+
+| Config | P01 `L` | P01 `U` | P06 `L` | P06 `U` |
+|---|---:|---:|---:|---:|
+| C00 | 28.0 | 30.0 | 2.3 | 2.4 |
+| C10 | 24.0 | 26.0 | 2.4 | 2.5 |
+| C01 | 28.0 | 30.0 | 2.2 | 2.3 |
+| C11 | 24.0 | 26.0 | 2.2 | 2.3 |
+
+The optimistic oracle upper bound chooses C00/C01 at `U=30` for P01 and C10
+at `U=2.5` for P06. The static lower bound chooses C10 below P01 weight `1/41`
+and C00 above it; they tie at the exact worst point.
+
+| Mixture | Oracle upper | Best-static lower | Conservative gap |
+|---|---:|---:|---:|
+| Pure P06 | 2.500000 | 2.400000 | 4.166667% |
+| Exact worst, P01 weight `1/41` | 3.170732 | 2.926829 | **8.333333%** |
+| Equal phase time | 16.250000 | 15.150000 | **7.260726%** |
+| Pure P01 | 30.000000 | 28.000000 | 7.142857% |
+
+The specialization exists but is too small. Reducing MNS from 1024 to 64
+raises the conservative P06 lower bound from 2.3 to 2.4 req/s, while lowering
+P01 from 28 to 24 req/s. Even after using infeasible `U` values for the oracle
+and feasible `L` values for the static baseline, the best possible phase-aware
+selection cannot reach 10%.
+
+## Robustness finding
+
+The strongest system finding is not config specialization but a repeat-level
+mode flip at P01/26 rps. Both C10 and C11 show the identical verdict sequence
+`rep0=0% pass`, `rep1=100% pass`, `rep2=0% pass`. For C10, TTFT p50 is
+3864.62, 252.71, and 3994.88 ms; for C11 it is 6533.62, 593.44, and
+5812.71 ms. TPOT remains below the 50 ms SLO in these trials.
+
+These trials are measurement-valid: exact outputs, offered-rate tolerance,
+schedule lag, timestamps, and client invariants all pass. The majority rule
+therefore classifies 26 rps as infeasible for both configs. The result does not
+change the oracle-gap decision because neither config supplies the P01 oracle
+maximum or the relevant best-static P01 lower bound.
+
+Token-domain seed and server execution history change together across
+repetitions, so this experiment cannot attribute the flip to MoE routing,
+cache/compiler state, or another source. It does motivate a narrower factorial
+study that crosses token seed with fresh/reused server state and randomizes
+order, while recording routed-expert and per-step scheduler telemetry. That is
+a mechanism question; it should not be presented as evidence for a phase-aware
+MNS/MBT controller.
+
+## Execution and audit history
+
+The 104 scored trials comprise 52 base-grid primaries, 18 registered upward or
+downward extensions, 32 boundary confirmations, and two boundary-closure
+trials. The final closure moved C00/P06 from the provisional `[2.4,2.5)` to
+`[2.3,2.4)` and repeated 2.3 rps to 3/3 feasible.
+
+Four frozen amendments are recorded in the protocol:
+
+- A-OG-1 extended only the P06 upward anchors after C00 remained feasible
+  through the original 2.3-rps limit.
+- A-OG-2 quarantined one P06/C01/2.2 local `ClientOSError` attempt and retried
+  the identical logical trial on a fresh server.
+- A-OG-3 generalized the same pre-score transport rule after one
+  P06/C10/1.9 attempt showed the identical signature.
+- A-OG-4, frozen before any confirmation score, added closure for a
+  majority-shifted boundary without adding new anchors.
+
+The two transport-invalid attempts and one pre-A-OG-4 interrupted attempt
+created no score and do not enter any metric. Their retained tree hashes are,
+respectively,
+`434863ba90513cbc54534ffbc1a13c980b3ef7d567190a0aa3f97b55650acbb2`,
+`a57b1ac5f090680bb70c16b5d709eb2b8ac47dce57c7a03b8077cd9b6d80d831`,
+and `bc3feb53514601e641ef1db204c74cffe3d282b24ad797e5b161468f5d15de5c`.
+
+Ten deliberately overloaded primary anchors exceed both the 1 s schedule-lag
+gate and the 5% offered-rate tolerance. They are correctly classified as
+infeasible and are not used as final boundary points. All 48 final boundary
+trials pass both client-side gates; their maximum schedule lag is 573.59 ms.
+
+The final experiment fingerprint uses repository commit `16177b0`, analyzer
+SHA-256
+`d86ecb1f077472906cbb729bd2c9d4b3a82ac6dfdc90838a17ae300d0767110d`,
+controller SHA-256
+`f7c5c2f74f2002f1e4b097e608d165a3a2e9374fbf07935a4d6d6a7c5d45a83a`,
+and protocol SHA-256
+`173f969a4428643cf6c4b950413aa82bef25cafd163e20d7944370a5d87af435`.
+The archived launch log SHA-256 is
+`54e2f4804a5efee072b670043c4092cf41f1a27664faa05ea71bfc1412c3e9db`.
+
+## GPU accounting and cleanup
+
+The campaign used **4.8877033845 H20-hours**, below the 6.0-hour cap, across
+5 h 41 min 51 s wall time including amendment, audit, restart, and server
+startup intervals. At completion, all eight H20s report 0 MiB, 0% utilization,
+and zero compute processes.
+
+## Sanity block
+
+There are no data-sanity red flags. The P01/26 all-or-none mode flip is a
+scientific robustness finding, not an invalid trial signature; it repeats in
+two configs and all measurement invariants pass.
+
+| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---|
+| Score-row indicator | 104 | 1 | 1 | 1 expected | 104 score files; no overwrite |
+| Target rate (req/s) | 104 | 1.4 | 36.0 | 19 | Non-negative; fixed grid/extensions only |
+| Clean cohort per trial | 104 | 168 | 2160 | 30 | Non-empty and non-negative |
+| Pass rate | 104 | 0.0 | 1.0 | 57 | All ratios in `[0,1]` |
+| SLO-goodput (req/s) | 104 | 0.0 | 27.816667 | 59 | Non-negative; per-cell results not all identical |
+| Boundary pass rate | 48 | 0.0 | 1.0 | 33 | Both final sides have three trials |
+| Boundary max schedule lag (ms) | 48 | 2.145861 | 573.593768 | 48 | All below the 1000 ms gate |
+| Exact-output indicator | 85,402 | 1 | 1 | 1 expected | Every clean request produced the requested token count |
+| Frontier cells | 8 | 8 | 8 | 1 expected | 8/8 bracketed and monotone |
+| Registered weight scan | 10,001 | 0.0 | 1.0 | 10,001 | Continuous 0.0001 increments, endpoints included |
+| Campaign H20-hours | 1 | 4.887703 | 4.887703 | 1 | Non-negative and below 6.0 |
+| Final GPU memory/utilization | 8 | 0 MiB / 0% | 0 MiB / 0% | 1 expected | Zero compute processes |
+
+Checked invariants: fixed model, runtime, manifests, SLOs, config values, seeds,
+and serialized placement; exact output work; nondecreasing timestamps;
+non-negative counters and latencies; pass ratios in range; offered-rate and
+schedule gates; majority-of-three final boundaries; monotone accepted
+frontiers; complete 10,001-point scan; independent exact-crossover
+recomputation; transport-attempt quarantine; GPU hard-cap compliance; and
+complete GPU cleanup.

docs/opprof/patch-design.md

@@ -0,0 +1,434 @@
+# OpProf dual-layer instrumentation patch design
+
+Status: **design for review; no vLLM patch has been implemented**.
+
+Target source: vLLM `v0.24.0`, commit `ee0da84ab9e04ac7610e28580af62c365e898389`.
+Target campaign: Qwen3-30B-A3B serving on NVIDIA H20 (SM90), using the V1 engine.
+All source paths and line numbers below are relative to the pinned clone.
+
+## Approved dispositions (2026-07-11)
+
+- Use JSONL with `msgspec`, the proposed context/chunk histogram edges, and an
+  8192-record bounded queue.
+- Keep exact expert loads Layer-2-only.
+- Use the community BF16 Qwen3-30B-A3B checkpoint. TP1 is primary, with TP2 and
+  TP4 counterpoints; record the selected MoE backend logs in every run.
+- Use two profiler warm-up iterations followed by eight active iterations.
+- Apply the 3% Layer-1 overhead gate to the **upper bound of the 95% confidence
+  interval** for every primary serving metric.
+- Reject `--disable-log-stats` when OpProf is enabled; retain the five-file
+  fail-fast design.
+
+## Goal and success criteria
+
+The patch should make every serving iteration conditionable on its request and
+execution pattern while keeping the always-on path below a 3% serving overhead
+budget. Heavy kernel tracing and exact MoE routes are sampled separately.
+
+The proposed split is:
+
+1. **Layer 1:** one compact composition record per scheduler step, emitted by
+   the scheduler process rather than every tensor-parallel worker.
+2. **Layer 2:** short, sampled windows using vLLM's existing torch-profiler
+   configuration and `/start_profile` and `/stop_profile` endpoints.
+
+The design deliberately does not add hooks to GPU kernels, attention layers,
+the Qwen model, or the OpenAI serving API. Those surfaces are not needed to
+answer the Phase 0 profiling questions.
+
+## Assumptions and tradeoffs
+
+- The campaign uses the V1 engine and the default vLLM scheduler. Both the
+  synchronous and `AsyncScheduler` paths inherit the base scheduler hooks.
+- The H20 backend statement assumes the usual unquantized BF16/FP16
+  Qwen3-30B-A3B checkpoint, `moe_backend=auto`, and no LoRA. Quantization,
+  batched expert format, or an explicit backend can change kernel selection.
+- Layer 1 requires normal stats collection, which is enabled by default. If
+  OpProf is enabled together with `--disable-log-stats`, initialization should
+  fail with an actionable error. Supporting that unusual combination would
+  require an extra stats path and a larger patch.
+- The context-length histogram records the sequence length at the end of the
+  scheduled model input (`num_computed_tokens` before scheduling plus tokens
+  scheduled in this step). This matches the worker-side sequence-length
+  construction at `vllm/v1/worker/gpu_model_runner.py:2010-2019` and avoids
+  retaining raw request lengths.
+- “Decode tokens” means tokens scheduled for requests classified by vLLM as
+  generation-phase requests, including scheduled speculative tokens. It is not
+  the number of subsequently accepted output tokens. The existing classifier
+  and its chunked-prefill semantics are at `vllm/v1/utils.py:780-813`.
+
+## Existing facilities to reuse
+
+These are zero-patch wins:
+
+| Need | Existing v0.24.0 facility | Design consequence |
+|---|---|---|
+| Scheduled request and token map | `SchedulerOutput.num_scheduled_tokens`, total tokens, preempted IDs, and new/cached request data at `vllm/v1/core/sched/output.py:180-219` | Derive composition in the scheduler; do not add fields to `SchedulerOutput`. |
+| Prefill/decode classification | `compute_iteration_details()` at `vllm/v1/utils.py:780-813` | Reuse the exact definition already used by the iteration log. |
+| Queue/KV/prefix stats | `SchedulerStats` at `vllm/v1/metrics/stats.py:170-198` and construction at `vllm/v1/core/sched/scheduler.py:2228-2264` | Reuse field semantics, but snapshot at schedule time to avoid async misattribution. |
+| Per-step CUDA-graph descriptor | `CUDAGraphStat` at `vllm/compilation/cuda_graph.py:32-37`, populated at `vllm/v1/worker/gpu_model_runner.py:3919-3934` | Reuse the object; only broaden the condition under which it is returned. |
+| Sampled CPU/CUDA tracing | profiler config and schedule at `vllm/config/profiler.py:33-105`, endpoints at `vllm/entrypoints/serve/profile/api_router.py:21-45` | Layer 2 needs configuration and orchestration, not a new profiler implementation or endpoint. |
+| Exact routed expert IDs | routed-expert capture callback and buffers at `vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58-84` and `vllm/v1/worker/gpu_model_runner.py:7382-7437` | Use only in a separate sampled Layer-2 run; it is too expensive for Layer 1. |
+
+## Layer 1: always-on composition records
+
+### Data flow and hook placement
+
+```text
+Scheduler.schedule()
+  -> snapshot request composition, queues, KV and prefix deltas
+  -> SchedulerOutput -> TP workers -> ModelRunnerOutput.cudagraph_stats
+  -> Scheduler.update_from_output()
+  -> finalize the same step record -> bounded queue -> background JSONL writer
+```
+
+The scheduler is the ownership boundary for Layer 1. It sees the full logical
+batch once, so recording in each TP worker would duplicate data and require an
+aggregation protocol.
+
+Exact proposed hooks against the pinned clone:
+
+1. **Initialize one recorder in the base scheduler constructor.** The scheduler
+   already owns stats state and passes the normal `log_stats` setting into the
+   KV-cache manager (`vllm/v1/core/sched/scheduler.py:78-87` and
+   `vllm/v1/core/sched/scheduler.py:250-260`). If `VLLM_OPPROF_DIR` is set,
+   construct the recorder and reject `log_stats=False`.
+2. **Begin a record inside `Scheduler.schedule()`.** Read prefix counters at
+   method entry, then finish the snapshot immediately after `SchedulerOutput`
+   and connector metadata are constructed and before
+   `_update_after_schedule()` mutates request state
+   (`vllm/v1/core/sched/scheduler.py:1012-1100`). Prefix lookup counters are
+   updated synchronously during this schedule call
+   (`vllm/v1/core/kv_cache_manager.py:202-242` and
+   `vllm/v1/core/sched/scheduler.py:675-712,897-915`), so before/after deltas
+   remain attributable even when several batches are in flight.
+3. **Finalize in `Scheduler.update_from_output()`.** Pair the returned
+   `ModelRunnerOutput` with the exact `SchedulerOutput` at
+   `vllm/v1/core/sched/scheduler.py:1464-1477`, and enqueue after normal output
+   processing near `vllm/v1/core/sched/scheduler.py:1791-1803`. Store pending
+   drafts by `id(scheduler_output)` and remove them on finalize; the identifier
+   never leaves the process. This is robust to the batch queue, which retains
+   and later returns the matching output object
+   (`vllm/v1/engine/core.py:519-632`).
+4. **Return the existing CUDA-graph stat when OpProf is enabled.** Change the
+   condition at `vllm/v1/worker/gpu_model_runner.py:3919-3926` from only
+   `observability_config.cudagraph_metrics` to that flag **or**
+   `VLLM_OPPROF_DIR`. The file already imports `vllm.envs` at line 23. No CUDA
+   synchronization or tensor transfer is introduced.
+5. **Close through the existing scheduler shutdown.** Drain and join the writer
+   before the scheduler reports shutdown complete at
+   `vllm/v1/core/sched/scheduler.py:2285-2295`; retain an `atexit` fallback for
+   abnormal embedding/test lifecycles.
+
+The scheduler assigns a monotonically increasing local `step_index` at begin.
+Every real scheduler call is represented, including zero-token steps. A DP-only
+dummy execution has no `SchedulerOutput` and is outside this composition
+stream; if DP is later in campaign scope, add a distinct `dummy_step` marker
+rather than pretending it has a request composition.
+
+### Record schema
+
+Use schema-versioned JSON Lines. An illustrative record is:
+
+```json
+{"schema":1,"engine_id":"dp0-pid1234","step_index":42,"submit_wall_ns":1783761000000000000,"submit_mono_ns":8920000000,"complete_mono_ns":8922371000,"model_executed":true,"scheduled_requests":37,"decode_batch_size":32,"prefill_requests":5,"prefill_tokens":896,"decode_tokens":32,"chunked_prefill":{"first":2,"middle":1,"final":1,"unsplit":1,"tokens":896,"chunk_size_hist":[0,0,1,1,1,1,1,0,0]},"context_length_hist":[0,0,3,6,12,10,5,1,0,0,0,0],"preemptions":0,"queues":{"running":37,"waiting":9,"deferred":0},"kv":{"total_blocks":120000,"free_blocks":42000,"used_blocks":78000,"usage":0.65},"prefix":{"local":{"requests":2,"queries":4096,"hits":3072,"preempted_requests":0,"preempted_queries":0,"preempted_hits":0},"external":null},"cudagraph":{"hit":true,"runtime_mode":"PIECEWISE","unpadded_tokens":928,"bucket_tokens":1024,"padding_tokens":96},"moe_expert_load":null,"dropped_records_before":0}
+```
+
+Required field semantics:
+
+- `submit_wall_ns` permits joining to external workload logs;
+  `submit_mono_ns` and `complete_mono_ns` provide stable within-process
+  ordering and elapsed time without wall-clock jumps.
+- `scheduled_requests` is the number of entries in
+  `num_scheduled_tokens`. `decode_batch_size`, `prefill_requests`,
+  `prefill_tokens`, and `decode_tokens` reuse vLLM's classifier.
+- `chunked_prefill` contains aggregate split information, never request IDs.
+  A context request is `first` when the scheduled range ends before its prompt
+  ends, `middle` when it was already a prefill chunk and remains incomplete,
+  `final` when a prior chunk completes, and `unsplit` when its prefill completes
+  in one step. `chunk_size_hist` uses token-count buckets
+  `(16, 32, 64, 128, 256, 512, 1024, 2048, +inf)`.
+- `context_length_hist` includes every scheduled request, with fixed upper
+  edges `(128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536,
+  131072, +inf)`. The final array therefore has 12 bins. The example above is
+  illustrative and will be checked against the final chosen edges in tests.
+- `preemptions` is the size of the step's `preempted_req_ids`, created at
+  `vllm/v1/core/sched/scheduler.py:1059-1076`. It is not an interval-derived
+  Prometheus value.
+- Queue lengths and KV blocks are captured immediately after scheduling. KV
+  usage follows the existing null-block-adjusted calculation at
+  `vllm/v1/core/block_pool.py:692-711`. `deferred` means skipped/deferred
+  waiting requests, matching `num_skipped_waiting_reqs` in
+  `vllm/v1/metrics/stats.py:174-183`.
+- Local and external prefix fields are per-schedule deltas copied from the
+  existing mutable counters. The existing stats drain resets those objects at
+  `vllm/v1/core/kv_cache_manager.py:190-200` and
+  `vllm/v1/core/sched/scheduler.py:2235-2242`; OpProf must read, not drain or
+  replace, them. These retain vLLM's lookup semantics: a local lookup can be
+  counted before allocation later rejects that waiting request, so prefix
+  `requests` is not required to be less than or equal to scheduled prefill
+  requests.
+- `cudagraph.hit` is the operational serving definition `runtime_mode != NONE`.
+  `FULL` selects the full graph path and `PIECEWISE` selects graph-wrapped
+  compiled regions; the latter must not be interpreted as full-step graph
+  coverage. Startup normally captures all dispatcher descriptors at
+  `vllm/v1/worker/gpu_model_runner.py:6595-6643`, but the stat itself has no
+  capture-versus-replay bit, so a rare lazy first capture cannot be
+  distinguished from replay. `bucket_tokens` is `num_padded_tokens` in the
+  existing stat.
+- `moe_expert_load` is explicitly null in Layer 1. This avoids accidentally
+  presenting unavailable data as zero.
+
+### Encoding, buffering, and flush
+
+JSONL is recommended over a compact binary ring buffer for Phase 1 because the
+schema will evolve during campaign bring-up, records need to be directly
+inspectable beside Kineto traces, and `msgspec` is already a vLLM dependency
+(`requirements/common.txt:35`).
+Encoding one small dictionary with a reused `msgspec.json.Encoder` avoids the
+standard `json` module's larger CPU cost. Once the schema stabilizes, the same
+record can be moved to a binary format only if Phase 2 measurements show JSONL
+is the bottleneck.
+
+The foreground path encodes once and performs non-blocking `put_nowait()` into
+a bounded queue of 8192 records. A single daemon writer thread drains into one
+file per EngineCore/DP rank and process. It flushes userspace buffers every
+1 MiB or one second, whichever comes first, and on clean process exit; it never
+calls `fsync()` per record. When the queue is full, the serving thread drops the
+new record, increments a counter, and the next successful record reports the
+gap in `dropped_records_before`. Shutdown emits a footer with encoded, written,
+and dropped counts. The file name includes schema, DP rank, PID, and start time,
+but not TP rank because there is one scheduler record stream.
+
+The sole on/off switch is:
+
+```text
+VLLM_OPPROF_DIR=/absolute/output/directory
+```
+
+Unset or empty means a true no-op: no recorder, queue, thread, record
+construction, or graph-stat broadening. Directory validation and a clear
+startup log happen before serving. Runtime toggling is intentionally omitted;
+it adds synchronization and ambiguous partial files without helping the
+campaign.
+
+### Expected cost and Phase 2 overhead gate
+
+These are estimates, not measurements:
+
+- composition and two fixed histograms: about 20-80 microseconds per step,
+  linear in scheduled requests;
+- `msgspec` encoding plus a non-blocking queue insertion: about 5-20
+  microseconds per step;
+- CUDA-graph stat construction: below 2 microseconds and no GPU sync;
+- disk I/O: off the serving critical path, subject to bounded-queue drops.
+
+The expected foreground total is roughly 25-100 microseconds per step. At a
+2 ms decode step, the high end would exceed 3%, so the budget is a measurement
+gate, not a claim.
+
+Phase 2 should use the same Qwen3 checkpoint, request trace, random seed,
+hardware, vLLM commit, TP/EP topology, CUDA-graph configuration, and cache
+state for off/on comparisons. Alternate off/on ordering, warm up before each
+measurement, and run at least five paired repeats. Report throughput and
+p50/p95 TTFT and TPOT, plus CPU utilization, log bytes per step, queue high
+watermark, and dropped-record count. The acceptance rule is less than 3%
+regression for every declared primary serving metric, with bootstrap confidence
+intervals reported. Whether the point estimate or upper 95% bound is the hard
+gate is an open decision.
+
+Correctness checks for every run are: contiguous step indices except explicitly
+reported drops, scheduled token sums matching the prefill/decode split,
+histogram counts matching scheduled request/chunk counts, KV ratio in `[0,1]`,
+non-negative counters, and identical generated outputs for the deterministic
+test trace.
+
+## Layer 2: sampled kernel windows
+
+### Trigger and window
+
+Use the existing profiler API with a run-specific output directory and
+`ignore_frontend=true`. The worker start/stop RPC already fans out to all
+workers (`vllm/v1/executor/abstract.py:256-257` and
+`vllm/v1/executor/multiproc_executor.py:340-402`), and each GPU worker creates a
+CPU+CUDA profiler with a rank-qualified trace name
+(`vllm/v1/worker/gpu_worker.py:929-980`). No Layer-2 vLLM code patch is needed.
+
+Recommended initial configuration:
+
+```text
+profiler=torch
+torch_profiler_dir=<run>/kineto
+ignore_frontend=true
+delay_iterations=0
+max_iterations=8
+wait_iterations=0
+warmup_iterations=2
+active_iterations=8
+torch_profiler_record_shapes=false
+torch_profiler_with_memory=false
+torch_profiler_with_stack=false
+torch_profiler_with_flops=false
+torch_profiler_use_gzip=true
+torch_profiler_dump_cuda_time_total=true
+```
+
+The primary trigger should be step-count based: an external campaign controller
+POSTs `/start_profile` immediately before a desired composition regime. The
+built-in schedule records two warm-up plus eight active model iterations and
+`max_iterations=8` then stops the underlying profiler on the following worker
+step. The “exceeds max” control semantics are explicit at
+`vllm/profiler/wrapper.py:83-114` and in
+`tests/v1/worker/test_gpu_profiler.py:77-98`; the controller should still POST
+`/stop_profile` afterward to clear the active control state.
+Worker iteration boundaries already call `profiler.step()` and annotate context
+and generation token counts at `vllm/v1/worker/gpu_worker.py:803-827`. On-demand
+manual POST remains useful for debugging. A time-based trigger is a fallback
+for live traffic but is less reproducible because it yields a variable number
+of steps. Phase 1 should confirm that the trace contains exactly eight active
+iteration annotations; profiler scheduling and trace callbacks live at
+`vllm/profiler/wrapper.py:159-226,290-307`.
+
+The profile directory should be unique per campaign server run. The HTTP start
+endpoint does not accept a `profile_prefix`, and a worker that is restarted for
+another window retains the trace name chosen on first initialization
+(`vllm/v1/worker/gpu_worker.py:939-975`). Multiple windows in one server run
+therefore share a directory and need a controller manifest containing start/
+stop wall times and produced filenames. Use a new directory only when the
+server is restarted; adding a new endpoint parameter is not justified for
+Phase 1. Every TP worker emits its own trace, and the rank suffix contains DP,
+PP, TP, DCP, EP, and global rank information
+(`vllm/distributed/utils.py:664-695`). Keep these traces separate and aggregate
+only offline.
+
+### CUDA graphs and Kineto interpretation
+
+Keep CUDA graphs enabled in the primary sampled windows so kernel time reflects
+the serving configuration. The pinned code profiles CUDA activity
+(`vllm/v1/worker/gpu_worker.py:953-960`) while a captured path invokes
+`CUDAGraph.replay()` instead of rerunning the Python callable
+(`vllm/compilation/cuda_graph.py:233-360`). Therefore Kineto/CUPTI can observe
+CUDA activity launched during replay, but the Python/PyTorch operator scopes
+that created the graph are not re-executed and cannot be assumed to retain
+per-layer attribution. vLLM explicitly documents the related limitation that
+layerwise NVTX tracing does not work with CUDA graphs
+(`vllm/config/observability.py:60-63`).
+
+The clone does not contain a stronger guarantee about whether a particular
+PyTorch/CUPTI build will present every graph node as an individually named
+kernel versus a coarser graph-launch view. Treat that as an empirical Phase 2
+check. Run one matched eager (`cudagraph_mode=NONE`) taxonomy window to identify
+kernel families, but do not use its timings as the production baseline.
+
+### Calibration and MoE routing
+
+Join each profiler iteration annotation to Layer-1 records by rank-independent
+step order, the profiler window marker, timestamps, and the prefill/decode token
+counts. Aggregate kernels into attention, router/top-k, expert GEMMs, dense
+GEMMs, normalization/activation, collectives, and sampling. For TP, report both
+sum-of-rank GPU work and the maximum per-rank critical-path time. Fit or tabulate
+kernel-time attribution conditioned on composition, context histogram, graph
+runtime mode, and capture bucket, then validate on held-out windows rather than
+the same samples used for calibration.
+
+For the Phase 2 “operator time approximately equals iteration time” gate, do
+not naively add overlapping kernels. Compute the union of CUDA kernel intervals
+per rank, use the maximum rank as the distributed GPU critical path, and retain
+an explicit CPU/queue/unattributed residual against Layer 1's submit-to-complete
+span. Operator-category interval unions plus the residual must reconstruct the
+span; summed GPU work is reported separately and may legitimately exceed wall
+time.
+
+Exact per-layer expert loads are Layer-2-only. A separate server start with
+`--enable-return-routed-experts` exposes per-token, per-layer top-k IDs through
+the existing router callback (`vllm/model_executor/layers/fused_moe/router/base_router.py:233-278`).
+Aggregate those arrays offline into per-layer expert counts, entropy, max/mean
+load, coefficient of variation, and top-k imbalance. Do not enable it in the
+normal Layer-1 or baseline Kineto run: its worker transit buffer costs a few MB,
+and the scheduler-side slot buffer can reach multiple GB with a CPU fancy-index
+copy per step (`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:223-305`).
+
+## Patch surface estimate
+
+Proposed implementation and tests:
+
+| File | Approximate changed/new lines | Purpose |
+|---|---:|---|
+| `vllm/envs.py` | 4 | Declare and parse `VLLM_OPPROF_DIR`. |
+| `vllm/v1/opprof.py` (new) | 220 | Schema, schedule snapshot/deltas, histograms, pending-step pairing, encoder, bounded writer, drop/footer handling. |
+| `vllm/v1/core/sched/scheduler.py` | 32 | Initialize recorder; begin before state mutation; finalize with the matching model output. |
+| `vllm/v1/worker/gpu_model_runner.py` | 4 | Return existing `CUDAGraphStat` when either built-in metrics or OpProf needs it. |
+| `tests/v1/core/test_opprof.py` (new) | 190 | Schema/invariants, chunk classification, async pairing, disabled no-op, bounded-queue drops, shutdown flush. |
+| **Total** | **5 files / about 450 lines** | About 260 production lines and 190 test lines. |
+
+No patch is proposed for `ProfilerConfig`, profile endpoints, profiler wrapper,
+`SchedulerOutput`, `SchedulerStats`, Prometheus/logging, CUDA kernels, fused MoE
+kernels, the Qwen model, or frontend APIs. If support for
+`--disable-log-stats` is required, add about 12 lines in
+`vllm/v1/core/kv_cache_manager.py`, making the estimate six files and about 462
+lines; the smaller fail-fast design is recommended for this campaign.
+
+## Risks and mitigations
+
+- **Foreground CPU overhead:** request scanning and JSON encoding are the likely
+  hot spots. Use fixed-size arrays, a reused encoder, no raw lists, no blocking
+  I/O, and enforce the Phase 2 gate.
+- **Async-scheduling races:** queue/KV/prefix state observed during
+  `update_from_output()` may include later in-flight schedules. Snapshot it
+  inside the synchronous `schedule()` call; only the immutable returned
+  CUDA-graph stat is added at completion. Pair by the exact `SchedulerOutput`
+  object and assert one begin/one finalize.
+- **TP and EP aggregation:** Layer 1 is scheduler-owned and emitted once. Layer
+  2 remains per rank. Interpret TP critical path as the slowest rank and, for an
+  expert-parallel routed-expert sample, aggregate expert ownership offline.
+- **Log volume:** at 500 steps/s, a 1.0 KiB record is about 44 GB/day per
+  EngineCore. Rotate by campaign run, compress completed files, and use the
+  bounded queue/drop counters. Production-long captures need a later sampling
+  or rotation policy; Phase 1 files should be bounded by run duration.
+- **Graph semantics:** `PIECEWISE` is partial graph coverage, not a binary full
+  hit. Preserve `runtime_mode`, unpadded tokens, bucket, and padding rather than
+  reducing the record to one boolean.
+- **Profiler perturbation:** Kineto has medium-to-high overhead and trace flushes
+  can stall. Use short windows, unique directories, disabled stacks/shapes and
+  memory by default, and never use Layer-2 latency as an unprofiled performance
+  result.
+- **Backend drift:** the unquantized SM90 oracle prioritizes Triton, but
+  shape-specific fallbacks remain possible. Record startup backend logs and the
+  full checkpoint/parallel/quantization configuration with every run.
+
+## Open decisions for review
+
+1. Approve JSONL with `msgspec`, the two proposed histogram edge sets, and a
+   bounded 8192-record writer queue.
+2. Approve exact expert-load collection as Layer-2-only, rather than adding a
+   new always-on GPU histogram kernel.
+3. Confirm the Qwen3 checkpoint precision/quantization and intended TP/EP/DP
+   topology; the Triton backend conclusion depends on these inputs.
+4. Choose the first profiler window: recommended two warm-up plus eight active
+   iterations, or a longer active window.
+5. Decide whether the 3% gate applies to metric point estimates or to the upper
+   bound of their 95% confidence intervals.
+6. Confirm that campaign runs may reject `--disable-log-stats`; supporting it
+   adds one file and about 12 lines.
+
+## Data sanity block
+
+- **Patch estimate:** n=5 files; per-file line-estimate min=4, max=220,
+  distinct values=4; total about 450 lines, of which about 260 are production
+  and 190 are tests.
+- **Histogram definitions:** n=2 arrays; bin-count min=9, max=12,
+  distinct=2. In the illustrative record, context-bin sum=37 equals scheduled
+  requests, chunk-bin sum=5 equals prefill requests, KV used=120000-42000, and
+  graph padding=1024-928.
+- **Estimated foreground components:** n=3 timed components; min is below 2
+  microseconds for graph-stat construction, max is 80 microseconds for
+  composition; distinct ranges=3. Their conservative combined foreground
+  estimate is 25-100 microseconds per step.
+- **Invariants checked:** counters and histogram bins are non-negative; KV
+  usage and prefix hit ratios are bounded in `[0,1]`; graph bucket is at least
+  unpadded tokens; `NONE` is the only graph miss mode; no raw request IDs or
+  context-length lists are serialized; one scheduler stream avoids TP
+  duplicates.
+- **Measurement status:** n=0 benchmark runs; min/max and cross-configuration
+  distinctness are not applicable. The cost figures are estimates and cannot
+  support an overhead conclusion until Phase 2.

docs/opprof/phase0-recon-vllm-0.24.0.md

@@ -0,0 +1,446 @@
+# Phase 0 recon: vLLM 0.24.0 observability
+
+Status: source recon and patch design complete; **no instrumentation patch was
+implemented**.
+
+Campaign target: operator-level, pattern-conditioned profiling of
+Qwen3-30B-A3B serving on NVIDIA H20. This report is based on the pinned source
+clone only. No GPU was used, no package or model was installed, and no dash
+host was contacted.
+
+Unless explicitly prefixed with `v0.20.0:`, every source path and line number
+below is relative to `/home/gahow/phd/vllm-v0.24.0` at the pinned commit.
+
+## Outcome
+
+vLLM 0.24.0 already contains most of the raw ingredients, but they are split
+across scheduler internals, aggregate metrics, worker traces, and an optional
+routed-expert return path. The smallest useful patch is therefore not a new
+profiler. It is one scheduler-owned per-step record that joins existing batch
+composition to the existing `CUDAGraphStat`; short torch-profiler windows and
+exact expert routes remain sampled Layer 2 facilities.
+
+The proposed patch is detailed in
+[`patch-design.md`](patch-design.md). Estimated surface: five files and about
+450 lines including tests, with about 260 production lines.
+
+## 1. Source pin and compatibility record
+
+The requested tag existed and the exact requested clone command completed
+successfully:
+
+```text
+git clone --depth 1 --branch v0.24.0 https://github.com/vllm-project/vllm /home/gahow/phd/vllm-v0.24.0
+```
+
+| Field | Pinned value |
+|---|---|
+| Repository | `https://github.com/vllm-project/vllm` |
+| Local clone | `/home/gahow/phd/vllm-v0.24.0` |
+| Tag | `v0.24.0` (`git describe --tags --exact-match`) |
+| Commit | `ee0da84ab9e04ac7610e28580af62c365e898389` |
+| Clone completion | `2026-07-11T08:26:16Z` (`2026-07-11 16:26:16 +08:00`) |
+| Checkout | Detached HEAD, clean |
+| Top-level entries | 45, including hidden entries and `.git` |
+| Listing SHA-256 | `75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58` |
+
+The listing digest is reproducible from entry names only, not file contents:
+
+```text
+LC_ALL=C find . -mindepth 1 -maxdepth 1 -printf '%f\n' |
+  LC_ALL=C sort | sha256sum
+```
+
+This definition includes `.git`, sorts bytewise under the C locale, and hashes
+the 45 LF-terminated root names. It is recorded to make the otherwise ambiguous
+phrase “sha256 of the top-level directory listing” precise.
+
+### Torch and CUDA requirements
+
+- Build and NVIDIA runtime requirements pin `torch==2.11.0` in
+  `pyproject.toml:1-13`, `requirements/build/cuda.txt:1-10`, and
+  `requirements/cuda.txt:1-10`; CMake repeats 2.11.0 as the supported CUDA and
+  ROCm torch version at `CMakeLists.txt:62-72`.
+- The NVIDIA requirements also pin `torchvision==0.26.0`,
+  `flashinfer-python==0.6.12`, `flashinfer-cubin==0.6.12`, and CUDA-13 extras
+  for CUTLASS DSL and Humming kernels (`requirements/cuda.txt:6-29`). Setup
+  strips/substitutes those extras for CUDA 12 builds
+  (`setup.py:1066-1083`).
+- The source default is CUDA 13.0 (`vllm/envs.py:85-87,555-563`) and the Docker
+  default is CUDA 13.0.2 (`docker/Dockerfile:14-42`). Wheel detection maps CUDA
+  major 12 to `cu129` and major 13 to `cu130` (`setup.py:528-568`).
+- The pinned installation document still describes the default precompiled
+  binaries as CUDA 12.9 and lists CUDA 12.8 and 13.0 variants
+  (`docs/getting_started/installation/gpu.cuda.inc.md:4-19,24-54`). This is a
+  packaging/default distinction, not evidence that 13.0 is unsupported.
+- NVIDIA compute capability 7.5+ is documented, and SM90 is in every relevant
+  CMake CUDA architecture set (`docs/getting_started/installation/gpu.cuda.inc.md:9-19`;
+  `CMakeLists.txt:105-118`). Optional FA3 and DeepGEMM builds need CUDA 12.3+,
+  FlashMLA needs 12.9+, and the Hopper CUTLASS MoE build needs 12.3+
+  (`setup.py:1111-1139`; `CMakeLists.txt:844-868`).
+
+Implication for dash0 later: H20/SM90 is a supported architecture, but the
+actual driver and installed CUDA were intentionally not queried in Phase 0.
+Phase 2 must choose a torch 2.11.0-compatible `cu129` or `cu130` artifact after
+checking the host driver. A wheel built against a different torch/CUDA build is
+not interchangeable; the source documentation explicitly warns about binary
+incompatibility (`docs/getting_started/installation/gpu.cuda.inc.md:14-19`).
+
+## 2. Built-in observability inventory
+
+### 2.1 Torch profiler integration
+
+**Configuration and control.** v0.24.0 does not contain the legacy
+`VLLM_TORCH_PROFILER_DIR` symbol: a repository-wide exact-name search at the
+pinned commit returned zero matches. Profiling is now configured through
+`ProfilerConfig`: `profiler` is `torch` or `cuda`, and
+`torch_profiler_dir` holds the absolute/URI output location
+(`vllm/config/profiler.py:33-46,125-145`). Optional controls include stacks,
+FLOPs, gzip, CUDA-time tables, shapes, memory, frontend exclusion, delayed
+start, maximum iterations, and wait/warmup/active iteration scheduling
+(`vllm/config/profiler.py:48-105`).
+
+The documented server form is a `--profiler-config` JSON object, for example
+`{"profiler":"torch","torch_profiler_dir":"/abs/run/kineto"}`, followed by
+POSTs to the two endpoints (`docs/contributing/profiling.md:46-85`).
+
+When profiler config is present, the serving router exposes POST
+`/start_profile` and `/stop_profile`; without profiler config the router is not
+attached (`vllm/entrypoints/serve/profile/api_router.py:21-45`). The endpoints
+are therefore not an unauthenticated always-on feature independent of server
+configuration; they exist within the configured serving application and should
+be protected like the rest of the control plane.
+
+**Captured activity.** Each GPU worker creates a torch profiler with both CPU
+and CUDA activities, using the configured shapes/memory/stack/FLOPs switches
+(`vllm/v1/worker/gpu_worker.py:929-980` and
+`vllm/profiler/wrapper.py:159-226`). TensorBoard-compatible traces are written
+with an optional gzip handler. The wrapper also writes CPU/CUDA aggregate
+tables; only rank 0 prints the table to stdout
+(`vllm/profiler/wrapper.py:251-287`). The source documentation calls the
+profiler medium-overhead and warns that trace size and final flushing can be
+large (`docs/contributing/profiling.md:1-38`).
+
+The V1 worker calls `profiler.step()` once per execution and wraps model
+execution in a record-function name containing context/generation request and
+token counts (`vllm/v1/worker/gpu_worker.py:803-827,895-898`). If frontend
+profiling is not ignored, `AsyncLLM` also creates a separate CPU-only trace
+(`vllm/v1/engine/async_llm.py:178-200`).
+
+**TP behavior.** `EngineCore.profile()` delegates to the executor
+(`vllm/v1/engine/core.py:662-663`), whose profile RPC is a collective worker
+call (`vllm/v1/executor/abstract.py:256-257`). Multiprocessing broadcasts calls
+that do not declare a unique output rank, so the profile call reaches every TP
+worker (`vllm/v1/executor/multiproc_executor.py:340-402`). Each worker emits its
+own rank-qualified trace; the suffix includes DP/PP/TP/DCP/EP/global rank
+components (`vllm/distributed/utils.py:664-695`). For TP, kernel analysis must
+therefore preserve per-rank files, use the slowest rank for critical-path time,
+and optionally sum ranks to quantify total GPU work.
+
+### 2.2 Iteration-level scheduler data and exported metrics
+
+The V1 scheduler interface states that each `schedule()` output corresponds to
+one model forward and maps request IDs to scheduled tokens
+(`vllm/v1/core/sched/interface.py:51-80`). `SchedulerOutput` already carries
+new/cached request data, per-request and total scheduled tokens, common-prefix
+blocks, and the request IDs preempted in that step
+(`vllm/v1/core/sched/output.py:180-219`). The base scheduler constructs this
+object at `vllm/v1/core/sched/scheduler.py:1012-1076`, then advances request
+computed-token and chunk state at `vllm/v1/core/sched/scheduler.py:1130-1155`.
+
+What is available today:
+
+| Requested datum | Internal per-step source | Externally exposed today |
+|---|---|---|
+| Batch size | `len(SchedulerOutput.num_scheduled_tokens)`; the map is exact | No raw per-step batch-size Prometheus series. Optional iteration log reports context and generation request counts. |
+| Scheduled prefill/decode tokens | `compute_iteration_details()` classifies new/cached context requests and sums scheduled tokens (`vllm/v1/utils.py:766-813`) | Optional log reports exact scheduled context/generation counts and elapsed time around the execution wait (`vllm/v1/engine/core.py:433-472`). Prometheus prompt/generation counters are cumulative output-side stats, not a raw scheduled-step stream. |
+| Chunked-prefill splits | `Request.is_prefill_chunk`, `num_computed_tokens`, and prompt/output state exist (`vllm/v1/request.py:130-175`); extended chunks are classified as context | No chunk ordinal, first/middle/final split, or chunk-size series. These must be derived before `_update_after_schedule()` mutates state. |
+| Preemptions | Exact step set in `SchedulerOutput.preempted_req_ids` | Stdout aggregates over its logging interval; `vllm:num_preemptions` is cumulative (`vllm/v1/metrics/loggers.py:219-281,624-631`). |
+| Running/waiting/deferred queues | `SchedulerStats.num_running_reqs`, `num_waiting_reqs`, and `num_skipped_waiting_reqs` (`vllm/v1/metrics/stats.py:170-183`) | Latest-value gauges `vllm:num_requests_running`, `vllm:num_requests_waiting`, and reason-labeled capacity/deferred gauges (`vllm/v1/metrics/loggers.py:453-496`). Running queue size is not the same as scheduled batch size. |
+| KV-cache usage | Scheduler reads `kv_cache_manager.usage`; the block pool computes a null-block-adjusted ratio in `[0,1]` (`vllm/v1/core/kv_cache_manager.py:181-188`; `vllm/v1/core/block_pool.py:692-711`) | Latest-value `vllm:kv_cache_usage_perc`; stdout prints the latest percent (`vllm/v1/metrics/loggers.py:250-260,524-532`). Raw total/free blocks are not exported per step. |
+| Prefix-cache counters | `PrefixCacheStats` contains request/query/hit and separately preempted request/query/hit counters (`vllm/v1/metrics/stats.py:114-143`) for local and optional connector caches | Prometheus exposes cumulative token queries/hits for local and external caches (`vllm/v1/metrics/loggers.py:547-565,1063-1101`); stdout reports interval-derived hit rates. It does not expose the preempted sub-counters separately. |
+| Total step tokens | Exact scheduled total is in `SchedulerOutput` | `vllm:iteration_tokens_total` is a histogram observed from output-side computed prompt plus generation tokens (`vllm/v1/metrics/loggers.py:693-724,1148-1171`), not a joinable raw step record. |
+
+`IterationStats` also has a wall timestamp, output-side prompt/generation data,
+preemptions, and finished-request latency samples
+(`vllm/v1/metrics/stats.py:325-347`). `Scheduler.make_stats()` produces latest
+queue/KV state, drains prefix stats, and passes through the step's CUDA-graph
+stat when an output is processed
+(`vllm/v1/core/sched/scheduler.py:2228-2264`). These are valuable existing
+plumbing, but async scheduling can schedule newer batches before an older
+output is processed. Consequently, those latest/drained values are not a safe
+per-step composition join without a schedule-time snapshot.
+
+Bottom line: no new scheduler instrumentation is needed to discover batch and
+token composition, preemptions, queues, KV use, or prefix events. A patch is
+needed only to serialize their **same-step association** and the missing
+histograms/chunk split.
+
+### 2.3 CUDA-graph observability
+
+Yes, v0.24.0 can determine the runtime graph mode and capture bucket for each
+model step internally:
+
+- `CUDAGraphStat` records unpadded tokens, padded tokens, padding, and runtime
+  mode (`vllm/compilation/cuda_graph.py:32-37`).
+- The GPU model runner asks the dispatcher for a concrete runtime mode and
+  `BatchDescriptor`, then conditionally attaches the stat to its output
+  (`vllm/v1/worker/gpu_model_runner.py:3822-3934`). It reaches the scheduler in
+  `ModelRunnerOutput` and `update_from_output()`
+  (`vllm/v1/outputs.py:231-281`;
+  `vllm/v1/core/sched/scheduler.py:1464-1477`).
+- The dispatcher is the source of truth: it pads to a captured descriptor,
+  tries `FULL`, then `PIECEWISE`, and returns `NONE` when no graph key matches
+  (`vllm/v1/cudagraph_dispatcher.py:235-324`; the design explanation is at
+  `docs/design/cuda_graphs.md:81-120`). At serving time, `FULL` and `PIECEWISE`
+  dispatch to a captured wrapper; `NONE` calls the runnable directly
+  (`vllm/compilation/cuda_graph.py:233-360`).
+
+Thus `runtime_mode != NONE` is a graph-path dispatch, and `num_padded_tokens` is
+the selected token bucket. `PIECEWISE` means graph-wrapped compiled pieces, not
+a full-step graph hit. Startup normally captures all dispatcher descriptors
+(`vllm/v1/worker/gpu_model_runner.py:6595-6643`), but `CUDAGraphStat` has no
+capture-versus-replay bit; a lazy first capture and a replay have the same stat.
+With built-in `cudagraph_metrics`, the stat is aggregated
+into a frequency table and printed at the logging interval
+(`vllm/config/observability.py:56-58`;
+`vllm/compilation/cuda_graph.py:40-124`); it is not exposed as a raw per-step
+Prometheus event. OpProf only needs to preserve the already-computed object on
+every enabled step.
+
+Capture modes and sizes live in `CompilationConfig`:
+`cudagraph_mode`, `cudagraph_capture_sizes`, and
+`max_cudagraph_capture_size` are defined at
+`vllm/config/compilation.py:587-688`. The V1 default is
+`FULL_AND_PIECEWISE`. Final sizes are generated or validated in
+`VllmConfig._set_cudagraph_sizes()`; the default is 1/2/4, then steps of 8 below
+256 and 16 above it, bounded by token/batch configuration, with user overrides
+written back to the final list (`vllm/config/vllm.py:1635-1800`).
+
+### 2.4 MoE routing and Qwen3-30B-A3B backend
+
+Qwen3 MoE creates a replicated gate and `FusedMoE` with the model's number of
+experts and top-k, and normally runs its router internally
+(`vllm/model_executor/models/qwen3_moe.py:137-249`). The causal-LM wrapper
+retains the list of MoE layers and their expert topology
+(`vllm/model_executor/models/qwen3_moe.py:662-729`). Router top-k IDs are
+available immediately after routing and before EPLB remapping
+(`vllm/model_executor/layers/fused_moe/router/base_router.py:233-278`).
+
+There is an exact, built-in access path: `--enable-return-routed-experts`
+(`vllm/config/model.py:209-215`; `vllm/engine/arg_utils.py:826-830`). It installs
+per-layer callbacks, writes top-k IDs to preallocated GPU buffers, copies the
+step to pinned CPU memory, and returns it through `ModelRunnerOutput`
+(`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58-84`;
+`vllm/v1/worker/gpu_model_runner.py:3652-3666,4637-4683,7382-7437`). This is
+reachable without modifying the fused kernels or saving router logits.
+Raw router-logit tensors are not plumbed to engine outputs; exposing those
+would be deeper and much higher-volume surgery than using the existing top-k
+ID callback.
+
+It is not cheap enough for Layer 1. The per-worker transit buffer is a few MB;
+the scheduler's whole-slot buffer can reach multiple GB and performs a CPU
+fancy-index write each step
+(`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:223-305`;
+the scheduler store is at `vllm/v1/core/sched/scheduler.py:1501-1520`). It is
+also incompatible with pipeline parallelism greater than one and with KV
+connectors (`vllm/config/vllm.py:871-893`). Exact per-layer expert load should
+therefore be collected in a separate sampled Layer-2 run and reduced offline.
+EPLB has load state, but enabling it adds communication and its logs are
+aggregate balancing summaries; it is not a free observability tap
+(`vllm/distributed/eplb/eplb_state.py:62-171,478-575`).
+
+For the normal unquantized BF16/FP16 Qwen3-30B-A3B configuration on H20/SM90,
+the automatic unquantized oracle explicitly moves both FlashInfer backends
+behind Triton on Hopper because they are expected to be slower
+(`vllm/model_executor/layers/fused_moe/oracle/unquantized.py:43-86`). It selects
+the first supported backend and documents possible shape-specific fallbacks
+(`vllm/model_executor/layers/fused_moe/oracle/unquantized.py:152-250`). The
+primary backend is therefore **Triton** under those assumptions. A quantized
+checkpoint, LoRA, expert activation format, or explicit `--moe-backend` can
+change that conclusion and must be recorded with the campaign run.
+
+### 2.5 V1 execution loop and natural hooks
+
+The control path is:
+
+```text
+EngineCore busy loop
+  -> Scheduler.schedule()
+  -> executor.execute_model(SchedulerOutput)
+  -> GPUWorker -> GPUModelRunner.execute_model()
+  -> Scheduler.update_from_output(SchedulerOutput, ModelRunnerOutput)
+```
+
+Evidence and hook implications:
+
+- `EngineCore.run_busy_loop()` and `_process_engine_step()` drive steps at
+  `vllm/v1/engine/core.py:1259-1317`.
+- The normal step retains the exact scheduler output across the future and
+  passes it back to `update_from_output()`
+  (`vllm/v1/engine/core.py:479-508`). The batch-queue path does the same for
+  multiple in-flight batches (`vllm/v1/engine/core.py:519-632`).
+- The scheduler's stable interface is schedule/update at
+  `vllm/v1/core/sched/interface.py:51-107`. The base implementation builds the
+  output at `vllm/v1/core/sched/scheduler.py:1012-1100`, mutates request state
+  immediately afterward at lines 1130-1155, and consumes model results at
+  lines 1464-1803.
+- Worker execution wraps `gpu_model_runner.execute_model()` at
+  `vllm/v1/worker/gpu_worker.py:895-898`; the model runner's entry is
+  `vllm/v1/worker/gpu_model_runner.py:4055-4069`.
+
+The natural Layer-1 hooks are therefore in the base scheduler: snapshot after
+the `SchedulerOutput` is complete but before request state advances, then
+finalize with the matching model output. This covers the async subclass and
+avoids frontend/output aggregation. The only worker change is broadening the
+existing CUDA-graph-stat condition. Layer 2 uses the existing profile control
+path unchanged.
+
+### 2.6 Relevant changes from 0.20.0
+
+For comparison only, tag `v0.20.0` was fetched into the allowed clone without
+changing the pinned working tree. It resolves to commit
+`88d34c6409e9fb3c7b8ca0c04756f061d2099eb1`.
+
+- Async scheduling and the batch-queue architecture already existed in 0.20.0;
+  `SchedulerConfig` selected `AsyncScheduler` at
+  `v0.20.0:vllm/config/scheduler.py:146-176`, and the core schedule/future/update
+  paths were at `v0.20.0:vllm/v1/engine/core.py:402-431,443-533`. It is
+  incorrect to treat async scheduling as new in 0.24.0.
+- 0.24.0 adds DP prefill cadence/throttling and passes a prefill-throttle flag
+  into `schedule()` (`vllm/config/scheduler.py:140-161` and
+  `vllm/v1/engine/core.py:474-490`). It also hardens zero-token/dummy-iteration
+  handling (`vllm/v1/engine/core.py:433-472`) and changes when the batch queue
+  fills.
+- The 0.24 scheduler contains explicit scheduled/processed sequence fences and
+  snapshots state that must survive later async schedules, including routed
+  experts (`vllm/v1/core/sched/scheduler.py:293-322,1093-1165`). That is direct
+  evidence that an OpProf hook must not read mutable request/queue state only
+  when an older output returns.
+- `vllm/config/profiler.py`, `vllm/profiler/wrapper.py`, and
+  `vllm/entrypoints/serve/profile/api_router.py` are byte-identical between the
+  two tags (`git diff --quiet v0.20.0..v0.24.0` returned success). Profiler
+  endpoint semantics did not move; the scheduler/core line numbers and safe
+  composition hook did.
+
+Practical conclusion: do not port a 0.20 line-number patch into EngineCore.
+Hook the 0.24 base scheduler's schedule/update pair so sync, async, and batch
+queue paths share one implementation.
+
+## 3. Proposed dual-layer design
+
+Layer 1 emits one schema-versioned JSONL record per scheduler step from the
+EngineCore/scheduler process. It contains step/timestamps; scheduled and decode
+batch counts; scheduled prefill/decode tokens; aggregate first/middle/final
+chunk information; fixed-bucket context and chunk-size histograms; exact step
+preemptions; schedule-time running/waiting/deferred queues; total/free/used KV
+blocks and ratio; local/external prefix deltas; and CUDA-graph mode/bucket. No
+request IDs or raw context-length lists are written. MoE expert load is null and
+explicitly marked Layer-2-only.
+
+Records are encoded with the existing `msgspec` dependency
+(`requirements/common.txt:35`), placed into a
+bounded non-blocking queue, and drained by one background JSONL writer. It
+flushes every 1 MiB or one second and at clean shutdown, with explicit drop/gap
+counters and no per-step `fsync`. `VLLM_OPPROF_DIR` is the startup-only switch;
+unset is a true no-op. The unmeasured foreground estimate is 25-100
+microseconds per step, so Phase 2 must enforce the less-than-3% overhead gate
+with paired off/on serving runs.
+
+Layer 2 configures the already-built torch profiler and samples short windows,
+initially two warm-up plus eight active worker iterations. Set
+`max_iterations=8`; the wrapper stops the underlying profiler on the subsequent
+step after the counter exceeds the limit, as its existing test documents
+(`vllm/profiler/wrapper.py:83-114`;
+`tests/v1/worker/test_gpu_profiler.py:77-98`). The controller then calls
+`/stop_profile` to clear control state. Step-count/on-demand endpoint control is
+preferred to wall time. Layer 2 writes one trace per TP worker and joins to
+Layer 1 through the existing iteration annotation, order, token counts, and
+timestamps.
+
+Under CUDA-graph replay, the profiler is configured for CUDA activity, but the
+Python callable is not re-executed: vLLM calls `CUDAGraph.replay()`
+(`vllm/compilation/cuda_graph.py:233-360`). CUDA activity may therefore be
+visible through Kineto/CUPTI, while Python/PyTorch scopes cannot be assumed to
+provide the original per-layer attribution. Layerwise NVTX is explicitly
+unsupported with graphs (`vllm/config/observability.py:60-63`). Phase 2 must
+check whether the pinned torch/CUPTI combination exposes individual replayed
+kernels or a coarser graph launch. A matched eager window can identify kernel
+families but must not replace graph-enabled production timing.
+
+Layer-2 kernel time is grouped into attention, router/top-k, expert/dense GEMMs,
+normalization/activation, collectives, and sampling. Per-rank critical-path
+times calibrate a composition/graph-bucket-conditioned Layer-1 attribution;
+held-out profiler windows validate that mapping. Exact routed experts are
+enabled only in a separate sampled server run.
+
+### Patch surface
+
+| File | Approximate lines |
+|---|---:|
+| `vllm/envs.py` | 4 |
+| `vllm/v1/opprof.py` (new) | 220 |
+| `vllm/v1/core/sched/scheduler.py` | 32 |
+| `vllm/v1/worker/gpu_model_runner.py` | 4 |
+| `tests/v1/core/test_opprof.py` (new) | 190 |
+| **Total** | **5 files / about 450 lines** |
+
+Zero-patch wins: profiler config/endpoints/wrapper, `SchedulerOutput`, existing
+metrics plumbing, CUDA kernels, fused MoE/router code, Qwen3 model code, and
+frontend APIs. Supporting OpProf together with `--disable-log-stats` would add
+about 12 lines in `vllm/v1/core/kv_cache_manager.py`; the design recommends
+failing fast instead.
+
+### Main risks
+
+- Histogram construction and JSON encoding are the likely foreground overhead;
+  the 3% budget must be measured, not inferred.
+- Async scheduling can misassociate latest queue/cache state with an older
+  output; snapshot inside `schedule()` and pair by the exact output object.
+- Layer 1 must be scheduler-owned to avoid TP duplicates; Layer 2 remains
+  per-rank and needs offline aggregation.
+- At 500 steps/s and about 1 KiB/record, an uncompressed stream is roughly
+  44 GB/day per EngineCore. Runs must be duration-bounded, with queue/drop
+  counters and post-run compression.
+- Kineto perturbs execution, and graph replay loses Python-scope attribution;
+  it is a sampled calibration instrument, not an always-on latency source.
+
+## 4. Open design decisions
+
+1. Approve JSONL/`msgspec`, an 8192-record queue, context edges
+   `(128, 256, 512, 1K, 2K, 4K, 8K, 16K, 32K, 64K, 128K, +inf)`, and chunk-size
+   edges `(16, 32, 64, 128, 256, 512, 1K, 2K, +inf)`.
+2. Approve exact MoE expert load as Layer-2-only. The alternative is a new GPU
+   histogram path, with a larger patch and an overhead/graph-capture risk.
+3. Confirm checkpoint precision/quantization and intended TP/EP/DP topology.
+   The H20 Triton conclusion assumes unquantized BF16/FP16, auto backend, and no
+   LoRA.
+4. Choose the initial profiler window: recommended two warm-up plus eight
+   active iterations, or a longer active window.
+5. Decide whether the less-than-3% gate applies to point estimates or the upper
+   95% confidence bound for every primary metric.
+6. Confirm that OpProf campaign runs may reject `--disable-log-stats`; support
+   for it increases the patch to six files and about 462 lines.
+
+## Data sanity block
+
+- **Pinned root listing:** n=45 names; lexicographic min=`.buildkite`,
+  max=`vllm`; distinct=45; SHA-256=`75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58`.
+- **Source versions compared:** n=2 tags; min=`v0.20.0`, max=`v0.24.0`;
+  distinct=2; pinned working-tree version remains exactly `v0.24.0`.
+- **Torch pin observations:** n=4 declarations
+  (`pyproject`, build requirements, CUDA runtime requirements, CMake);
+  min=max=`2.11.0`; distinct=1.
+- **Evidence-bearing source files inspected and cited:** n=44
+  distinct v0.24.0 paths; min/max not applicable to categorical paths;
+  v0.20.0 comparison touched n=5 distinct paths; min/max not applicable.
+- **Invariants checked:** exact tag equals `v0.24.0`; HEAD equals the recorded
+  commit; clone is clean; top-level names are unique; requested path was used;
+  torch pins agree; CUDA-graph modes are in `{NONE, PIECEWISE, FULL}`; KV usage
+  definition is bounded in `[0,1]`; scheduled-token total is defined as the sum
+  of per-request scheduled tokens; graph bucket is not smaller than unpadded
+  tokens; no remote/GPU/install/model-download action occurred.
+- **Benchmark/statistical conclusion check:** no performance benchmark was run,
+  so there are no per-configuration curves or measured overhead values on which
+  to perform identical-value, monotonicity, or continuity checks. All overhead
+  numbers in the design are labeled estimates.

docs/opprof/phase3-results.md

@@ -0,0 +1,262 @@
+# OpProf Phase 3 dash0 results
+
+Status: **FINAL A-P3-7 PARTIAL-MATRIX ANALYSIS — H1a INCONCLUSIVE; H1b PASS**.
+
+Date: 2026-07-12. The matrix is permanently frozen at 40/52 accepted measured
+runs and 20/24 complete pattern/config cells. A-P3-7 permits existential
+confirmation on complete comparisons but forbids refutation from incomplete
+coverage. The result is therefore **PARTIAL**: H1b is confirmed by five
+evaluable contrasts, while H1a cannot be evaluated on a valid pattern pair.
+
+The final machine result is
+`runs/opprof-phase3/phase3/metrics.json` (SHA-256
+`dcc44941bfe1eb56c7a069148e5565eddd1ccf966e6914aa16b64bd67d02f0f0`).
+The final analyzer SHA-256 is
+`483c170838a1e81a170d8425b4be22cc1be02ea4bc9492b474daf7ae86186d67`;
+7/7 no-GPU analysis tests passed, including ResourceWarning-as-error.
+
+## Data-sanity result first
+
+All **23/23** machine-checked invariants pass before inference: 40 unique
+accepted primary markers selected only through completed-stage records; exact
+240-second clean windows; zero clean failures; finite moderate rates within
+5%; balanced Layer-1 footer/sidecar accounting; ratios in range; 72 accepted
+trace files with the registered eight-file first-wave deviation; zero other
+GPU processes; exact 20-cell coverage; and the exact A-P3-7 missing-cell and
+missing-contrast sets.
+
+The accepted data contain 77,109 clean completions, 352,907 clean Layer-1
+steps, and 542,350 total Layer-1 records. Device-time classifiability is
+97.05–99.64%. No malformed trace, accounting mismatch, counter underflow,
+private-text leak, or unexpected identical-result family was found.
+
+Declared limitations are not hidden as sanity passes: eight accepted
+saturation trace files were lost during the registered first-wave controller
+cleanup; all four confirmation runs are absent; MoE per-layer CV is unavailable
+because layer scopes cover less than 80% of MoE GEMM time; and only one of nine
+completed C00-moderate patterns passes the Layer-2 inference gates.
+
+## Frozen coverage and A-P3-7 logic
+
+The four incomplete cells, each missing saturation and moderate, are exactly:
+
+- P03/C11;
+- P05/C00;
+- P10/C00-TP2; and
+- P11/C00.
+
+The four missing confirmations are P10, P06, P03, and P01 C00-moderate.
+Formal H1a coverage is therefore the nine completed C00 patterns P01, P02,
+P03, P04, P06, P07, P08, P09, and P10. Formal H1b has six evaluable and two
+non-evaluable frozen contrasts. A-P3-7's asymmetry is mandatory: one valid hit
+can confirm an existential hypothesis, but absence of a hit cannot support the
+original complete-matrix null.
+
+## H1a — operator composition
+
+**Verdict: INCONCLUSIVE; refutation is not allowed.**
+
+All moderate traces were parseable and classifiable, but only P04's two
+windows jointly passed classifiability, `abs(SMD)<=0.25` representativeness,
+and fixed recovery. P01/P02/P03/P09 failed representativeness in both windows;
+P06/P07/P08/P10 additionally failed at least one recovery. Thus only one of
+nine completed patterns is inferentially evaluable, leaving zero pattern pairs,
+zero ranking tests with data, and no Kendall tau-b pairs. No qualifying
+inversion can be accepted, and this is not evidence for a shared ranking.
+
+Descriptively, P04 is attention-led at both loads. P06 and P10 change from
+attention-led at saturation to MoE-GEMM-led at moderate load; the other
+available patterns remain MoE-GEMM-led. These are ranking changes in sampled
+windows, not H1a evidence because their window gates fail.
+
+### Per-pattern operator shares, both load points
+
+Values are the mean percentage across the two Layer-2 windows. `E` means the
+row is inferentially evaluable; `D` means data are available but descriptive
+only because a registered window/recovery gate failed; `NE-cell` means the cell
+is missing; `NE-trace` means the accepted run has no retained trace. `Other` is
+unclassified device activity. Collective, sampler, and dense-GEMM shares round
+to 0.00% in every available C00 row.
+
+| Pattern | Load | Use | Attention | MoE GEMM | Router | Collective | Sampler | Dense GEMM | Norm/elt | KV | Other | Top family |
+|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
+| P01 | saturation | D | 16.32 | 77.81 | 0.85 | 0.00 | 0.00 | 0.00 | 3.90 | 0.10 | 1.02 | MoE GEMM |
+| P01 | moderate | D | 12.09 | 82.43 | 1.03 | 0.00 | 0.00 | 0.00 | 3.38 | 0.11 | 0.96 | MoE GEMM |
+| P02 | saturation | D | 32.67 | 62.39 | 1.05 | 0.00 | 0.00 | 0.00 | 2.98 | 0.16 | 0.74 | MoE GEMM |
+| P02 | moderate | D | 20.91 | 72.53 | 1.64 | 0.00 | 0.00 | 0.00 | 3.68 | 0.25 | 1.00 | MoE GEMM |
+| P03 | saturation | D | 39.11 | 56.60 | 0.44 | 0.00 | 0.00 | 0.00 | 2.98 | 0.03 | 0.84 | MoE GEMM |
+| P03 | moderate | D | 23.32 | 57.45 | 3.54 | 0.00 | 0.00 | 0.00 | 12.29 | 0.83 | 2.57 | MoE GEMM |
+| P04 | saturation | D | 65.88 | 29.77 | 1.16 | 0.00 | 0.00 | 0.00 | 2.30 | 0.17 | 0.72 | Attention |
+| P04 | moderate | **E** | 47.88 | 40.64 | 3.28 | 0.00 | 0.00 | 0.00 | 5.85 | 0.45 | 1.90 | Attention |
+| P05 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — |
+| P05 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — |
+| P06 | saturation | D | 56.59 | 39.35 | 1.04 | 0.00 | 0.00 | 0.00 | 2.25 | 0.15 | 0.62 | Attention |
+| P06 | moderate | D | 31.60 | 57.52 | 2.97 | 0.00 | 0.00 | 0.00 | 5.69 | 0.43 | 1.80 | MoE GEMM |
+| P07 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — |
+| P07 | moderate | D | 28.67 | 62.18 | 2.42 | 0.00 | 0.00 | 0.00 | 4.86 | 0.37 | 1.49 | MoE GEMM |
+| P08 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — |
+| P08 | moderate | D | 34.59 | 57.06 | 2.22 | 0.00 | 0.00 | 0.00 | 4.44 | 0.34 | 1.34 | MoE GEMM |
+| P09 | saturation | D | 26.23 | 69.03 | 0.36 | 0.00 | 0.00 | 0.00 | 3.43 | 0.00 | 0.95 | MoE GEMM |
+| P09 | moderate | D | 16.19 | 75.00 | 2.14 | 0.00 | 0.00 | 0.00 | 4.83 | 0.35 | 1.49 | MoE GEMM |
+| P10 | saturation | D | 67.80 | 28.90 | 0.76 | 0.00 | 0.00 | 0.00 | 1.87 | 0.10 | 0.57 | Attention |
+| P10 | moderate | D | 30.61 | 54.04 | 2.81 | 0.00 | 0.00 | 0.00 | 9.80 | 0.68 | 2.06 | MoE GEMM |
+| P11 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — |
+| P11 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — |
+
+### Clean CUDA-graph modes at moderate load
+
+| Pattern | FULL | PIECEWISE | NONE/eager |
+|---|---:|---:|---:|
+| P01 | 1.98% | 73.19% | 24.83% |
+| P02 | 83.16% | 13.00% | 3.84% |
+| P03 | 96.37% | 0.00% | 3.63% |
+| P04 | 98.32% | 0.02% | 1.66% |
+| P05 | — | — | — |
+| P06 | 99.10% | 0.05% | 0.85% |
+| P07 | 98.76% | 0.00% | 1.24% |
+| P08 | 98.64% | 0.01% | 1.36% |
+| P09 | 91.79% | 3.69% | 4.52% |
+| P10 | 98.33% | 0.12% | 1.55% |
+| P11 | — | — | — |
+
+P01 is the clear mode outlier: PIECEWISE covers 73.19% of clean moderate
+steps. Full per-mode operator segments and active-step counts remain in the
+machine result; no mode with fewer than eight sampled steps is summarized.
+
+## H1b — rectangular-benchmark miss
+
+**Verdict: PASS.** Five of six evaluable frozen contrasts pass at least one
+waste threshold with a simultaneous positive bound and at least 5% useful-token
+efficiency loss. The two P05 contrasts are `NOT EVALUABLE`; no value is
+imputed. Holm correction retains the original eight planned contrasts per
+metric. Reported simultaneous intervals below are percentage-point effects;
+passing bootstrap p-values are zero at 100,000-resample resolution.
+
+| Frozen contrast | Status | Passing evidence | Useful-token efficiency loss |
+|---|---|---|---:|
+| P05 vs P01 | **NOT EVALUABLE** | P05/C00 missing | — |
+| P05 vs P03 | **NOT EVALUABLE** | P05/C00 missing | — |
+| P06 vs P02 | **PASS** | R64 +23.015 pp, simultaneous 95% [22.345, 23.683] | 11.609% |
+| P06 vs P04 | **PASS** | R64 +35.440 pp [34.826, 36.059] | 22.846% |
+| P09 vs P01 | **PASS** | padding +6.673 pp [5.887, 7.492]; R64 +39.616 pp [39.065, 40.166] | 8.325% |
+| P09 vs P03 | EVALUABLE, NO HIT | padding +8.542 pp and R64 +52.041 pp are material, but association gate fails | **3.840%**, below 5% |
+| P10 vs P03 | **PASS** | padding +5.565 pp [2.010, 10.341]; R64 +44.787 pp [43.635, 45.971] | 44.693% |
+| P10 vs P04 | **PASS** | padding +5.399 pp [1.868, 10.213]; R64 +44.787 pp [43.638, 45.949] | 14.260% |
+
+This confirms the existential H1b claim on completed contrasts. It cannot
+support a null claim about P05 or the missing matrix.
+
+## Waste accounting
+
+The preregistered contrast thresholds are 5 percentage points for padding,
+10 points for graph miss/overflow, 0.15 absolute for R64, 0.10 for mixed-batch
+interference, and 0.15 for MoE layer CV, plus the 5% efficiency/residual
+association gate. Values below are clean C00-moderate results; efficiency is
+scheduled useful tokens per model-step millisecond.
+
+| Pattern | Padding | Graph miss | Overflow | R64 | Mixed interference | Efficiency | Supported/total mixed steps |
+|---|---:|---:|---:|---:|---|---:|---:|
+| P01 | 1.869% | 24.826% | 24.826% | 0.3687 | N/A | 4.9673 | 0/5,646 |
+| P02 | 1.998% | 3.837% | 3.837% | 0.3687 | N/A | 2.6664 | 0/1,536 |
+| P03 | 0.000% | 1.736% | 1.736% | 0.2444 | N/A | 4.7356 | 0/138 |
+| P04 | 0.167% | 1.301% | 1.301% | 0.2444 | N/A | 3.0547 | 0/141 |
+| P05 | — | — | — | — | — | — | — |
+| P06 | 0.276% | 0.830% | 0.830% | 0.5988 | N/A | 2.3568 | 0/152 |
+| P07 | 0.140% | 1.240% | 1.240% | 0.0000 | N/A | 2.7513 | 0/186 |
+| P08 | 0.005% | 1.357% | 1.357% | 0.0000 | N/A | 2.1478 | 0/178 |
+| P09 | **8.542%** | 4.518% | 4.518% | **0.7648** | N/A | 4.5537 | 0/1,054 |
+| P10 | **5.565%** | 0.921% | 0.921% | **0.6923** | N/A | 2.6191 | 0/92 |
+| P11 | — | — | — | — | — | — | — |
+
+No irregular-versus-control graph-miss or overflow contrast reaches the
+positive 10-point H1b threshold. P01 has the largest absolute miss rate but is
+a rectangular control; P09 actually reduces miss versus P01 by 20.308 points.
+
+The registered leave-one-pattern-out robust fits were applied for mixed-batch
+interference, but no pattern retained 30 supported mixed steps inside both
+pure-fit convex supports; every result is N/A rather than extrapolated. No
+separate LOAO operator-ranking procedure was preregistered, so none was added
+post hoc. The required two-window rule is used unchanged, and confirmation-run
+robustness is `NOT EVALUABLE` because all four confirmations are missing.
+
+### Top three waste findings
+
+1. **P10 real trace:** R64 is 44.787 points above both long rectangular
+   controls; padding is 5.565/5.399 points higher; efficiency is 44.693% worse
+   than P03 and 14.260% worse than P04.
+2. **P09 production-shaped mix:** versus P01, R64 rises 39.616 points and
+   padding 6.673 points with 8.325% lower efficiency. Its still-larger P03
+   contrast does not pass because efficiency loss is only 3.840%.
+3. **P06 bimodal long-output burst:** R64 rises 23.015 points versus P02 and
+   35.440 points versus P04, paired with 11.609% and 22.846% efficiency loss.
+
+## Pattern-conditioned operational findings
+
+- **P10/TP2 non-stabilization:** the fresh run completed 17 warm-up requests,
+  but trailing scheduled-token throughput fell 18,022.4 → 16,384.0 → 14,062.6
+  tokens/s. Its 36.76% fitted drift exceeds A-P3-6's 10% limit; same-wave
+  synthetic P11 and P03 completed 512 and 102 warm-up requests and passed their
+  applicable registered gates. The orchestrator accepts this as a
+  pattern-conditioned finding, not an accepted throughput measurement.
+- **Long-context drain:** P10/C01 saturation naturally drained for 288.619
+  seconds. The clean window and accounting were valid; the original 120-second
+  short-pattern watchdog was miscalibrated, while the amended 600-second P10
+  budget correctly retained the run.
+- **Failure boundary:** P01/C01 moderate had 5,828 clean successes and zero
+  clean failures. Two `ServerDisconnectedError` records occurred only in
+  excluded post-profile time; separating clean and auxiliary windows prevented
+  valid data from being discarded.
+- **Layer-2 perturbation:** despite 97%+ kernel classifiability, eight of nine
+  completed moderate patterns failed representativeness or recovery. Together
+  with the Phase-2 51.3% active-window perturbation, this shows that sparse
+  Kineto windows are operationally fragile for these serving patterns.
+
+These incidents consistently show long-context and mixed serving patterns
+breaking assumptions calibrated on shorter rectangular loads: fixed drain
+watchdogs, completion-count warm-up, global failure scope, and profile-window
+representativeness.
+
+## Matrix and GPU accounting
+
+| Item | Final value |
+|---|---:|
+| Planned measured runs | 52 |
+| Accepted measured runs | **40** |
+| Complete cells | **20/24** |
+| Accepted confirmations | 0/4 |
+| Accepted clean failures | 0 |
+| Drain-quarantined accepted runs | 0 |
+| Accepted Layer-2 trace files | 72 |
+| Registered missing trace files | 8 |
+| Other-user GPU processes | 0 |
+| Cumulative GPU use | **14.025875 H20-hours** |
+| Reserved/unused headroom | **1.974125 H20-hours** |
+
+No GPU work was run for A-P3-7 analysis. The reserved headroom remains unused,
+and all eight GPUs were at 0 MiB/0% before and after the CPU-only analysis.
+
+## Data sanity block
+
+| Numeric family | n | finite | missing | min | max | distinct | Check |
+|---|---:|---:|---:|---:|---:|---:|---|
+| Accepted throughput (req/s) | 40 | 40 | 0 | 0.445833 | 43.300000 | 37 | finite, positive, not identical |
+| Token efficiency (tokens/ms) | 40 | 40 | 0 | 2.147827 | 8.295849 | 40 | finite, positive, distinct |
+| Drain duration (s) | 40 | 40 | 0 | 0.305419 | 288.619108 | 40 | non-negative; no quarantine |
+| Clean Layer-1 steps/run | 40 | 40 | 0 | 718 | 20,171 | 40 | sum 352,907; continuous per run |
+| Operator classifiable fraction | 72 | 72 | 0 | 0.970458 | 0.996378 | 72 | all >=0.70 and within [0,1] |
+| Operator-family shares | 576 | 576 | 0 | 0.000000 | 0.873417 | 348 | within [0,1] |
+| Waste ratios | 240 | 240 | 0 | 0.000000 | 1.000000 | 168 | within [0,1] |
+| Waste-contrast effects | 36 | 24 | 12 expected | -0.203082 | 0.520413 | 17 | signed; N/A preserved |
+| Kendall tau-b | 0 | 0 | 0 | — | — | 0 | expected: only one valid pattern |
+| Accepted clean failures | 40 | 40 | 0 | 0 | 0 | 1 expected | sum zero |
+| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | cleanup passed |
+
+Checked invariants: accepted IDs unique; exact 40-run/20-cell coverage; exact
+four missing cells and two missing contrasts; 240-second clean windows; zero
+clean failures; moderate rate within 5%; balanced Layer-1 accounting and zero
+drops; trace parsing and classifiability; ratios in range; exact 72+8 trace
+accounting; clock/load snapshots present; no other-user process; drain
+quarantine below 20%; non-identical pattern results; and exact reproduction of
+the 36.76% A-P3-6 operational result. **No data-sanity red flag remains.** H1a
+is inconclusive because of declared Layer-2 validity limits; H1b passes on five
+complete contrasts; the compound hypothesis is partial and cannot be refuted.

docs/opprof/phase4-optimization-plan.md

@@ -0,0 +1,306 @@
+# OpProf Phase 4 measured optimization plan
+
+Status: **PROPOSAL FROM ACCEPTED PHASE-3 DATA; NO UNMEASURED GAIN CLAIMS**.
+
+Date: 2026-07-12. This plan uses only the accepted Phase-3 40/52-run,
+20/24-cell dataset and the single optional Phase-4 capture-size validation.
+Phase-3 protocol and results are frozen. Bounds below are ceilings in the
+units actually measured; padding or raggedness percentages are not relabeled
+as end-to-end throughput gains.
+
+## Pinned implementation context
+
+- Model/hardware: Qwen3-30B-A3B BF16, one H20, TP1 primary.
+- vLLM source: accepted OpProf tip
+  `23450fb21ac255b0cf710f4ee965ee694921975d` on v0.24.0.
+- vLLM 0.24.0 exposes `--cudagraph-capture-sizes`,
+  `--max-cudagraph-capture-size`, `--max-num-seqs`, and
+  `--max-num-batched-tokens` (`vllm/engine/arg_utils.py:1390-1467`).
+- An explicit capture-size list replaces the inferred list. The default is
+  `[1,2,4]`, multiples of eight below 256, then multiples of sixteen through
+  the maximum, normally 512 (`vllm/config/vllm.py:1669-1792`).
+- Chunked prefill remains enabled. vLLM schedules decode first, then fills the
+  remaining MBT budget with prefill and chunks an over-budget prefill
+  (`docs/configuration/optimization.md:45-59`).
+
+The ranking is by measured opportunity, readiness, and downside together—not
+by a normalized composite score. Items that share the same raggedness bound
+are explicitly non-additive.
+
+## Ranked optimization list
+
+| Rank | Tier | Target and affected regime | Measured bound | Owner | Decision |
+|---:|---|---|---|---|---|
+| 1 | Config now / scheduler backlog | Preserve prefix affinity and prefix caching for P08-like shared-prefix traffic | P08 vs matched P07: **+82.14% saturation req/s**, 80% prefix-hit ratio, **62.29% fewer prefill tokens** | Serving/config owner now; cache-aware dispatch upstream | Deploy behind a workload classifier; do not apply to no-sharing traffic |
+| 2 | Scheduler | Length-aware cohorting/admission for ragged P10/P09/P06 | At most **44.79 pp** R64 contrast and **44.69%** measured efficiency gap on P10; 39.62 pp/8.32% on P09; 35.44 pp/22.85% on P06 | Upstream vLLM scheduler | Highest structural backlog item; preserve fairness and arrival semantics |
+| 3 | Config now | Add exact small CUDAGraph sizes for P09/P10-like moderate decode batches | Distributional bound **4.98 pp P09 / 5.26 pp P10** padding; P09 validation achieved **4.980 pp** | Serving/config owner | Mechanism confirmed; canary only because p95 was +3.01% in one pair |
+| 4 | Config now | Route P06/P10-like pools to MNS=64 | Saturation throughput **+3.37% P06 / +3.70% P10** versus C00 | Serving/config owner | Pattern-specific trial only; same setting was −24.27% on P01 |
+| 5 | Kernel backlog | Ragged-aware MoE GEMM and attention over the measured shape stream | Shares the rank-2 ceiling; no independent additive gain. Descriptive MoE share is 54.04–75.00% on P10/P09/P06 moderate | Engine/kernel colleagues | Optimize exact weighted shapes below; require serving confirmation |
+| 6 | Config guardrail | Keep MBT=8192 for short/high-throughput and long-prefill classes; do not globally set 2048 | Avoided saturation regressions up to **11.64% P03**, 6.53% P01, 5.82% P10 | Serving/config owner | Encode as a policy guardrail, not a positive optimization claim |
+
+## Tier A — configuration-level actions deployable today
+
+### A1. Prefix-affine routing with prefix caching
+
+**Measured regime.** P07 and P08 have the same 1,280-token prompt length,
+512-token output, burst-of-eight arrival, C00 config, and seed. P08 alone uses
+eight 1,024-token shared prefixes plus a unique 256-token suffix. With prefix
+caching already enabled:
+
+| Metric, saturation | P07 no sharing | P08 high sharing | Delta |
+|---|---:|---:|---:|
+| Prefix-query hit ratio | 0.00 | 0.80 | +0.80 |
+| Clean prefill tokens | 1,528,968 | 576,512 | **−62.29%** |
+| Completed throughput | 5.1083 req/s | 9.3042 req/s | **+82.14%** |
+
+**Root mechanism.** The matched cell changes only controlled prefix sharing;
+the observed hit and prefill-token changes directly identify cache reuse. The
+82.14% throughput delta is a point observation and the maximum evidence-backed
+gain for this exact regime, not a fleet-wide forecast.
+
+**Action now.** Keep `--enable-prefix-caching`; hash an application-known
+stable prefix or conversation identity to a replica so related requests do
+not destroy affinity through round-robin load balancing. Apply only when the
+online prefix-query hit ratio resembles P08, not P07.
+
+**Verification.** Interleave affinity ON/OFF on the same replicas and fixed
+request stream. Primary gates are prefix-hit ratio, prefill-token reduction,
+completed req/s, TTFT p95, per-replica queue imbalance, and KV occupancy. A
+throughput gain with queue/fairness or KV-capacity regression does not pass.
+
+### A2. Measured CUDAGraph capture sizes
+
+**Measured regime.** P09 moderate has decode-batch p50/p95/max 4/16/25. P10
+moderate has 1/4/7. Default captures skip sizes 3, 5, 6, 7, and 9, so those
+batches pad upward. Replaying the Phase-3 hit distribution predicts that
+adding exactly `{3,5,6,7,9}` to the complete default list can remove:
+
+- **4.9776 percentage points** of P09's 8.5421% graph-hit padding; and
+- **5.2579 points** of P10's 5.5652% padding.
+
+The list must be `default ∪ {3,5,6,7,9}`; passing only five sizes would replace
+and discard the rest of the default list.
+
+**Closed-loop result.** One P09 moderate ON/OFF pair used fresh servers, the
+same fixed seed and accepted saturation-rate source, 60 excluded warm-up
+seconds, and 240 clean seconds per arm. No Layer-2 profiler ran.
+
+| Metric | ON: exact sizes | OFF: default | ON relative to OFF |
+|---|---:|---:|---:|
+| Graph-hit padding | **3.5659%** | 8.5456% | **−4.9798 pp; −58.27%** |
+| Useful tokens/model-step ms | 4.56222 | 4.55405 | +0.179% |
+| Completed throughput | 4.9583 req/s | 4.9333 req/s | +0.507% |
+| Mean E2E latency | 1.6123 s | 1.6812 s | −4.10% |
+| p95 E2E latency | 3.9930 s | 3.8763 s | **+3.01%** |
+| Clean failures | 0 | 0 | equal |
+
+The observed padding reduction differs from the Phase-3 bound by only 0.0022
+percentage points, confirming the mechanism. It does **not** establish a broad
+performance win: token efficiency and throughput moved less than 1%, p95 moved
+the wrong way, and there is one ordered pair with no CI.
+
+Operational cost also matters: ON captured 56 FULL and 56 PIECEWISE sizes
+versus 51/51 OFF, estimated graph memory increased 0.64→0.68 GiB, and server
+ownership was 76.9 seconds longer. Deploy only as a pattern-specific canary;
+require interleaved replication with a p95 non-regression gate before rollout.
+
+### A3. Pattern-specific MNS pools
+
+The complete saturation comparisons show that `--max-num-seqs 64` is an
+interaction, not a globally better default:
+
+| Pattern | C10 MNS=64 vs C00 | Interpretation |
+|---|---:|---|
+| P01 short/short | **−24.27% req/s** | Reject for dense short traffic |
+| P03 long/short | −1.41% | No measured benefit |
+| P06 bimodal/long burst | **+3.37%** | Candidate pattern pool |
+| P10 real long-context | **+3.70%** | Candidate pattern pool |
+
+The evidence identifies the config×pattern interaction but not a lower-level
+cause. Do not attribute it to a particular kernel or queue effect without a
+new bisection. Verification is five interleaved saturation pairs per intended
+class plus TTFT, queue depth, preemption, KV usage, and exact-work checks.
+
+### A4. MBT policy guardrail
+
+`--max-num-batched-tokens 2048` versus the default 8192 changed saturation
+throughput by −6.53% P01, −11.64% P03, +0.35% P06, and −5.82% P10. The combined
+MNS64/MBT2048 setting was −30.22% on P01. Phase 3 therefore supports retaining
+MBT8192 for these classes and rejects a global MBT2048 rollout. The bound is
+an avoided regression, not new speedup.
+
+## Tier B — upstream scheduler changes
+
+### B1. Length-aware cohorting without starvation
+
+**Measured mechanism.** R64 is the rectangular padding fraction of the exact
+arrival-order prompt stream. It is 0.6923 for P10, 0.7648 for P09, and 0.5988
+for P06. Their passing control contrasts are:
+
+| Irregular pattern | Control | R64 excess | Useful-token efficiency loss |
+|---|---|---:|---:|
+| P10 | P03 | **44.79 pp** | **44.69%** |
+| P10 | P04 | **44.79 pp** | **14.26%** |
+| P09 | P01 | **39.62 pp** | **8.32%** |
+| P06 | P02 | **23.01 pp** | **11.61%** |
+| P06 | P04 | **35.44 pp** | **22.85%** |
+
+These are upper bounds on waste a length-aware path could avoid. R64 is not
+observed GPU time, and efficiency association is not causal. The scheduler
+change should maintain several ready queues by remaining prompt/context band,
+select a less-ragged cohort subject to the existing decode-first token budget,
+and impose a finite age/fairness bound. It must not rewrite request arrivals or
+drop long requests.
+
+**Verification.** Add a runtime per-step raggedness counter rather than using
+manifest R64 as a surrogate. Compare fixed-arrival ON/OFF runs for useful
+tokens/model-step ms, TTFT/E2E p95, queue age, starvation count, preemption,
+KV occupancy, and the full length histogram. The gain cannot exceed the
+corresponding R64/efficiency bounds above, and it is non-additive with a
+ragged-aware kernel.
+
+### B2. Automatic cache-aware dispatch
+
+The upstream form of A1 is a scheduler/replica dispatcher that chooses a live
+prefix-cache owner while respecting load. Its collaboration contract is the
+measured P07/P08 tuple: 1,024 shared + 256 unique prompt tokens, eight prefix
+IDs, burst size eight, output 512, target hit ratio 0.80. The load-balancing
+penalty and lost cache hits must be reported together; a synthetic cache hit
+increase without end-to-end balance is insufficient.
+
+### B3. Histogram-driven capture-list generation
+
+Static A2 proves that Layer-1 can choose useful sizes. An upstream controller
+could select a bounded number of exact sizes from padding contribution
+`count(size) * (next_bucket-size)`, while retaining the default list and a
+memory/startup budget. P09's top five `{3,5,6,7,9}` recovered 4.98 points at a
+0.04-GiB graph-memory and 76.9-second server-lifetime cost in this run. The
+selector must freeze its list before measurement and never continually tune on
+the scored window.
+
+## Tier C — kernel-level backlog and collaboration interface
+
+H1a is inconclusive, so Phase 3 does not prove a universal top operator. The
+available moderate windows are still useful shape inputs: descriptive MoE-GEMM
+shares are 57.52% P06, 75.00% P09, and 54.04% P10; attention shares are 31.60%,
+16.19%, and 30.61%. Only P04's operator windows pass inference gates, where
+attention is 47.88% and MoE GEMM 40.64%. Kernel work must therefore claim
+shape-local improvement, not a resolved global bottleneck.
+
+### Exact shape stream for kernel engineers
+
+`P`, `D`, and `N` below are per-step prefill tokens, decode tokens, and
+scheduled requests. Counts are from clean C00-moderate Layer-1 records.
+Context mix is the fraction of scheduled request-context observations in
+`<=1024 / 1025–8192 / 8193–32768 / >32768` bins.
+
+| Pattern | Model steps: decode / mixed / prefill | N p50 / p95 / max | Dominant exact `(P,D,N): count` | Context mix | Chunk signal |
+|---|---|---|---|---|---|
+| P01 | 5,760: 114 / 5,646 / 0 | 69 / 74 / 77 | `(0,66,66):18`, `(0,67,67):18`; mixed P p50=334, D p50=68 | 100.00 / 0 / 0 / 0% | 6,235 unsplit; sizes 129–512 dominate |
+| P04 | 12,455: 12,291 / 141 / 23 | 8 / 8 / 16 | `(0,8,8):12,129`, `(8191,1,3):23` | 0.03 / 94.62 / 5.35 / 0% | 238/319 chunks >2,048; first/final 122/123 |
+| P06 | 17,464: 17,310 / 152 / 2 | 8 / 16 / 16 | `(0,8,8):13,103`, `(0,16,16):4,098` | 55.11 / 42.93 / 1.96 / 0% | 174/422 >2,048; 136 in 257–512 |
+| P09 | 12,837: 11,783 / 1,054 / 0 | 4 / 16 / 25 | `(0,3,3):3,397`, `(0,2,2):1,549`, `(0,4,4):1,526`, `(0,5,5):1,050` | 51.70 / 48.26 / 0.04 / 0% | 228/1,193 >2,048; 345 in 1,025–2,048 |
+| P10 | 18,130: 17,941 / 92 / 97 | 1 / 4 / 7 | `(0,1,1):13,572`, `(0,2,2):2,675`, `(0,3,3):792`, `(0,4,4):524`, `(8192,0,1):38` | 12.25 / 39.94 / 47.76 / 0.05% | 138/191 >2,048; first/middle/final/unsplit 49/29/49/64 |
+
+The exported kernel-benchmark interface should be a prompt-free weighted table
+with `(P,D,N,context_bin,chunk_class,chunk_size_bin,runtime_mode,count)` plus
+step duration and useful tokens. Use the frozen histogram edges already emitted
+by Layer 1: context `128..131072` powers of two and chunk `16..2048` powers of
+two. Preserve the joint tuples; independent marginal sampling would erase the
+mixed-batch structure.
+
+### Kernel targets and acceptance
+
+1. **Ragged MoE GEMM:** accept variable token counts without padding every
+   expert/layer tile to the largest sequence. Weight microbenchmarks by the P06,
+   P09, and P10 tuples above. The ceiling is the same R64/efficiency opportunity
+   as B1, not an additional gain.
+2. **Attention:** retain P04 `(D,N)=(8,8)` as the valid long rectangular control
+   and test P10's mostly 1–4 decode batches plus 8,192-token chunks. Report
+   useful-token time, workspace, and graph compatibility.
+3. **Serving confirmation:** kernel time must improve on the exact weighted
+   stream, then pass a fixed-arrival serving A/B for throughput, TTFT/p95,
+   memory, and correctness. A rectangular-only kernel win does not close the
+   Phase-3 finding.
+
+`moe_expert_load` was unavailable in Phase 3. No expert-imbalance mechanism or
+gain is claimed; expert-specific packing requires a new low-overhead route
+histogram before implementation.
+
+## Honest limits and Phase-5 measurement requirement
+
+- H1a remains inconclusive: only P04 had two representative/recovered operator
+  windows. Eight other completed moderate patterns failed window validity even
+  though kernel classifiability was 97.05–99.64%.
+- A Phase 5 operator study needs longer, time-stratified samples that reproduce
+  clean scheduled-token, prefill-fraction, decode-batch, and graph-mode
+  distributions, plus a lower-perturbation per-op timer. It must demonstrate
+  overhead before using shares for optimization; the Phase-2 Kineto active
+  window perturbed throughput by 51.3%.
+- Confirmation runs are absent. The MNS and MBT config effects are single-run
+  point estimates and require replication before production decisions.
+- R64 is an offline rectangular-padding upper bound, not measured GPU idle
+  time. H1b's efficiency association does not establish causality.
+- Mixed-batch interference was N/A because no cell retained 30 supported mixed
+  steps inside both leave-one-pattern-out pure-fit supports.
+- Results cover one model, BF16, H20, mostly TP1, and 20/24 cells. P03/C11,
+  P05/C00, P10/C00-TP2, and P11/C00 are absent.
+- The capture validation is one ordered ON/OFF pair. Its padding endpoint is
+  mechanism-valid, but performance deltas have no CI and p95 regressed.
+- Layer 1 did not collect expert-route identities; kernel engineers cannot infer
+  routed-expert imbalance from these artifacts.
+
+## Verification and stop rules for Phase 4 work
+
+Every proposed experiment keeps the original fixed manifest/seed/work, excludes
+warm-up, records Layer-1 accounting, and changes one mechanism. A candidate
+stops on clean failure, footer imbalance/drop, output mismatch, GPU
+contamination, memory regression beyond its declared budget, or violation of
+the 16-H20-hour campaign cap. Throughput, latency, memory, and correctness are
+reported together; no metric shopping or silent pattern substitution is
+allowed.
+
+## GPU accounting
+
+The optional capture pair consumed **0.296389 H20-hours**, taking cumulative
+campaign use from 14.025875 to **14.322265 H20-hours**. Remaining headroom is
+**1.677735 H20-hours**. Both arms returned GPU0 to zero, all eight GPUs were
+0 MiB/0% at final inspection, and no other-user process appeared.
+
+Artifacts are under
+`runs/opprof-phase3/phase4/capture-p09/`; `result.json` SHA-256 is
+`5bb91df28790f6f3c34e4e9ed8e35a1cb8100f93086a4286689d587fd732f2a4`.
+
+## Final ranked one-liners
+
+1. **Prefix affinity (config now):** P08's measured ceiling is **+82.14% req/s** with 62.29% fewer prefill tokens versus matched P07.
+2. **Length-aware scheduler:** raggedness ceiling is **44.79 pp R64 / 44.69% efficiency gap** on P10; smaller confirmed bounds apply to P09/P06.
+3. **Exact capture sizes (config now):** ceiling **5.26 pp P10 / 4.98 pp P09 padding**; P09 validation removed 4.980 pp but did not prove p95 gain.
+4. **MNS64 pattern pools (config now):** measured ceiling **+3.70% req/s P10 / +3.37% P06**, with a −24.27% P01 counterexample.
+5. **Ragged kernels (kernel backlog):** share rank 2's bound; no additive E2E bound is supported while H1a is inconclusive.
+6. **MBT8192 guardrail (config now):** avoids measured regressions up to **11.64%**; MBT2048 has no general positive case.
+
+## Data sanity block
+
+| Numeric family | n | finite | missing | min | max | distinct | Invariant/result |
+|---|---:|---:|---:|---:|---:|---:|---|
+| Ranked items | 6 | 6 | 0 | rank 1 | rank 6 | 6 | Three tiers represented; bounds not summed |
+| Sentinel saturation config deltas | 11 | 11 | 0 | −30.216% | +3.704% | 11 | Both gains and regressions retained |
+| Passing R64 contrast effects | 5 | 5 | 0 | 0.230148 | 0.447872 | 4 | Ratios in [0,1]; duplicate P10 controls expected |
+| Capture-arm padding fraction | 2 | 2 | 0 | 0.035659 | 0.085456 | 2 | Non-negative; ON < OFF |
+| Capture-arm token efficiency | 2 | 2 | 0 | 4.554051 | 4.562222 | 2 | Positive; +0.179% ON |
+| Capture-arm throughput (req/s) | 2 | 2 | 0 | 4.933333 | 4.958333 | 2 | Same offered rate 4.920833 req/s |
+| Capture-arm clean failures | 2 | 2 | 0 | 0 | 0 | 1 expected | Exact 240 s and zero failures |
+| Capture-arm Layer-1 records | 2 | 2 | 0 | 16,491 | 17,579 | 2 | Every footer/sidecar invariant true; zero drops |
+| Optional validation GPU-hours | 1 | 1 | 0 | 0.296389 | 0.296389 | 1 | Positive; cumulative 14.322265 < 16 |
+| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | Cleanup passed |
+
+Checked invariants: Phase-3 metrics remain frozen; every cited number resolves
+to accepted metrics or the checksum-recorded validation; config comparisons
+use saturation rather than normalized moderate throughput; padding/raggedness
+bounds are not presented as throughput; duplicate/non-independent bounds are
+not added; both validation arms use identical work and offered rate; clean
+failures are zero; output work, Layer-1 schema, step continuity, footer/sidecar
+balance, and zero drops pass; ratios lie in their declared domains; all GPU
+memory returned to zero; and cumulative GPU use stays below 16 H20-hours. No
+data-sanity red flag remains.

docs/opprof/phase5-protocol.md

@@ -0,0 +1,640 @@
+# OpProf Phase 5 pre-registered mechanism-decomposition protocol
+
+Status: **ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED**.
+
+Date frozen: 2026-07-12 (Asia/Singapore). This document specifies Phase 5 only.
+It does not authorize a GPU launch, helper implementation, trace transfer, or
+change to the accepted vLLM patch series. Any change to an estimand, request
+set, service order, arrival transform, capture-size set, load, validity gate,
+or decision threshold requires a dated amendment before the affected run.
+
+## Approved dispositions (orchestrator; 2026-07-12)
+
+All five open decisions below are approved before execution. These dispositions
+are normative and close the protocol review gate without changing any frozen
+estimand, command delta, validity threshold, or budget:
+
+1. The recorded-arrival **bridge ledger** is approved. Every machine and human
+   output must state that it decomposes the recorded-arrival P5 gap anchored to
+   P3 controls, not literally P3's already-uniform P10 gap; A3 supplies the
+   explicit bridge back to P3.
+2. The dual P03/P04 control ledgers are approved. Both are always reported and
+   a dominant-mechanism call must pass the frozen rule under both denominators.
+3. A1 is approved exactly as specified: 142 requests, 32-request reorder
+   blocks, 16-request analysis cohorts, frozen bins, and a 64-second fairness
+   cap.
+4. Three P10 replicates per arm, P3 control reuse behind the 3% bridge gate,
+   the conditional control reruns, and optional-tier ordering under the 6.0
+   H20-hour hard cap are approved.
+5. Layer-1-only primary measurement is approved. Routed-expert telemetry is
+   analysis-only, private, optional, and never receives a causal ledger share.
+
+This approval authorizes the later execution turn only when its preflight,
+echo-before-launch, detached-controller, long-context, privacy, accounting,
+cleanup, and budget gates all pass.
+
+## Amendment A-P5-1 — rate-following cold-start gate (orchestrator; 2026-07-12)
+
+The first Phase-5 wave correctly hard-stopped because all four offered-load
+arms failed the inherited A-P3-6 throughput-drift gate: recorded base/A4 drift
+was approximately 216.5%, A1 was 190.6%, and uniform A3 was 13.23%, versus the
+frozen 10% limit. Their 240-second clean windows, output work, offered rates,
+Layer-1 accounting, and drains otherwise passed. The gate was semantically
+wrong for these arms: a rate-following run's scheduled-token throughput follows
+its arrival process by design, so recorded non-stationarity is treatment signal,
+not cold-start contamination. The large recorded-versus-uniform drift gap is
+retained as direct arrival-mechanism evidence. Throughput-drift stationarity
+remains appropriate and unchanged for saturation arms.
+
+For every **rate-following/offered-load** arm, A-P3-6 is replaced by all three
+of the following cold-start-artifact gates:
+
+1. Every logged torch.compile or CUDA-graph capture first-occurrence event must
+   precede the clean boundary. Match server-log event messages containing
+   `torch.compile took`, `Directly load AOT compilation`, `Compiling`, or
+   `Capturing CUDA graphs` (case-insensitive). Timestamped events are compared
+   against `t0_wall_ns + 60 s`. vLLM's capture progress lines have no timestamp;
+   they count as pre-client only when their log-line order precedes the server
+   ready/startup-complete marker, which itself precedes client `t0`. Any matching
+   event after readiness must have a parseable timestamp and precede clean;
+   otherwise the run is invalid.
+2. At least 16 requests must complete successfully in `[0,60 s)`, including at
+   least one request whose recorded `input_tokens >= 8192`.
+3. No first-occurrence capture event may appear inside `[60,300 s)`. Parse the
+   configured startup capture-size set and the completed FULL/PIECEWISE startup
+   capture passes from the server log. From Layer 1, define a captured replay
+   descriptor as `(runtime_mode,bucket_tokens)` for every model-executed
+   `cudagraph.hit=true` step. Every clean descriptor must be covered by the
+   startup-captured mode and bucket set, and no server-log compile/capture event
+   may occur inside clean. Warm and clean descriptor sets are both reported;
+   a descriptor's first **replay** in clean is not mislabeled as a first capture
+   when startup logs prove it was already captured. An uncovered descriptor or
+   clean-window capture/compile event invalidates that run only.
+
+The report records matched server-log events, warm-up completion/long-request
+counts, warm-up and clean descriptor sets, and clean-only descriptors per run.
+Absence of any required log timestamp, Layer-1 interval, or request record is a
+gate failure. The original 10% A-P3-6 drift criterion remains mandatory for
+closed-loop saturation arms. This amendment changes no request set, arrival
+transform, service order, server configuration, clean interval, metric,
+bootstrap, share estimator, dominance rule, control-reuse gate, or GPU budget.
+
+## Goal, system boundary, and success criterion
+
+Phase 3 measured a total useful-token-efficiency gap between irregular patterns
+and rectangular controls but did not causally allocate it. Phase 5 asks how much
+of the P10 gap is recovered when one treatment at a time removes:
+
+1. intra-cohort input-length raggedness;
+2. CUDA-graph decode-batch capture-bucket mismatch;
+3. recorded arrival burstiness; or
+4. usable natural-prefix structure.
+
+The primary system remains Qwen3-30B-A3B BF16, patched vLLM 0.24.0, C00, TP1,
+one H20 per server on dash0. The primary load is the P3 P10 rate
+`lambda = 0.60 * 0.7875 = 0.4725 request/s`; it is held fixed across arms so an
+ablation changes one treatment, not offered demand. The 240-second clean window
+and Layer-1 definition of
+
+```text
+E_token = sum(prefill_tokens + decode_tokens) / sum(model-step duration_ms)
+```
+
+are inherited unchanged. Layer 2 is not needed for the causal ledger and is not
+enabled in primary throughput runs.
+
+Success is a mechanism ledger with an absolute `E_token` for every arm, an
+un-normalized share and bootstrap confidence interval for every mechanism, and
+an explicit residual/interaction line. A null or negative share is a valid
+result. Merely recovering the expected sign is not success.
+
+## Pinned Phase-3 evidence and the arrival-estimand discrepancy
+
+The frozen P3 C00-TP1 moderate values are:
+
+| Cell | `E_token` (tokens/ms) | Role |
+|---|---:|---|
+| P10 | 2.6191132083 | irregular P3 base |
+| P03 | 4.7355997154 | long-input/short-output rectangular control |
+| P04 | 3.0547035940 | long-input/long-output rectangular control |
+
+P10 therefore lost 44.693% versus P03 and 14.260% versus P04. P3 used both
+controls, so Phase 5 reports two parallel ledgers, one per frozen control. It
+never selects the denominator that makes a mechanism look largest. A mechanism
+is called control-robust only when its decision agrees under both ledgers.
+
+There is one blocking provenance discrepancy. The private source contains
+`timestamp` and `source_timestamp`, but P3's materializer discarded them and
+set every P10 row to `arrival=steady`; the P3 client then admitted requests at
+exactly `1/lambda`. Thus P3 has no recorded-arrival burstiness to remove.
+
+The recommended resolution, pending orchestrator approval, is a **P5 bridge
+ledger**: the P5 base replays the same P10 requests at rate-normalized recorded
+timestamps, A3 uniformizes those timestamps, and P03/P04 remain the frozen P3
+controls. A3 also acts as a bridge back to the P3 steady workload. This meets
+the requested arrival ablation but decomposes a recorded-arrival P5 gap, not
+literally the already-uniform P3 gap. The report must show both
+`E_A3 - E_P3_P10` and its CI before relating the P5 ledger to P3.
+
+If the orchestrator rejects this rebase, the P3-exact alternative is mandatory:
+A3's share is `0 / N/A by construction`, and recorded arrival is reported only
+as a stress sensitivity, not as a mechanism share. It is scientifically invalid
+to call the reverse, burstiness-injecting treatment an ablation that removes
+arrival dynamics. No GPU work may begin before this decision is recorded.
+
+## Common request set and exact arrival transforms
+
+The primary request set is the first **142** rows of the frozen 4,011-row P10
+selection in original source order. This is exactly the number of admissions at
+`lambda=0.4725` over `[0,300 s)`: scheduled times are
+`0, 1/lambda, ..., 141/lambda`. All five arms contain the same 142 request IDs,
+prompts, per-request input/output lengths, and aggregate input/output token
+totals. There is no wrap, replacement, cancellation, or resampling.
+
+The Phase-5 materializer must preserve `timestamp` as private metadata. Let
+`z_i` be its stable source-order timestamp for request `i`, and let `N=142`.
+The two arrival vectors are frozen as:
+
+```text
+recorded-scaled: a_i = (z_i-z_0) * ((N-1) / (lambda*(z_(N-1)-z_0)))
+uniformized:     a_i = i / lambda
+```
+
+`z_(N-1)` must exceed `z_0`; timestamps must be finite and nondecreasing.
+Ties remain ties. Both vectors start at zero, end at `141/lambda`, have the same
+mean rate, and use the same request order except in A1. The client schedules
+against `a_i` directly and does not add jitter. This preserves the recorded
+inter-arrival shape while preventing mean-rate differences from masquerading as
+an arrival mechanism.
+
+The protocol requires a small `scripts/opprof_phase5_client.py` extension in a
+later, no-GPU implementation turn. Before execution, CPU-only tests must prove:
+
+- exact 142-row identity and token sums across all manifests;
+- timestamp normalization endpoints and nonnegative gaps;
+- uniform gaps equal `1/0.4725` within 1 microsecond;
+- the A1 fairness bound and deterministic ordering;
+- fixed 60+240-second timing, no wrap, exact output work, and text redaction.
+
+Its reviewed SHA-256 and all manifest SHA-256 values are frozen in the detached
+controller before GPU use.
+
+## Falsifiable mechanism estimands
+
+For arm `m` and rectangular control `c in {P03,P04}`:
+
+```text
+gap_c       = E_control,c - E_base
+delta_m     = E_ablated,m - E_base
+share_m,c   = delta_m / gap_c
+```
+
+The same base and same control are used for all four mechanisms within a
+ledger. No share is clipped to `[0,1]`. Shares may be negative, exceed one, or
+sum above/below one because single-factor interventions can overlap or interact.
+
+The arithmetic residual/interaction line is
+
+```text
+share_residual+interaction,c = 1 - sum_m share_m,c
+```
+
+with a joint bootstrap CI. This is bookkeeping, not proof that the remainder is
+one separable mechanism. It may contain unmeasured mechanisms, non-additivity,
+double-counting, MoE routing, chunked-prefill interference, and measurement
+error. Individual shares are never renormalized to total 100%; the residual is
+never clipped to make the table visually close.
+
+### A1 — length-binned service order
+
+**Hypothesis.** Length-homogeneous local cohorts reduce ragged-attention/SM
+imbalance, so `E_A1 > E_base`. The hypothesis is falsified if the registered
+manipulation check fails or the Holm-corrected efficiency contrast is not
+positive.
+
+Starting from consecutive **32-request reorder blocks** in original P10 order,
+assign each request to the fixed input-length bins
+
+```text
+[0,512], [513,1024], [1025,2048], [2049,4096],
+[4097,8192], [8193,16384], [16385,32768]
+```
+
+and stable-sort each block by `(bin_id, input_tokens, original_index)`. This
+creates two more homogeneous 16-request analysis cohorts per complete reorder
+block. Assign the sorted requests to the block's unchanged recorded-scaled
+arrival slots. If that assignment would delay any request by more than **64
+seconds** relative
+to its original slot, choose the earliest-deadline request first until all
+deadlines are feasible, then resume the length order. Early movement is allowed;
+late movement is capped. Ties are stable and no sorting crosses a 32-request
+block.
+
+On consecutive complete 16-request cohorts of evaluation-slice service order,
+define `R16 = 1 - sum(L_i) / sum(16*max_cohort(L_i))`; the incomplete final
+cohort is excluded and reported. Frozen pre-run values are base `R16=0.641744`
+and sorted `R16=0.473409`, a 0.168334 absolute (26.23% relative) reduction;
+plain sorting's maximum added delay is 62.744 seconds. The manipulation passes
+only if regenerated values match these within `1e-6`, `R16` falls by at least
+20% relative and 0.15 absolute, and no request violates the 64-second delay
+bound.
+Arrival-slot timestamps, request/content multiset, per-request output lengths,
+total tokens, server config, and prefix-caching setting are identical to base.
+
+A1 estimates the total effect of changing cohort composition. It does not
+claim to isolate a particular attention kernel: service order can mediate
+decode-batch composition, chunked-prefill mixing, cache locality, and
+content-bound MoE routing. If the prefix-query hit ratio changes by more than
+one percentage point, or the normalized inter-arrival vector changes at all,
+the arm is labeled confounded and has no publishable raggedness share.
+
+### A2 — measured decode-B capture sizes (config-tier deliverable)
+
+**Hypothesis.** Exact capture sizes for P10's observed pure-decode batch support
+remove decode-bucket slack, so pure-decode padding falls and `E_A2 > E_base`.
+No recovery falsifies the efficiency hypothesis; failure to remove the targeted
+padding invalidates the ablation rather than supporting a null mechanism.
+
+The P3 P10/C00/rho=0.60 clean Layer-1 stream has SHA-256
+`51ad4be12178da91d2af484d0946a2274afd3bcbbee33f37940cfe0ff2ea7fa7`.
+Its 17,941 pure-decode steps have the exact decode-B histogram:
+
+| B | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+|---:|---:|---:|---:|---:|---:|---:|---:|
+| Steps | 13,572 | 2,675 | 792 | 524 | 161 | 202 | 15 |
+
+Defaults already contain 1, 2, 4, and 8. A2 therefore adds exactly
+`{3,5,6,7}` and freezes the complete server list as:
+
+```text
+1 2 3 4 5 6 7 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128
+136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256
+272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512
+```
+
+This covers 100% of P3's observed P10 pure-decode B support. The A2
+manipulation requires at least 99% support coverage in the new clean runs and a
+90% reduction in pure-decode padding tokens versus base. It targets decode
+capture-bucket mismatch; it does not remove prefill-token padding, eager
+overflow, graph launch overhead, or length raggedness itself. Capture startup
+time and memory are reported as config cost, not treated as free.
+
+### A3 — recorded arrival to uniform arrival
+
+**Hypothesis.** Uniformizing the same rate and request order reduces burst-driven
+decode-batch/queue variance, so `E_A3 > E_base`. The mechanism is falsified if
+the clean 5-second decode-B CV and waiting-queue CV do not fall, or if the
+Holm-corrected efficiency contrast is not positive.
+
+A3 changes only `recorded-scaled` arrival slots to `i/lambda`. Content, order,
+input/output lengths, aggregate tokens, server config, prefix caching, and
+capture sizes are unchanged. It estimates the total service effect of arrival
+shape, including its legitimate downstream changes to batching and queueing.
+It does not isolate a scheduler instruction cost. Under the P3-exact fallback,
+this arm is the base-equivalent bridge and its arrival share is N/A as described
+above.
+
+### A4 — natural prefix caching disabled
+
+**Hypothesis.** Direction is deliberately two-sided. Natural P10 reuse may
+increase `E_token` through KV reuse, in which case disabling it gives a negative
+share; alternatively, low-value/fragmented cache structure may impose overhead
+or alter batching, giving a positive share. Either direction is publishable.
+
+A4 omits only `--enable-prefix-caching`. Prompts, natural repeated-prefix
+structure, recorded arrival slots, request order, token totals, scheduler limits,
+and capture sizes remain identical to base. Required manipulation checks are
+zero local prefix cache hits/queries in the disabled arm and unchanged prompt
+hashes. This estimates the contribution of exploiting C structure, not the
+intrinsic content similarity of the prompts.
+
+### Mechanisms without a clean ablation
+
+MoE routing skew cannot be removed while preserving P10 content and model
+semantics: changing tokens, router weights, top-k, or expert placement changes
+more than routing skew. It receives no causal share.
+
+If budget remains, one **analysis-only**, separately started P10 sample may add
+`--enable-return-routed-experts`. It is excluded from every `E_token` clean
+window and ledger numerator. Offline analysis reports per-layer expert-count
+entropy, Gini coefficient, coefficient of variation, max/mean load, and their
+association with same-step token-normalized duration. The arrays remain private.
+This telemetry is sampled and perturbing, can consume a large scheduler-side
+buffer, and provides correlation rather than causal attribution; it can only
+help interpret the residual/interaction line. Phase 3's MoE layer-duration CV
+was N/A, so it cannot substitute for these routed-expert counts.
+
+## Exact commands and config deltas
+
+The following interface is normative for the later implementation. Variables:
+
+```bash
+P5C='python scripts/opprof_phase5_client.py'
+PRIVATE=/home/admin/cpfs/wjh/opprof-phase5-private/manifests
+P3PRIVATE=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl
+P3SOURCE=/home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/chat_w20260311_1000.jsonl
+MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
+RATE=0.4725
+```
+
+Materialize the five private manifests without printing prompt text:
+
+```bash
+$P5C transform --in "$P3PRIVATE" --take-first 142 \
+  --timestamp-source "$P3SOURCE" --join-key source_index \
+  --timestamp-field timestamp --arrival recorded-scaled --target-rate "$RATE" \
+  --service-order original --out "$PRIVATE/P10-base.jsonl"
+$P5C transform --in "$P3PRIVATE" --take-first 142 \
+  --timestamp-source "$P3SOURCE" --join-key source_index \
+  --timestamp-field timestamp --arrival recorded-scaled --target-rate "$RATE" \
+  --service-order length-binned --reorder-block-size 32 \
+  --analysis-cohort-size 16 \
+  --length-bin-edges 512,1024,2048,4096,8192,16384,32768 \
+  --max-added-delay-seconds 64 --out "$PRIVATE/P10-A1.jsonl"
+$P5C transform --in "$P3PRIVATE" --take-first 142 \
+  --timestamp-source "$P3SOURCE" --join-key source_index \
+  --timestamp-field timestamp --arrival uniform --target-rate "$RATE" \
+  --service-order original --out "$PRIVATE/P10-A3.jsonl"
+ln -s P10-base.jsonl "$PRIVATE/P10-A2.jsonl"
+ln -s P10-base.jsonl "$PRIVATE/P10-A4.jsonl"
+```
+
+The symlinks make unchanged request bytes explicit; the controller hashes the
+resolved content and requires A2/A4 hashes to equal base. A1/A3 require equal
+sorted request-ID sets and equal input/output token sums.
+
+Common server command (`ARM` is `base`, `A1`, `A2`, `A3`, or `A4`):
+
+```bash
+taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU" \
+  VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
+  vllm serve "$MODEL" --host 127.0.0.1 --port "$PORT" \
+  --tensor-parallel-size 1 --enable-chunked-prefill \
+  --enable-prefix-caching --shutdown-timeout 600
+```
+
+- A2 adds
+  `--cudagraph-capture-sizes 1 2 3 4 5 6 7 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512`.
+- A4 removes `--enable-prefix-caching`.
+- Base, A1, and A3 have no server delta.
+
+Every moderate client command is:
+
+```bash
+taskset -c "$CPUSET" $P5C run \
+  --manifest "$PRIVATE/P10-$ARM.jsonl" \
+  --base-url "http://127.0.0.1:$PORT" --model "$MODEL" \
+  --load-point moderate --fixed-request-rate "$RATE" \
+  --max-concurrency 256 --ignore-eos --temperature 0 \
+  --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 \
+  --post-clean-seconds 0 --drain-timeout-seconds 600 \
+  --workload-seed 20260712 --server-seed 20260712 \
+  --result-dir "$RUN_DIR/client"
+```
+
+There are no profiler endpoint calls. Exact commands, effective config, startup
+capture list, compile-cache key, hashes, clocks, host load, and GPU process list
+are recorded per run.
+
+## Execution and validity discipline
+
+### Placement, order, and detached ownership
+
+A-P3-1 already rejected eight-way and authorized four-way placement. Phase 5
+therefore uses at most four simultaneous TP1 servers, GPU0--GPU3, with the
+frozen disjoint CPU masks `0-19`, `20-39`, `40-59`, and `60-79`. Topology must
+still match P3. A changed host topology, source/patch/runtime hash, or clock
+policy invalidates reuse of the placement gate and stops for review.
+
+Three independent replicates are run for each of the five primary arms (15
+measured runs). Sort all `(replicate,arm)` assignments by SHA-256 of
+`"20260715:<replicate>:<arm>"`, pack them into waves of 4, 4, 4, and 3, and
+rotate GPU assignments. The final three-target wave uses the frozen P06/C00
+saturation background in the fourth slot so measurements retain the validated
+four-way host regime. No wave contains two replicates of the same arm; if the
+hash order would do so, stable-swap the later item with the first legal item and
+record the resolved order before launch. Background data are not analyzed.
+
+As required by A-P3-2, a dash0-resident `setsid`/`nohup` controller owns every
+process, has `--resume`, atomically records state, and skips only fully validated
+runs with matching hashes. Interactive SSH never owns a server. Before each
+wave it logs one echo line with resolved arms, GPUs/CPUs, manifests, rate,
+paths, reserved H20-hours, expected duration, and disk headroom.
+
+Use three unmeasured 60-second burn-ins before measured arms: common C00,
+A2 capture-list C00, and prefix-cache-off C00. They warm compile/AOT artifacts
+but are not measurements. Per-run-unique OpProf directories must remain ignored
+by vLLM compile factors, preserving the accepted Phase-2 fix.
+
+### Long-context-safe warm-up and drain gates
+
+These gates are active from the first run; no short-context default is tried
+first.
+
+- Warm-up is exactly 60 seconds and excluded. P10 passes with at least 32
+  successful warm-up completions, **or** at least 16 completions plus the exact
+  A-P3-6 stabilization test: model-executed steps in `[45,50)`, `[50,55)`, and
+  `[55,60)`; at least 16 steps/bin; positive scheduled-token rates `R_j`; and
+  OLS drift `abs(slope)*15/mean(R_j) <= 0.10`. Missing bins, discontinuity, or
+  accounting failure invalidates the run.
+- Drain timeout is 600 seconds. A timeout marks the run drain-quarantined but
+  does not invalidate an otherwise valid clean window. More than 20% of primary
+  runs quarantined stops Phase 5 for review.
+- The clean window is exactly three contiguous 80-second segments. Admission
+  and completion accounting follows P3. Offered rate must be within 5% of
+  0.4725 request/s, with zero clean failures and exact output tokens.
+- Layer-1 JSONL/footer/sidecar accounting, zero drops, contiguous steps, no
+  profile leakage, clock/load capture, other-user-process absence, and final
+  zero GPU memory are hard gates inherited from P3.
+
+No semantic failure is retried with altered parameters. One exact retry is
+allowed for an infrastructure artifact failure, and both attempts remain in
+the operational findings.
+
+## Control reuse, secondary scope, and budget
+
+P03/P04 P3 controls may be reused because they already use the same model,
+patch, C00-TP1 config, normalized load, 240-second clean metric, placement
+regime, and Layer-1 schema. Reuse avoids spending GPU time without changing the
+denominator.
+
+Reuse is valid only if source, patch, model, runtime, clocks, and placement
+hashes match and fresh A3-uniformized P10 differs from frozen P3 P10 by at most
+3% in `E_token` with a bootstrap CI containing zero difference. If this bridge
+gate fails, temporal/runtime drift is plausible: rerun P03 and P04 under their
+exact P3 manifests and commands, three replicates each, before computing any
+share. A failed bridge never licenses rescaling old controls.
+
+After the complete P10 ledger, optional work is ordered as follows and starts
+only if the controller's conservative reservation remains below the 6.0
+H20-hour hard cap:
+
+1. one saturation run per P10 arm, using the full 4,011-row manifest, for
+   descriptive mechanism persistence only;
+2. one five-arm moderate ledger each for P09 and P06, reported as exploratory
+   within-run-bootstrap evidence, not equal replication to P10; and
+3. one routed-expert analysis-only sample.
+
+For secondary A2, the precomputed P3 pure-decode support additions are P09
+`{3,5,6,7,9,10,11,12,13,14,15,17,18,19,20,21,22,23,25}` and P06
+`{3,9,10,11,12,13,14,15}`; default sizes are retained. P09 A3 is a no-change
+steady negative control, while P06 A3 changes its registered burst-8 arrivals
+to uniform spacing at the same mean rate. P09/P06 conclusions are always
+labeled secondary.
+
+Expected accounting, including server-owned startup/shutdown time:
+
+| Tier | New measured runs | Expected H20-hours | Expected 4-way wall |
+|---|---:|---:|---:|
+| P10: 5 arms x 3 replicates | 15 | 1.6-1.9 | 35-55 min |
+| Three compile burn-ins | 0 | 0.1-0.2 | 3-6 min |
+| Conditional P03/P04 reruns | 6 | 0.6-0.8 | 15-25 min |
+| Optional P10 saturation | 5 | 0.5-0.8 | 15-25 min |
+| Optional P09/P06, 5 arms each | 10 | 0.9-1.2 | 20-35 min |
+| Optional routed-expert sample | 1 analysis-only | 0.1-0.2 | 5-10 min |
+| **Maximum planned** | **36 ledger + 1 analysis-only** | **3.8-5.1 expected** | **about 1.5-2.5 h** |
+
+The hard cap is **6.0 H20-hours new Phase-5 spend**, not a target. Actual time
+while a server owns GPU memory is charged. Optional tiers are skipped rather
+than overrunning the cap. The 600-second drain allowance is a watchdog, not an
+assumption in the expected estimate; repeated long drains consume the optional
+budget first.
+
+With no Kineto traces, primary P10 artifacts are estimated at 0.4-0.6 GB and
+all planned public artifacts below 1.5 GB. Stop if public Phase-5 artifacts
+exceed 3 GB or CPFS free space falls below 100 GB. Prompt-bearing manifests and
+routed-expert arrays remain in mode-0700 private storage and are not counted as
+public deliverables.
+
+## Statistical analysis and decision rules
+
+Use 5-second moving-block bootstrap over clean time, 100,000 resamples, seed
+20260716. For P10, first resample the three run IDs, then resample 5-second
+blocks within each selected run; arms and reused P3 controls are resampled
+independently. Every `E`, `delta`, share, share sum, and residual/interaction
+gets a percentile 95% CI. Absolute `E` and delta accompany every ratio.
+
+Bootstrap ratio draws are never deleted because their denominator is
+inconvenient. If either control gap has a point estimate `<=0`, its CI includes
+zero, or more than 5% of bootstrap denominator draws are `<=0`, that control's
+ledger is **INCONCLUSIVE (unstable denominator)**.
+
+The confirmatory family is the four two-sided tests of `E_ablated-E_base=0` on
+P10. Apply Holm correction at family-wise alpha 0.05 across A1--A4. The same
+corrected delta test serves both control ledgers; duplicating denominators does
+not create eight tests. A1--A3 only support the expected recovery claim when
+their corrected contrast is positive. A4 may be significant in either
+direction. Manipulation-check failure makes the corresponding share N/A even
+if efficiency changes.
+
+A mechanism is **dominant** only if, under both P03 and P04 ledgers:
+
+- point `share >= 0.30`;
+- the 95% share CI excludes 0.15 on the high side (`CI_low > 0.15`); and
+- its Holm-corrected efficiency contrast is significant in the expected
+  direction (two-sided for A4).
+
+Meeting the rule under only one control is reported as **control-sensitive**,
+not dominant.
+
+The primary ledger is **publishable** when all five P10 arms have three valid
+replicates, both control denominators are stable, all four manipulation checks
+are evaluable, every share/residual has a finite CI, the fresh A3/P3 bridge is
+reported, and no privacy/data-sanity red flag exists. It may be publishable
+with no dominant mechanism. It is **inconclusive** if any primary arm is
+missing, a denominator is unstable, the arrival-rebase decision is unresolved,
+or two or more share CIs have width greater than 0.50. One failed mechanism
+manipulation yields a publishable partial ledger only if that line is explicitly
+N/A and the headline claim excludes it.
+
+No additive causal claim is made from `sum(share_m)`. Pairwise and higher-order
+interactions are not identified by this one-factor-at-a-time matrix; a combined
+all-off arm would be a new experiment requiring amendment, not an improvised
+way to close the ledger.
+
+## Deliverable and artifact contract
+
+Execution, if later approved, must produce:
+
+- `docs/opprof/phase5-results.md`;
+- `runs/opprof-phase5/phase5/metrics.json` with schema, arm/run values,
+  bootstrap draws' seed and summary, Holm results, dual-control ledgers,
+  residual/interaction, gates, and GPU accounting;
+- per-run exact commands, environment/provenance, client records, Layer-1
+  stream/footer/sidecar, monitor data, and machine-readable sanity JSON; and
+- an **Operational findings** section covering warm-up stabilization, drains,
+  compile/capture startup cost, contamination, retries, and any mismatch between
+  expected and realized mechanism removal.
+
+Every raw-run summary, aggregate table, and final metrics file ends with the P3
+sanity schema: `n`, finite/missing, min, max, distinct count, and applicable
+invariants. Red flags are reported first and stop inferential analysis. Public
+artifacts may contain request IDs, hashes, lengths, counts, and timing only;
+prompt, messages, content, generated text, source substrings, and routed-expert
+arrays are forbidden.
+
+## Final ablation table
+
+| Mechanism | What changes | What is preserved |
+|---|---|---|
+| Base | Recorded P10 timestamps are rate-normalized and replayed in source order | Frozen 142 requests, content/tokens, C00-TP1, prefix cache on, default capture list, `lambda=0.4725` |
+| A1: length raggedness | Stable length-bin sort within 32-request reorder blocks into 16-request cohorts, 64-second late cap | Requests/content/token totals, arrival-slot vector, output lengths, server config, prefix setting |
+| A2: capture mismatch | Add exact P10 decode-B sizes `{3,5,6,7}` to the full default capture list | Manifest/order/arrivals/content/tokens, scheduler limits, prefix setting |
+| A3: arrival dynamics | Recorded-scaled slots become uniform `i/0.4725` slots | Request order/content/tokens, mean rate, server/capture/prefix config |
+| A4: prefix structure | Remove `--enable-prefix-caching` | Natural prompt structure, request order/arrivals/content/tokens, capture/scheduler config |
+| Residual + interactions | No extra run; joint arithmetic remainder `1-sum(shares)` | Raw unnormalized mechanism shares; no clipping or forced 100% allocation |
+
+## Final run count and GPU estimate
+
+Primary commitment: **15 new measured P10 runs plus three unmeasured burn-ins,
+1.7-2.1 expected H20-hours, 35-60 minutes four-way wall, and 0.4-0.6 GB public
+disk**. Conditional control reruns add 6 runs; all optional tiers bring the
+maximum to **36 ledger runs plus one analysis-only run, 3.8-5.1 expected
+H20-hours, about 1.5-2.5 hours wall, and less than 1.5 GB public disk**. New
+Phase-5 GPU use stops at **6.0 H20-hours** under all circumstances.
+
+## Resolved decisions from the orchestrator
+
+1. **Approved:** use the recommended recorded-arrival P5 bridge ledger, with
+   its explicit limitation that it is not a literal decomposition of P3's
+   already-uniform P10 gap; otherwise select the P3-exact A3=N/A fallback.
+2. **Approved:** dual P03/P04 control ledgers and the requirement that “dominant” hold
+   under both, rather than selecting one P3 denominator.
+3. **Approved:** the 142-request slice, 32-request reorder blocks, 16-request analysis
+   cohorts, fixed bins, and 64-second fairness cap as A1's isolation/latency
+   tradeoff.
+4. **Approved:** three P10 replicates, reuse of P3 controls behind the 3% bridge gate,
+   and the optional-tier order within the 6.0-H20-hour cap.
+5. **Approved:** Layer-1-only primary runs and treating routed-expert telemetry as
+   private analysis-only evidence with no causal share.
+
+## Protocol sanity block
+
+| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---|
+| P3 control/base `E_token` | 3 | 2.619113 | 4.735600 | 3 | Finite, positive, not identical |
+| P3 P10 control gaps | 2 | 0.435590 | 2.116487 | 2 | Positive; dual denominators retained |
+| P10 request rows/arm | 5 arms | 142 | 142 | 1 expected | Same IDs and input/output token sums required |
+| Selected source timestamps (s) | 142 | 0.014 | 21.445 | 142 | Finite, nondecreasing; gap min/max 0.002/0.851 s, 127 distinct gaps |
+| Primary offered rate (req/s) | 5 arms | 0.4725 | 0.4725 | 1 expected | Positive; achieved rate must be within 5% |
+| Warm-up / clean / drain gates (s) | 3 values | 60 | 600 | 3 | Clean is exactly `3*80=240`; long-context drain is 600 |
+| A1 input-length bins | 7 | 0 | 32768 | 7 intervals | Ordered, contiguous, cover frozen P10 range |
+| A1 frozen `R16` values | 2 | 0.473409 | 0.641744 | 2 | Sorted is lower by 0.168334 absolute / 26.23% relative; not identical |
+| A1 added-delay values (s) | 142 | 0 | 62.744 | >1 expected | Non-negative and all below 64-second cap |
+| P3 P10 pure-decode steps | 17,941 | B=1 | B=7 | 7 B values | Counts sum to 17,941; non-negative; stream SHA pinned |
+| P10 A2 added capture sizes | 4 | 3 | 7 | 4 | Exactly missing observed support `{3,5,6,7}`; 100% P3 support covered |
+| Primary measured runs | 15 | 3/arm | 3/arm | 1 expected | `5 arms * 3 replicates`; no arm omitted |
+| Maximum GPU analysis/ledger runs | 1 plan | 37 | 37 | 1 expected | `15+6+5+10+1=37`; burn-ins excluded |
+| Expected H20-hours | 1 plan | 3.8 | 5.1 | 2 bounds | Finite, non-negative, below hard cap 6.0 |
+| Share domain | 5 ledger lines/control | Unbounded | Unbounded | N/A | No clipping; ratios may be negative or >1 |
+| Bootstrap resamples | 1 setting | 100,000 | 100,000 | 1 expected | Seed 20260716; no denominator-draw deletion |
+| Phase-5 GPU runs in this protocol turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied |
+
+Checked invariants: `0.60*0.7875=0.4725`; P3 control gaps are positive;
+the seven decode-B counts sum to 17,941; default plus `{3,5,6,7}` covers all
+observed P10 pure-decode B values; `5*3=15` primary runs; maximum planned count
+is `15+6+5+10+1=37`; expected GPU use remains below the 6.0-hour hard cap;
+ratios are not constrained to `[0,1]`; expected constants are labeled; and no
+GPU command, helper change, manifest transform, or experiment was executed in
+this protocol-only turn. The unresolved P3-arrival/P5-arrival estimand mismatch
+is reported first as a blocking decision rather than hidden in the ledger.

docs/opprof/phase5-results.md

@@ -0,0 +1,223 @@
+# OpProf Phase 5 dash0 results
+
+Status: **FINAL — INCONCLUSIVE MECHANISM LEDGER; NO DOMINANCE CALL**.
+
+Date: 2026-07-12. All 15 registered P10 primary runs completed with three
+replicates per arm and passed A-P5-1. The fresh A3 bridge passed, so the frozen
+P3 P03/P04 controls were reused. The ledger is nevertheless inconclusive:
+P04's reused-control denominator is unstable, and A2, A3, and A4 fail their
+predeclared manipulation checks. The protocol therefore permits one official
+share (A1 under P03), requires the other lines to be N/A, and forbids a
+dual-control dominance conclusion.
+
+The machine result is `runs/opprof-phase5/phase5/metrics.json` (SHA-256
+`fe16934866806ac542f7974cac061beeb1f1c58f5bfb36698efd8aeaa99b8cb7`).
+The final analyzer SHA-256 is
+`8b0ff8658614227ee00cf8d10c82c4b5edf6f6210db172fd6efc9cc149e51e10`;
+its no-GPU tools and analysis tests pass.
+
+This remains explicitly a **recorded-arrival P5 bridge ledger anchored to P3
+controls**, not a literal decomposition of P3's already-uniform P10 gap.
+
+## Data-sanity result first
+
+**RED FLAG: the P04 control denominator is unstable.** P5 recorded-arrival
+base has `E_token=3.013752`, while reused P3 P04 has `E_token=3.054704`, leaving
+only a `0.040952` gap with bootstrap 95% CI `[-0.649096, 0.752594]`; 45.353% of
+denominator draws are nonpositive. This violates every registered stability
+form: the CI contains zero and the nonpositive fraction exceeds 5%.
+
+P03 remains stable: its gap is `1.721848`, CI `[1.459092, 1.988772]`, with zero
+nonpositive draws. All execution-validity checks pass: 15/15 primary runs,
+three replicates per arm, exact 240-second clean windows, zero clean failures,
+offered rates within 5%, continuous/balanced Layer-1 accounting, identical
+request IDs and token sums, all A-P5-1 cold-start gates, no drain quarantine,
+and GPU spend below the cap. The red flag is reported before the ledger and no
+conclusion is built on P04's raw ratio draws.
+
+## Bridge and absolute efficiency
+
+A3 reproduces the P3 uniform-arrival P10 base: `delta_E=+0.007041`, 95% CI
+`[-0.329799, 0.354859]`, or 0.269% absolute relative difference. The CI contains
+zero and the point difference is below 3%, so the bridge passes and P3 control
+reuse is valid under the frozen rule. Conditional control reruns were not
+launched.
+
+| Arm | `E_token` | Bootstrap 95% CI | Delta from P5 base |
+|---|---:|---:|---:|
+| Base, recorded | 3.013752 | [2.754462, 3.267964] | — |
+| A1 length-binned | 3.078999 | [2.801659, 3.349307] | +0.065247 |
+| A2 capture sizes | 3.062128 | [2.802812, 3.314276] | +0.048376 |
+| A3 uniform arrival | 2.626154 | [2.451852, 2.792684] | -0.387598 |
+| A4 prefix cache off | 3.020959 | [2.760494, 3.274105] | +0.007207 |
+
+## Manipulation checks and Holm family
+
+| Arm | Manipulation result | Efficiency delta 95% CI | Raw p | Holm p |
+|---|---|---:|---:|---:|
+| A1 | **PASS:** R16 0.641744 -> 0.473409; max delay 62.743 s; prefix-hit-ratio delta -0.296 pp | [-0.309461, 0.442565] | 0.73108 | 1.00000 |
+| A2 | **FAIL:** support coverage 98.879% <99%; padding reduction 82.541% <90% | [-0.315361, 0.411446] | 0.79022 | 1.00000 |
+| A3 | **FAIL:** decode-B CV falls 0.8706 -> 0.6551, but waiting CV rises 6.0145 -> 6.8557 | [-0.693444, -0.078141] | 0.01384 | 0.05536 |
+| A4 | **FAIL:** disabled arm still reports 3,093,324 local prefix queries and 52,896 hits, not zero | [-0.357637, 0.369309] | 0.97230 | 1.00000 |
+
+Per protocol, a failed manipulation makes that mechanism's official share N/A
+even if its raw efficiency contrast is nonzero. A3's uncorrected contrast is
+negative, and it also misses family-wise significance after Holm correction.
+
+## A-P5-1 arrival evidence
+
+The four originally invalidated arms supplied the evidence for A-P5-1. Under
+the superseded throughput-drift calculation, recorded base/A1/A4 span
+190.55%-216.50% normalized drift, while uniform A3 is 13.23%. This gap is
+retained as direct evidence that the old gate measured arrival shape in a
+rate-following experiment; it is not converted into an efficiency share.
+
+All 15 replacement/continued primary runs pass the amended cold-start gate:
+25-28 warm-up completions, 12-16 completions with input at least 8192 tokens,
+all compile/capture first occurrences before clean, zero clean-window
+first-capture events, and zero uncovered Layer-1 graph descriptors. The 10%
+drift gate was retained unchanged for the P06 saturation background arm.
+
+## Operational findings
+
+- The detached controller completed three cache burn-ins, four four-way waves,
+  all drains, validation, budget accounting, and cleanup. The four invalidated
+  first-wave arms were rerun exactly; none of their superseded measurements
+  enters the ledger.
+- The first exact rerun produced complete clean artifacts but hit a post-hoc
+  validator `NameError` from a missing `math` import. Immutable artifacts were
+  CPU-re-adjudicated under A-P5-1, the import was repaired, and no third GPU run
+  was charged.
+- CPU analysis repairs admitted legitimate idle fixed-time blocks as `[0,0]`,
+  supported the older P3 control schema, converted one NumPy boolean at the
+  JSON boundary, and explicitly separated diagnostic from reportable shares.
+  These changes alter no GPU data, estimator, seed, threshold, or resample
+  count.
+- P10 drains range from 0.676 to 6.004 seconds. The separate P06 saturation
+  background run drained naturally in 87.903 seconds, below the 600-second
+  gate, with 564/564 clean completions and zero failures.
+- Public Phase-5 artifacts occupy 286,205,365 bytes (304 MiB), below the 3 GB
+  stop threshold; CPFS retained 1,286 GB free.
+- MoE routing skew remains content-bound and receives no causal share. The
+  optional routed-expert analysis-only run was not required for this primary
+  ledger and was not launched.
+
+## Launch echo
+
+```text
+LAUNCH_ECHO utc=2026-07-12T12:07:54Z host=dash0 gpus=0-3 cpus=0-79 source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0@4b253fd manifests=/home/admin/cpfs/wjh/opprof-phase5-private/manifests outputs=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5 runs=3burnin+15primary+1background rate=0.4725 warmup=60s clean=240s drain=600s est_wall=35-60min est_gpu=1.7-2.1_H20h hard_cap=6.0_H20h conditional_controls=6_if_bridge_fails
+```
+
+Resume after A-P5-1 reused the completed burn-ins and compile/cudagraph caches.
+The first launch through final controller completion took approximately 57.9
+minutes wall time, including the adjudication/resume interval.
+
+## Run completion stats
+
+| Item | Count/result |
+|---|---:|
+| Valid primary P10 runs | **15/15** |
+| Replicates | **3 per arm** |
+| Valid P06 background runs | 1 |
+| Completed burn-ins | 3 |
+| Superseded pre-amendment arms | 4 |
+| Conditional fresh controls | 0; bridge passed, P3 reused |
+| Optional secondary/routed-expert runs | 0 |
+| A-P5-1 primary passes | 15/15 |
+| Clean failures | 0 |
+| Drain quarantines | 0 |
+
+## Mechanism ledger
+
+Official shares follow the frozen N/A rules. Values in parentheses are raw
+diagnostic ratios retained for audit, not causal/reportable shares.
+
+| Mechanism | P03 share (95% CI) | P04 share (95% CI) | Disposition |
+|---|---|---|---|
+| A1 length raggedness | **0.0379 [-0.2011, 0.2344]** | N/A; unstable denominator (diagnostic 1.5933 [-6.8925, 7.0636]) | Manipulation passes; no significant recovery |
+| A2 capture mismatch | N/A; manipulation failed (diagnostic 0.0281 [-0.2049, 0.2170]) | N/A; unstable denominator (diagnostic 1.1813 [-6.2871, 6.4924]) | Config did not meet 99%/90% removal gates |
+| A3 arrival dynamics | N/A; manipulation failed (diagnostic -0.2251 [-0.4625, -0.0403]) | N/A; unstable denominator (diagnostic -9.4647 [-17.2450, 17.4102]) | Queue-variance gate failed; Holm p=0.05536 |
+| A4 prefix structure | N/A; manipulation failed (diagnostic 0.0042 [-0.2340, 0.1948]) | N/A; unstable denominator (diagnostic 0.1760 [-6.2590, 6.4925]) | Zero-query/hit gate failed |
+
+Shares are neither clipped nor renormalized. P04's extreme raw ratios are the
+registered consequence of dividing by a near-zero, sign-unstable denominator,
+not evidence of large mechanisms.
+
+## Dominance verdicts
+
+| Mechanism | P03 rule | P04 rule | Dual-control verdict |
+|---|---|---|---|
+| A1 | Fails share/CI/Holm thresholds | Not evaluable | **NOT EVALUABLE** |
+| A2 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
+| A3 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
+| A4 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
+
+No mechanism may be called dominant because the frozen rule requires both
+control denominators. This is an inconclusive verdict, not evidence that every
+mechanism is small.
+
+## Residual + interaction line
+
+The official residual is N/A under both controls because the official share
+ledger is incomplete. The unchanged arithmetic over all raw diagnostic shares
+is reported only for transparency:
+
+| Control | Diagnostic share sum | Diagnostic residual + interaction (95% CI) | Status |
+|---|---:|---:|---|
+| P03 | -0.1549 | 1.1549 [0.5536, 1.9397] | Diagnostic only |
+| P04 | -6.5142 | 7.5142 [-21.3965, 22.8767] | Diagnostic only; unstable denominator |
+
+Nothing is forced to 100%. The residual can include interactions, unablated
+MoE routing, chunked-prefill effects, double-counting, and measurement error.
+
+## Config-tier A2 measured delta
+
+A2 added the P3-derived sizes `{3,5,6,7}` to the full default capture list.
+Its measured `delta_E` is **+0.048376**, 95% CI
+`[-0.315361, 0.411446]`, or **+1.605%** relative to base. Pure-decode padding
+falls from 13.7405% to 2.3989%, an 82.541% reduction. New recorded-arrival
+runs expose decode-B support 1-13, so uncaptured 9-13 leave coverage at 98.879%;
+both this and the sub-90% padding reduction invalidate the causal A2 share.
+
+The config cost is reproducible: base graph capture is 11 seconds and 0.80 GiB
+in all three runs; A2 is 12-14 seconds and 0.84 GiB, a 1-3 second and 0.04 GiB
+increase. The measured config-tier result is therefore a small, statistically
+uncertain efficiency gain with a failed preregistered removal gate.
+
+## GPU total
+
+New Phase-5 spend is **2.4388167443 H20-hours** of the 6.0-hour cap, including
+the superseded first wave, exact reruns, continued primary matrix, burn-ins,
+and background arm. **3.5611832557 H20-hours remain unspent.** Final state on
+all eight H20s is 0 MiB and 0% utilization with zero compute processes.
+
+## Sanity block
+
+**Anomaly first:** P04's gap CI contains zero and 45.353% of its denominator
+draws are nonpositive; the dual-control ledger is inconclusive and inferential
+claims stop there.
+
+| Numeric family | n | Finite | Missing | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---:|---:|---|
+| Primary run `E_token` | 15 | 15 | 0 | 2.623534 | 3.083263 | 15 | Positive; per-arm results not all identical |
+| Clean Layer-1 steps | 15 | 15 | 0 | 11,178 | 18,400 | 15 | Non-negative; continuous/balanced per run |
+| Clean duration (s) | 15 | 15 | 0 | 240 | 240 | 1 expected | Exact `3*80`; zero clean failures |
+| Offered rate (req/s) | 15 | 15 | 0 | 0.470833 | 0.487500 | 2 | Every run within 5% of 0.4725 |
+| Warm-up completions | 15 | 15 | 0 | 25 | 28 | 2 | Every run >=16 |
+| Warm-up long completions | 15 | 15 | 0 | 12 | 16 | 3 | Every run >=1 with input >=8192 |
+| Clean first-capture events | 15 | 15 | 0 | 0 | 0 | 1 expected | Zero events and zero uncovered descriptors |
+| P10 drain duration (s) | 15 | 15 | 0 | 0.676181 | 6.003959 | 15 | Non-negative; all below 600 |
+| Control gap `E_token` | 2 | 2 | 0 | 0.040952 | 1.721848 | 2 | P03 stable; **P04 unstable red flag** |
+| Raw diagnostic shares | 8 | 8 | 0 | -9.464746 | 1.593268 | 8 | Finite; intentionally not constrained to [0,1] |
+| Diagnostic residuals | 2 | 2 | 0 | 1.154931 | 7.514211 | 2 | Finite; never forced to close |
+| New H20-hours | 1 | 1 | 0 | 2.438817 | 2.438817 | 1 | Non-negative; below 6.0 |
+
+Checked invariants: source/helper/protocol provenance; identical request IDs,
+content/token totals, and prompt-preserving manifests; C00-TP1 four-way
+placement; detached controller ownership; exact clean windows; zero failures;
+offered-rate tolerance; A-P5-1 compile/capture ordering and coverage; Layer-1
+footer/sidecar balance, continuous step indices, token composition, and zero
+drops; non-negative counters; drain gates; bridge decision; no public prompt or
+generated-text leakage; public-disk/free-space limits; GPU hard-cap compliance;
+and zero-process/zero-memory cleanup. Manipulation failures are A2, A3, and A4;
+the sole machine sanity red flag is P04 denominator instability.

docs/opprof/phase6-protocol.md

@@ -0,0 +1,638 @@
+# OpProf Phase 6 pre-registered cross-version churn protocol
+
+Status: **ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED**.
+
+Date frozen: 2026-07-12 (Asia/Singapore). Phase 6 asks whether the vLLM 0.20.0
+best configuration and ranking on the C1
+`TP={1,2,4} x max-num-seqs={8,16,32,64}` surface survive an upgrade to patched
+vLLM 0.24.0. This is a paired historical-anchor experiment, not a fresh tuner
+search. It measures churn at the exact recorded C1 request sets and reports
+where the recorded local frontier moved.
+
+## Approved dispositions (orchestrator; 2026-07-12)
+
+All five open decisions are approved exactly as proposed, without changing an
+anchor, threshold, estimator, validity gate, or budget:
+
+1. The upgrade-path estimand is approved. Resolved default changes are part of
+   observed churn; dash1->dash0 and co-location remain explicit limitations and
+   the result is not described as a pure engine-version causal effect.
+2. The adaptive 25-anchor subset is approved: every cell receives its old peak
+   plus the direction-relevant adjacent anchor, and TP4/MNS16 receives both
+   neighbors.
+3. The 5% material-frontier threshold, `tau-b>=0.8` ranking-survival threshold,
+   and zero-anchored floor-bucket argmax/trap rules are approved.
+4. One primary observation per historical anchor plus the 0.35-H20-hour
+   confirmation reserve is approved.
+5. The A-P5-1-class warm-up long tier is approved as raw input `>4096` tokens
+   for this 0-8192 workload.
+
+This approval authorizes the later execution only after pinned-input, detached
+ownership, cold-start, echo-before-launch, placement, privacy, cleanup, and
+projected-budget gates pass. The 3.0-H20-hour cap remains absolute.
+
+## Amendment A-P6-1 — checkpoint-sidecar accounting and conditional cap
+
+The first Phase-6 W1 request replays were incorrectly rejected by requiring a
+clean in-stream footer and `final=true` sidecar after a non-graceful shutdown.
+That contradicts the already accepted A-P3-5/P5 accounting rule. Before any
+resume, all four W1 cell streams and their eight measured anchor intervals must
+be CPU-re-adjudicated against the latest atomic checkpoint sidecar. With no
+in-stream footer, the sidecar is authoritative when every complete JSONL line
+decodes, step indices are contiguous, on-disk data-record count equals
+`written_records`, final data-line step equals `last_step_index`,
+`encoded_records = written_records + dropped_records`, `dropped_records=0`, and
+the checkpoint is within the configured one-second flush interval (plus the
+accepted 100-ms scheduling tolerance) of the recorded shutdown boundary. At
+most one post-checkpoint flush interval may be absent. A balanced stream accepts
+all covered W1 replay intervals without a GPU rerun.
+
+All subsequent waves use the graceful P5 path: start `vllm serve` with positive
+`--shutdown-timeout`, signal each API parent with SIGINT, wait up to 150 seconds
+for EngineCore drain/finalization, and use process-group SIGTERM/SIGKILL only as
+fallback. The preferred result is one in-stream footer plus an agreeing
+`final=true` sidecar; the atomic checkpoint rule remains the crash fallback.
+
+Only if any W1 sidecar fails the CPU balance/coverage gate does A-P6-1c activate:
+the hard cap becomes **3.5 H20-hours**, the activation is logged before launch,
+and W1 is rerun under graceful shutdown. If all W1 sidecars balance, A-P6-1c is
+not activated, W1 is accepted in place, its already charged 0.299393 H20-hours
+is retained, and execution resumes at W2 under the original 3.0-hour cap.
+
+## Amendment A-P6-2 — authoritative solo frontier tier and 6.0-hour cap
+
+The user raises the cumulative Phase-6 hard cap from **3.0 to 6.0 H20-hours**.
+This amendment responds to the W2/W3 confirmation evidence: when neighboring
+wave clients became idle, pass rate changed from `0.412 -> 1.000` and `0.028 ->
+0.957` at TP2/MNS32 and from `0.036 -> 1.000` at TP4/MNS16, while Layer-1
+waiting means collapsed from `1.22 -> 0.00`, `8.50 -> 0.35`, and `28.33 ->
+0.06`. Because the v0.20 C1 baseline ran one cell at a time on dash1,
+co-located SLO feasibility is not a valid authoritative tier for the paired
+frontier. Co-location remains an explicitly reported, metric-dependent
+methodological ablation; its throughput-only observations are indicative.
+
+Every SLO-frontier-decisive Phase-6 anchor is therefore run **solo**: exactly
+one vLLM server and one replay client are active on dash0, no other Phase-6
+server/client is resident, and only the cell's `TP` GPUs are allocated. Each
+solo cell gets a fresh server process, the frozen exact-selection client, an
+independent 16-request long-tier warm-up, A-P5-1 cold-start gates, Layer-1
+telemetry, and graceful A-P6-1 shutdown. The solo result is authoritative for
+feasibility, frontier, floor bucket, argmax, tau-b, and trap decisions. A
+co-located-only result may not fill a missing solo frontier or support a paper
+claim.
+
+The mandatory solo set has no cell exclusions:
+
+| Cell set | Mandatory solo anchors | Inclusion reason |
+|---|---|---|
+| TP1/MNS8 | existing L+P | co-located peak failed |
+| TP1/MNS16,32,64 | existing P+U | co-located U passed and censored the frontier |
+| TP2/MNS8,16 | existing L+P | co-located peak failed; L distinguishes bracket vs left censoring |
+| TP2/MNS32 | existing L+P | old argmax; both anchors split primary/confirmation |
+| TP2/MNS64 | existing L+P | the indicative `>29.4% down` bound is unquotable until solo-confirmed |
+| TP4/MNS8 | existing L+P | co-located peak failed; trap-neighbor comparison |
+| TP4/MNS16 | existing L+P+U | named trap; peak split and L failed co-located |
+| TP4/MNS32,64 | P plus direction-relevant L/U | W4 completion; unmeasured under the 3.0-hour cap |
+
+After these 25 anchors, the controller may crawl only the pinned 92-anchor
+history: upward while the highest solo anchor passes or downward while the
+lowest solo anchor fails, stopping at the first opposite-feasibility anchor.
+There is no interpolation and no unrecorded request set. A pass rate in
+`[0.93,0.97]`, or a solo/co-located disagreement that changes argmax/trap or a
+material-frontier claim, triggers a same-solo-placement confirmation; a split
+gets a third trial for the frozen 2-of-3 rule if projected budget permits.
+Unresolved or history-edge frontiers remain explicitly censored.
+
+Priority is W4, TP2/MNS32, TP2/MNS64, TP4/MNS16, then the remaining failed or
+censored cells, so a budget stop preserves the most decision-relevant evidence.
+All twelve cells are nevertheless included. The planning envelope is 3.44 new
+H20-hours for serialized startup, warm-up, 25 mandatory anchors, bounded crawl,
+graceful drain, and a 0.20-hour safety reserve. Added to the already charged
+2.291173 hours, the launch projection is **5.731173/6.0 H20-hours**. Before
+each one-cell wave, the controller echoes cell, anchors, GPU placement, input
+paths, charged spend, remaining projection, and cap. Projected or actual spend
+reaching 6.0 stops further GPU work.
+
+Final analysis preserves the complete W1-W3 co-located attempt history and
+reports a solo-versus-co-located delta table for every exact anchor pair.
+ARGMAX, RANKING, and TRAP use solo values only. RANKING remains inconclusive
+and tau-b non-evaluable unless all 12 solo frontiers are bounded under the
+pinned history.
+
+## Headline claim and falsifiable outcomes
+
+The paper-facing question is:
+
+> Does the C1 vLLM 0.20.0 per-GPU SLO-feasible ranking, especially global best
+> TP2/MNS32 and local trap TP4/MNS16, remain valid on vLLM 0.24.0 when the exact
+> historical request-selection and SLO semantics are replayed?
+
+Phase 6 can produce four distinct outcomes:
+
+1. **SURVIVES:** TP2/MNS32 remains in the top floor bucket, the full-surface
+   Kendall tau-b is at least 0.8, and the TP4/MNS16 trap persists.
+2. **ARGMAX MOVED:** TP2/MNS32 is not in the top v0.24 measured-frontier bucket.
+3. **RANKING MOVED WITHOUT ARGMAX MOVEMENT:** the old best survives but tau-b is
+   below 0.8 or the trap changes status.
+4. **INCONCLUSIVE/PARTIAL:** a decision-critical frontier is unbracketed, an
+   anchor is not reproducible, repeated boundary verdicts disagree, the 12-cell
+   surface is incomplete, or the 3.0-H20-hour cap stops required work.
+
+The experiment quantifies observed upgrade churn. Because engine version,
+host, resolved defaults, and co-location differ together, it does not identify
+a pure causal effect of one vLLM commit.
+
+## Frozen vLLM 0.20.0 surface
+
+The historical objective is the maximum measured SLO-feasible offered rate
+divided by TP:
+
+```text
+f20(c) = max feasible selected_request_count / 60 seconds / TP
+```
+
+| TP | MNS8 | MNS16 | MNS32 | MNS64 |
+|---:|---:|---:|---:|---:|
+| 1 | 2.1000 | 2.3500 | 2.2833 | 2.2833 |
+| 2 | 2.2750 | 2.2750 | **3.2833** | 3.2583 |
+| 4 | 1.2833 | **2.4417** | 2.4417 | 2.4417 |
+
+Values are request/s/GPU. The global best is TP2/MNS32. TP4/MNS16 is the
+registered local trap: it is the first point on a TP4 plateau, has no improving
+adjacent MNS move, but is below the global best.
+
+The pinned ground-truth asset is
+`/home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json`, SHA-256
+`23ff3e6f6b88df8632bf37d37c0ff87bfef9d1676db81592948c08e898f7a670`.
+It contains 92 probe observations: eight anchors for each TP1/TP2 cell and seven
+for each TP4 cell.
+
+## Comparability contract
+
+### Exact workload and selection semantics
+
+The private materialized workload is
+`trace_windows/traces/chat_w20260311_1000.jsonl`, 32,606 rows, SHA-256
+`f539f38eb0ee0f750e3c23ff47df6eed3faf723a25f1444d55665a85871750b9`.
+Its window record is `trace_windows/windows.json`, SHA-256
+`23c432e7439508b07991cabfe1db9977ebb55f8ddc39ca214a627fc5f5ae4725`:
+window `[0,600]` seconds becomes `[0,60]` under replay scale 0.1.
+
+The later helper must call or byte-for-byte reproduce the following pinned
+AITuner implementations, rather than use the Phase-3/5 client:
+
+| Source | SHA-256 | Frozen role |
+|---|---|---|
+| `src/aituner/trace.py` | `6bb19f0e3eb42b1aecc6969a5f20d65eb505a91564dd117dd149155a6114eb35` | load, stable time sort, cap/downsample, threshold selection |
+| `src/aituner/worker.py` | `18337fe273a48d1cbb2bc364f774d100f8ffa2be5f9cd6a7c58c8ed4717c5e48` | rate-following submit loop, early stop, denominator accounting |
+| `src/aituner/slo.py` | `1d2e5b1ed9d0d6d6c07f8b5b6fa56d36ed9bb62710be54c125e9b2890f4f96d5` | stepped SLO evaluation |
+| `src/aituner/http_client.py` | `213b80d5b37cc74333517f46ab4e8cdba1a274bdbfabdfc14207b314f61799ac` | urllib/SSE TTFT, TPOT, usage accounting |
+
+The exact order is normative:
+
+1. Read all JSONL rows in file order. Retain raw `input_length` in `[0,8192]`.
+2. Build the same chat body and stable-sort by scaled arrival time only. Stable
+   ties retain source order; sorting by `(timestamp,sampling_u,...)` is forbidden.
+3. For TP1/TP2, even-downsample the full filtered, time-sorted list to 512
+   entries **before** threshold selection, using index
+   `floor(i*N/512)` for `i=0..511`. TP4 is uncapped.
+4. Select exactly the requests with `sampling_u <= anchor`. Preserve their
+   selected service order. Do not resample, wrap, shuffle, or interpolate.
+5. Submit at each row's materialized `timestamp*0.1`; do not uniformize,
+   rebase individual gaps, or replace the recorded burst process.
+6. Send the original messages through `/v1/chat/completions` with `stream=true`,
+   `stream_options.include_usage=true`, and the row's temperature when present.
+   Set both `min_tokens=128` and `max_tokens=128`.
+7. Use maximum concurrency 64. Client request timeout remains 900 seconds.
+8. Verify `usage.completion_tokens==128`; missing usage, a different token count,
+   HTTP failure, timeout, and every request not submitted because the SLO verdict
+   became unrecoverable are failures in the original selected-request denominator.
+9. Preserve original early-stop logic: after more than
+   `N-ceil(0.95*N)` failed evaluations, synthesize failures for the remaining
+   selected requests. Disable adaptive Stop-A; it was absent in C1.
+
+An offline, prompt-free preflight must regenerate all 92 historical request
+counts exactly. The local reconstruction already yields 17,710 filtered rows,
+512 capped rows, and **92/92 count matches**. Any future mismatch in count,
+request-ID hash, arrival hash, raw-length hash, or requested-token sum stops the
+GPU launch.
+
+### Exact SLO and score
+
+Request feasibility uses the original raw trace input length, not tokenizer
+usage after chat templating:
+
+```text
+raw input <= 4096:  TTFT <= 2000 ms
+raw input <= 32768: TTFT <= 4000 ms
+otherwise:          TTFT <= 6000 ms
+all requests:       TPOT <= 50 ms
+anchor feasible:    SLO-passing selected requests / all selected requests >= 0.95
+```
+
+Although the C1 filter limits inputs to 8192, the complete three-step rule is
+retained. TTFT is client wall time from request start to first nonempty content.
+TPOT is `(last_content_time-first_content_time)/(usage_completion_tokens-1)`.
+The paired surface score is always selected count divided by 60 seconds and TP;
+completed throughput is a diagnostic and never replaces the historical score.
+
+### Serving configuration
+
+The v0.24 command repeats the explicit C1 launch surface:
+
+```bash
+MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
+VENV=/tmp/wjh-opprof-phase2-dash0-20260711/.venv
+RUN_ROOT=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6
+
+taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU_IDS" \
+  VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
+  "$VENV/bin/vllm" serve "$MODEL" \
+  --host 127.0.0.1 --port "$PORT" \
+  --served-model-name qwen3-30b-a3b-community \
+  --max-num-batched-tokens 8192 \
+  --max-num-seqs "$MNS" \
+  --tensor-parallel-size "$TP" \
+  --shutdown-timeout 120
+```
+
+`--shutdown-timeout` is operational and outside the measured interval. As in
+C1, do **not** explicitly set block size, GPU-memory utilization, prefix
+caching, chunked prefill, CUDA-graph capture sizes, DP, or EP. Their resolved
+v0.24 values must be parsed from startup logs and reported. Allowing version
+defaults to change is intentional: this experiment asks whether an operator's
+old explicit configuration remains good after upgrading, not whether internal
+defaults can be manually made identical.
+
+Preflight records `vllm.__version__`, source commit, patch checksums, Python,
+PyTorch, CUDA, NCCL, Triton, driver, model/tokenizer file hashes, MoE backend,
+resolved scheduler/cache/graph config, compile-cache key, and exact command.
+The expected source is the existing Phase-3/5 patched vLLM 0.24 environment;
+any missing OpProf patch or unexpected model path is a stop condition.
+
+### Client equivalence, not a new client covariate
+
+The later `scripts/opprof_phase6_client.py` is permitted only as a thin anchor
+driver around the four pinned AITuner modules. Its interface is normative:
+
+```bash
+P6C='python scripts/opprof_phase6_client.py'
+
+$P6C preflight \
+  --trace trace_windows/traces/chat_w20260311_1000.jsonl \
+  --windows trace_windows/windows.json --window-id chat_w20260311_1000 \
+  --ground-truth /home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json
+
+$P6C run-anchor --cell "$CELL" --anchor "$U" \
+  --base-url "http://127.0.0.1:$PORT" \
+  --model qwen3-30b-a3b-community --completion-tokens 128 \
+  --replay-time-scale 0.1 --max-concurrency 64 \
+  --max-requests "$CAP_OR_NONE" --target-pass-rate 0.95 \
+  --ttft-step-ms '4096:2000,32768:4000,inf:6000' --tpot-ms 50 \
+  --request-timeout-seconds 900 --probe-deadline-ceiling-seconds 900 \
+  --result-dir "$ANCHOR_DIR"
+```
+
+Before GPU execution, a no-server golden test must prove identical selected IDs,
+order, arrivals, raw input lengths, serialized bodies excluding model/URL, and
+SLO decisions for synthetic outcomes. The wrapper may add interval markers and
+sanity JSON, but may not change HTTP transport, scheduling, completion checks,
+early stop, or latency formulas.
+
+### Covariate ledger and estimand limit
+
+| Covariate | C1 v0.20 | Phase 6 v0.24 | Treatment/handling |
+|---|---|---|---|
+| Engine | community vLLM 0.20.0 | patched vLLM 0.24.0 | Primary intended churn dimension; exact builds recorded |
+| Host | dash1 | dash0 | Both 8x H20 96 GB class; CPU/NUMA/clocks/driver/kernel recorded, not assumed identical |
+| GPU visibility | TP1 C1 env exposed GPUs `0,1`, TP1 consumed one | expose exactly TP GPUs | Declared operational covariate; H20-hour charge follows TP |
+| Trial placement | recovered C1 harness trials | up to four co-located servers | CPU/NUMA pinned; reject outside GPU processes and report co-location limitation |
+| HTTP client | pinned AITuner urllib/SSE | same code paths through thin wrapper | Eliminated as a semantic covariate by golden tests |
+| Workload | materialized C1 chat window | byte-identical file/hash | Fixed |
+| Selection/cap | cap-before-threshold; TP4 uncapped | exact same | Fixed and 92-count preflighted |
+| SLO length | raw input length | raw input length | Fixed; server usage length is diagnostic only |
+| Explicit flags | TP, MNS, MBT8192 | same | Fixed |
+| Unspecified defaults | v0.20 defaults | v0.24 defaults | Intentionally allowed to churn; resolved values reported |
+| Telemetry | no Layer 1 | zero-overhead OpProf Layer 1 | Added measurement; overhead evidence -0.04196%, CI [-0.17443%, 0.04550%] |
+
+Therefore “version churn” means the observed paired upgrade path under these
+documented platform covariates. It must not be rewritten as a same-host causal
+microbenchmark.
+
+## Budget-constrained anchor subset
+
+### Adaptive adjacent-anchor design
+
+The primary subset improves on measuring next-infeasible anchors only for a
+chosen top four:
+
+1. Measure the recorded v0.20 peak anchor for all 12 cells.
+2. For every non-trap cell, select exactly one adjacent recorded anchor by a
+   result-independent rule fixed now:
+   - if the v0.20 peak remains feasible on v0.24, measure the nearest higher
+     recorded anchor, which was infeasible on v0.20;
+   - if the peak becomes infeasible, measure the nearest lower recorded anchor,
+     which was feasible on v0.20.
+3. For TP4/MNS16, measure lower, peak, and higher anchors unconditionally. This
+   directly brackets the registered trap.
+
+Thus exactly 25 primary anchors are measured: TP1 eight, TP2 eight, and TP4
+nine. The design covers every cell and spends the second anchor in the direction
+where its frontier could have moved. It also includes the suggested top-four
+test. Under the floor-bucket rule, the nominal top-four cutoff expands to five
+cells because TP4/MNS16/32/64 share one bucket; all five receive an upward test
+whenever their old peak still passes.
+
+If an adjacent anchor does not bracket the transition—higher also passes or
+lower also fails—the cell is censored. Remaining budget may crawl one recorded
+anchor farther outward, in this priority order: old global best TP2/MNS32,
+TP2/MNS64, the TP4 plateau MNS16/32/64, then other cells by old score. Crawling
+uses only that cell's recorded history and stops at the first transition. No
+new `sampling_u` value may be invented. Failure to close a decision-critical
+bracket before the budget stop makes the applicable argmax/ranking verdict
+inconclusive.
+
+### Boundary confirmations
+
+The original surface has one observation per anchor. Phase 6 keeps that paired
+unit but reserves at most 0.35 H20-hours for decision-triggered confirmation.
+Repeat an anchor if its v0.24 pass rate lies in `[0.93,0.97]`, or if its single
+verdict alone changes argmax or trap status. Priority is argmax, trap, material
+frontier calls, then other cells. Agreement across two runs freezes the verdict;
+disagreement requests a third run if budget permits and uses 2-of-3. Without a
+third run, that anchor and every dependent headline verdict are inconclusive.
+Request outcomes are arrival-correlated, so a binomial CI is descriptive only
+and cannot replace these repeat rules.
+
+## Execution plan
+
+### Preflight and detached ownership
+
+The Phase-6 controller follows A-P3-2 detached ownership: one controller PID,
+per-stage atomic state, process-group/environment markers, resumable completed
+anchors, exact command logs, and cleanup restricted to its own markers. Before
+launch it must verify dash0 has eight H20s, no outside compute processes on
+assigned GPUs, at least 100 GB CPFS free, pinned source/workload/helper hashes,
+all 92 selection counts, legal TP placement, and a projected total below 3.0
+H20-hours.
+
+The autonomous launch log must echo one resolved line before starting:
+
+```text
+LAUNCH_ECHO host=dash0 engine=vllm-0.24.0-patched model=Qwen3-30B-A3B trace_sha=f539f38e... cells=12 primary_anchors=25 waves=4 gpus=0-7 warmup=16 clean_horizon=60s drain=derived<=120s est_wall=25-35min est_gpu=2.65_H20h confirm_reserve=0.35_H20h hard_cap=3.0_H20h
+```
+
+### Rate-following cold-start and drain gates
+
+The throughput-drift stationarity gate is forbidden for these rate-following
+anchors. A-P5-1-class cold-start gates are applied instead:
+
+1. Every torch.compile/CUDA-graph first-occurrence event must precede the first
+   measured anchor, verified from server logs and Layer 1.
+2. A deterministic unmeasured warm set must complete at least 16 exact-output
+   requests, including at least one request with raw input `>4096` tokens. The
+   peak-selected sets contain 43-184 such requests, so this gate is feasible.
+3. No first-occurrence compile/capture event may occur in any measured anchor;
+   all graph-hit `(runtime_mode,bucket_tokens)` descriptors must be covered by
+   startup capture logs. A violation invalidates that anchor only.
+4. Before the next anchor, all selected requests must be accounted for and three
+   consecutive controller samples must show running/waiting/deferred queues at
+   zero. Prefix cache is not explicitly enabled; nevertheless a full drain is
+   required.
+
+The probe deadline reproduces `_probe_drain_deadline`: last selected arrival +
+raw-length TTFT budget + `128*50 ms` + 30 seconds, capped by the historical
+900-second ceiling. With raw inputs <=8192 this is about 100.4 seconds from
+probe start; the controller has a 120-second class watchdog for cleanup. A
+deadline miss is an SLO-infeasible anchor, not permission to omit denominator
+requests. The server then shuts down through the official 120-second path.
+
+### Four-wave packing
+
+| Wave | Cells | Placement | Per-server measured anchors | Planned wall |
+|---|---|---|---:|---:|
+| W1 | TP1/MNS8,16,32,64 | four servers on GPUs 0,1,2,3; GPUs 4-7 idle | 2 each | 4-6 min |
+| W2 | TP2/MNS8,16,32,64 | four servers on GPU pairs 0-1,2-3,4-5,6-7 | 2 each | 4-6 min |
+| W3 | TP4/MNS8 and TP4/MNS16 trap | GPU quads 0-3 and 4-7 | 2 and 3 | 5-7 min |
+| W4 | TP4/MNS32 and TP4/MNS64 | GPU quads 0-3 and 4-7 | 2 each | 4-6 min |
+
+Each server starts once, passes warm-up once, runs its peak first, drains, then
+runs the predeclared adjacent anchor. Trap lower/upper order after peak is
+counterbalanced by execution seed 20260717. Anchor confirmation occurs before
+that server is released when possible. The controller records GPU clocks,
+power, utilization, memory, CPU load, NUMA placement, and outside processes
+through every wave. Any contamination invalidates the whole co-located wave.
+
+Actual H20-hours are charged from server launch through verified cleanup for
+every allocated GPU, including idle time while a paired TP4 cell finishes.
+Before each wave or confirmation, the controller recomputes
+`spent + current + conservative_remaining`; it stops before launch if the
+projection can reach 3.0. Optional crawling and confirmations are sacrificed
+before any hard-cap violation.
+
+## Measurement and mechanism notes
+
+For every anchor, report selected/admitted/completed/exact-output/SLO-pass
+counts; early-stop reason; offered and completed req/s; TTFT/TPOT mean and
+p50/p90/p95/p99; failures by raw-length TTFT bucket; elapsed/drain time; and
+peak GPU memory/utilization. The surface objective remains offered req/s/GPU at
+the highest measured feasible recorded anchor.
+
+Layer 1 is active over the complete anchor service interval, from the first
+submission through the last selected-request accounting event. Aggregate:
+
+- prefill, decode, and mixed-step counts and token shares;
+- decode-batch distribution and 5-second CV;
+- FULL/PIECEWISE/NONE graph-mode shares, bucket padding, and uncovered support;
+- running/waiting/deferred queues and queue CV;
+- preemption count/rate;
+- KV blocks used/total, mean/max usage, and allocation pressure; and
+- prefix queries/hits as a resolved-default diagnostic.
+
+Mechanism notes compare v0.24 cells/anchors and may relate a frontier change to
+preemptions, KV pressure, batching, or graph modes. C1 lacks Layer-1 telemetry,
+so these notes cannot claim that a measured composition field itself changed
+from v0.20 or caused the version drift. Historical v0.20 TTFT/TPOT failure
+reasons may be paired directly; device-mechanism attribution remains
+descriptive.
+
+## Statistical and ranking analysis
+
+### Floor buckets
+
+For each unrounded score vector `s`, use the SimFid inventory rule:
+
+```text
+tol(s) = max(1e-9, 1e-6 * max_c(abs(s(c))))
+bucket_s(c) = floor(s(c) / tol(s))
+```
+
+Two scores tie only when their integer buckets match. Bucketing occurs before
+display rounding, pair signs, tau-b, top selection, or trap evaluation. Report
+`tol`, bucket integers, and occupied intervals `[j*tol,(j+1)*tol)`. For v0.20,
+`tol20=3.283333333333333e-6`; the TP4 MNS16/32/64 tie makes nominal top-4 an
+effective top-5 set.
+
+### Per-cell frontier
+
+For each cell, define the v0.24 measured recorded-anchor frontier:
+
+```text
+f24(c) = max_{measured u with accepted feasible verdict} N_c(u) / 60 / TP
+drift(c) = (f24(c) - f20(c)) / f20(c)
+```
+
+The predeclared material threshold is **X=5% relative**, large enough to avoid
+calling one-request anchor quantization a paper-level churn effect. Report two
+orthogonal labels:
+
+- **BOUNDARY DOWN/UP:** old peak flips feasible->infeasible, or the old adjacent
+  infeasible anchor flips to feasible, respectively;
+- **MATERIAL FRONTIER MOVED:** `abs(drift)>5%`; otherwise report the boundary
+  flip as sub-material.
+
+If peak passes and higher fails, and `abs(drift)<=5%`, the local recorded
+frontier is **STABLE**. If peak and lower both fail or peak and every affordable
+higher anchor pass, report a directional bound and **UNBRACKETED**, never an
+invented peak.
+
+### Argmax, ranking, and trap
+
+- **ARGMAX MOVED** when TP2/MNS32 is not in the top `bucket_f24` among all 12
+  decision-bounded cells. **ARGMAX SURVIVED** requires it to share the top
+  bucket and no censored competitor capable of overtaking it. Otherwise it is
+  inconclusive.
+- Compute Kendall tau-b over all 12 floor-bucketed `f20/f24` scores and report
+  concordant, discordant, v0.20-tied, v0.24-tied, and comparable-pair counts.
+  **RANKING SURVIVES** requires an evaluable 12-cell surface, argmax survival,
+  `tau-b>=0.8`, and no >5% top-bucket pair reversal. `tau-b<0.8` is **RANKING
+  MOVED**. A partial surface receives no suite-level tau claim.
+- **TRAP PERSISTS** when TP4/MNS16 is in a bucket at least as high as both
+  adjacent TP4 cells MNS8 and MNS32, yet below the global-best bucket. It
+  **ESCAPES** if an adjacent TP4 move is strictly better, and **CEASES TO BE A
+  TRAP** if it joins the global-best bucket. Missing/unbracketed inputs make trap
+  status inconclusive.
+
+No arithmetic average of normalized scores is reported. Absolute rates,
+relative drift, the paired table, and every excluded/censored anchor accompany
+all ranking summaries.
+
+## Later deliverables and artifact contract
+
+Execution, after approval, must produce:
+
+- `docs/opprof/phase6-results.md`;
+- `runs/opprof-phase6/phase6/metrics.json` containing the complete paired
+  surface, anchor/run values, selection hashes, feasibility, floor buckets,
+  frontier/argmax/ranking/trap verdicts, confirmation history, Layer-1 notes,
+  covariates, and GPU accounting;
+- exact commands, controller state, environment/machine fingerprints, server
+  logs, client outcomes, Layer-1 JSONL/footer/sidecar, monitor samples, and
+  per-anchor sanity JSON; and
+- an operational-findings section covering startup/capture gates, drains,
+  early stops, co-location, default changes, retries, contamination, and budget.
+
+The results begin with data sanity and stop inferential claims on any red flag.
+Every numeric family ends with `n`, finite/missing, min, max, distinct count,
+and applicable invariants. Prompt text, messages, generated content, source
+substrings, and private request bodies stay in mode-0700 storage. Public files
+contain only hashes, IDs, lengths, timestamps, outcomes, counters, and summaries.
+
+## Anchor-subset table
+
+Notation is `u / selected N / recorded req/s/GPU`. `P` is always measured. For
+non-trap cells, measure `U` if P passes on v0.24, otherwise `L`. TP4/MNS16
+measures all `L+P+U`. Planning cost per anchor includes the 60-second replay and
+20 seconds expected drain/accounting; shared startup/warm-up is budgeted below.
+
+| Cell | L: nearest recorded feasible below P | P: recorded peak | U: nearest recorded infeasible above P | Primary selection | H20-hour cost each |
+|---|---|---|---|---|---:|
+| TP1/MNS8 | 0.21875 / 121 / 2.0167 | 0.2265625 / 126 / 2.1000 | 0.23046875 / 130 / 2.1667 | P + (U if pass else L) | 0.0222 |
+| TP1/MNS16 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | 0.25 / 143 / 2.3833 | P + (U if pass else L) | 0.0222 |
+| TP1/MNS32 | 0.234375 / 132 / 2.2000 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | P + (U if pass else L) | 0.0222 |
+| TP1/MNS64 | 0.234375 / 132 / 2.2000 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | P + (U if pass else L) | 0.0222 |
+| TP2/MNS8 | 0.4921875 / 269 / 2.2417 | 0.49609375 / 273 / 2.2750 | 0.5 / 276 / 2.3000 | P + (U if pass else L) | 0.0444 |
+| TP2/MNS16 | 0.4921875 / 269 / 2.2417 | 0.49609375 / 273 / 2.2750 | 0.5 / 276 / 2.3000 | P + (U if pass else L) | 0.0444 |
+| TP2/MNS32 | 0.75 / 391 / 3.2583 | 0.75390625 / 394 / **3.2833** | 0.7578125 / 396 / 3.3000 | P + (U if pass else L) | 0.0444 |
+| TP2/MNS64 | 0.5 / 276 / 2.3000 | 0.75 / 391 / 3.2583 | 0.75390625 / 394 / 3.2833 | P + (U if pass else L) | 0.0444 |
+| TP4/MNS8 | 0.016055910008 / 301 / 1.2542 | 0.016591107009 / 308 / 1.2833 | 0.017126304009 / 317 / 1.3208 | P + (U if pass else L) | 0.0889 |
+| TP4/MNS16 trap | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / **2.4417** | 0.034252608017 / 600 / 2.5000 | **L + P + U** | 0.0889 |
+| TP4/MNS32 | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / 2.4417 | 0.034252608017 / 600 / 2.5000 | P + (U if pass else L) | 0.0889 |
+| TP4/MNS64 | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / 2.4417 | 0.034252608017 / 600 / 2.5000 | P + (U if pass else L) | 0.0889 |
+
+## Total GPU estimate versus 3.0 cap
+
+The 25 primary anchors contain 8 TP1, 8 TP2, and 9 TP4 replays. At the
+80-second per-anchor planning charge they consume `0.1778 + 0.3556 + 0.8000 =
+1.3334` H20-hours. Four shared startup/warm-up/cleanup envelopes plus runtime
+variance are budgeted at 1.3166 hours, for a **2.65-H20-hour primary plan**.
+The remaining **0.35 H20-hours** is reserved for boundary confirmations or
+decision-critical outward crawling. The absolute launch cap is **3.0
+H20-hours**; projected or actual spend reaching the cap stops all further work.
+Expected wall time is 25-35 minutes over four waves. Expected public disk is
+0.3-0.6 GB and must remain below 3 GB.
+
+## Decision rules
+
+1. Anchor feasibility is exact-output request pass rate `>=0.95` using raw-length
+   stepped TTFT and 50-ms TPOT; absent/early-stopped requests remain failures.
+2. Boundary movement is an old peak feasible->infeasible flip or an old adjacent
+   infeasible->feasible flip. Material frontier churn is `abs(drift)>5%`.
+3. Apply zero-anchored floor buckets with set-wide `tol=max(1e-9,1e-6*max|s|)`
+   before every tie/rank decision.
+4. ARGMAX MOVED iff TP2/MNS32 is outside the v0.24 top bucket on a bounded
+   12-cell subset; censored contenders make survival inconclusive.
+5. RANKING SURVIVES only with all 12 cells, argmax survival, tau-b >=0.8, and no
+   >5% top-bucket pair reversal. Tau-b <0.8 is RANKING MOVED.
+6. TP4/MNS16 trap persists only when it is no worse than adjacent MNS8/MNS32 and
+   remains below the global best; an improving neighbor means escape.
+7. Boundary `[0.93,0.97]` or verdict-changing anchors require confirmation when
+   budget permits. Disagreement without a 2-of-3 resolution is inconclusive.
+
+## Open decisions for orchestrator
+
+1. Approve the **upgrade-path estimand**: resolved default changes are part of
+   churn, while dash1->dash0 and co-location remain explicit limitations rather
+   than pretending this is a pure engine-version causal effect.
+2. Approve the **adaptive 25-anchor subset**, which gives every cell a
+   direction-relevant neighbor and the trap both neighbors, instead of spending
+   next-infeasible anchors only on a tie-ambiguous top four.
+3. Approve **5% material frontier drift**, `tau-b>=0.8`, and the stated
+   argmax/trap floor-bucket rules.
+4. Approve one primary observation per historical anchor with the 0.35-hour
+   reserve for `[0.93,0.97]`/verdict-changing confirmations, rather than reducing
+   surface coverage to replicate every anchor.
+5. Approve the A-P5-1-class warm-up adaptation from `input>=8192` to
+   **raw input >4096**, the longest SLO-relevant tier available reliably in this
+   0-8192 C1 workload.
+
+## Sanity block
+
+| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---|
+| C1 cells | 12 | TP=1,MNS=8 | TP=4,MNS=64 | 12 | Exact 3x4 Cartesian surface |
+| Historical probe observations | 92 | 7/cell | 8/cell | 2 per-cell counts | `8*8 + 4*7 = 92` |
+| Materialized trace rows | 1 file | 32,606 | 32,606 | 1 expected | SHA pinned; prompt content not emitted |
+| Raw-length-filtered rows | 1 reconstruction | 17,710 | 17,710 | 1 expected | Raw length 22-8192; non-negative |
+| Capped TP1/TP2 source rows | 1 set | 512 | 512 | 1 expected | Stable time sort, then even downsample |
+| Reconstructed historical counts | 92 | 66 | 600 | 34 | 92/92 match ground truth |
+| C1 peak scores (req/s/GPU) | 12 | 1.283333 | 3.283333 | 8 | Finite, positive, not all identical |
+| v0.20 floor tolerance | 1 vector | 3.283333e-6 | 3.283333e-6 | 1 | Applied before display rounding |
+| Nominal top-4 effective size | 1 cutoff | 5 | 5 | 1 | TP4 three-way cutoff tie retained |
+| Primary peak anchors | 12 | 1/cell | 1/cell | 1 expected | No cell omitted |
+| Realized adjacent/trap anchors | 13 | 1/non-trap | 2/trap | 2 | Predeclared direction rule; total 25 |
+| Primary anchors by TP | 25 | 8 at TP1/TP2 | 9 at TP4 | 2 counts | `8+8+9=25` |
+| Selected requests in anchor candidates | 36 candidates | 121 | 600 | >1 expected | TP1/2 cap-before-threshold; TP4 uncapped |
+| Per-anchor planning cost (H20-hour) | 3 TP classes | 0.0222 | 0.0889 | 3 | Proportional to TP for 80 seconds |
+| Primary planned H20-hours | 1 plan | 2.65 | 2.65 | 1 | Non-negative; below 3.0 |
+| Confirmation/crawl reserve | 1 plan | 0.35 | 0.35 | 1 | Primary plus reserve equals hard cap |
+| GPU work in this protocol turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied |
+
+Checked invariants: `12=3*4`; `92=8*8+4*7`; all 92 local selection counts
+match; cap precedes threshold; arrivals span the fixed 60-second replay; raw
+input length, not retokenized usage, selects TTFT thresholds; completion is
+exactly 128; SLO denominator includes all selected requests; surface scores use
+60 seconds and divide by TP; floor buckets precede ranking; the top-four cutoff
+expands to five; `25=12+11+2`; planned `2.65+0.35=3.0`; ratios and counters have
+their declared domains; and no GPU run, helper, manifest, remote artifact, or
+results file was created in this protocol-only turn.

docs/opprof/phase6-results.md

@@ -0,0 +1,226 @@
+# OpProf Phase 6 dash0 results
+
+Status: **COMPLETE, DATA-VALID A-P6-2 SOLO TIER; ARGMAX/RANKING/TRAP INCONCLUSIVE UNDER THE FROZEN RULES**.
+
+Phase 6 completed all 12 C1 cells on patched vLLM
+`0.24.1.dev3+g668cfb7e2` (`4b253fd`). A-P6-2 makes the serialized solo tier
+authoritative because the v0.20 baseline was solo and the original W2/W3
+confirmations exposed large pass-rate and queue changes after neighboring wave
+clients became idle. The authoritative campaign contains 37 primary anchors,
+29 same-placement confirmations, and 12 valid Layer-1 streams.
+
+The machine-readable result is
+`runs/opprof-phase6/phase6/metrics.json`, SHA-256
+`290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16`.
+Co-located W1-W3 artifacts are preserved beside the solo tree rather than
+overwritten. Public artifacts contain no prompt or generated text.
+
+## Data sanity first
+
+There are **no data-sanity red flags**. All 37 authoritative primary anchors
+reproduce the pinned v0.20 request count exactly; all client invariants pass;
+all 12 solo cells have one graceful footer and agreeing final sidecar; Layer-1
+indices are contiguous with zero drops; every anchor interval is represented;
+and all cold-start compile/capture events precede measurement.
+
+The scientific decision gate is separate from data validity. Eight cells have
+bounded monotonic solo frontiers. TP2/MNS16, TP4/MNS32, and TP4/MNS64 pass the
+highest recorded request set and are right-censored at the pinned-history edge.
+TP4/MNS16 is non-monotonic: L resolves feasible, P resolves infeasible, and U
+resolves feasible. These four cells prohibit a full-surface floor-bucket rank,
+tau-b, and trap decision; no value is imputed beyond the recorded history.
+
+## Launch echo
+
+```text
+A-P6-2_LAUNCH_ECHO host=dash0 engine=vllm-0.24.1.dev3+g668cfb7e2@4b253fd authoritative=solo cells=12 mandatory_anchors=25 crawl=recorded-only confirmations=2of3 prior_h20h=2.291173 new_est_h20h=3.440 cumulative_est_h20h=5.731173 cap_h20h=6.0 est_wall=45-70min
+SOLO_WAVE_ECHO order=TP4/MNS32,TP4/MNS64,TP2/MNS32,TP2/MNS64,TP4/MNS16,TP2/MNS8,TP2/MNS16,TP4/MNS8,TP1/MNS8,TP1/MNS16,TP1/MNS32,TP1/MNS64 placement=one_server_one_client
+FINAL_ACCOUNTING status=complete cells=12 primary=37 confirmations=29 solo_h20h=3.353371 campaign_h20h=5.644544 cap_h20h=6.0
+```
+
+The exact per-cell echoes, GPU placement, trace and ground-truth paths, spend,
+and projection are retained in
+`runs/opprof-phase6/phase6/solo-authoritative/launch-echo.log`.
+
+## Run completion stats
+
+| Item | Result |
+|---|---:|
+| Historical selections reconstructed | 92/92 exact |
+| Authoritative solo cells | 12/12 |
+| Solo primary anchors | 37 |
+| Solo confirmations | 29 |
+| Solo anchor trials | 66 |
+| Solo warm-ups | 12; exactly 16 completions each |
+| Long prompts per warm-up | 2-8 raw inputs `>4096` |
+| Bounded solo frontiers | 8/12 |
+| History-edge right-censored cells | 3 |
+| Non-monotonic cells | 1 |
+| Prior co-located primary/confirm trials | 21 + 3 |
+| Total accepted anchor trials across attempts | 90 |
+| Co-located spend before A-P6-2 | 2.291173 H20-hours |
+| A-P6-2 solo spend | 3.353371 H20-hours |
+| Total campaign spend | **5.644544 H20-hours** |
+| Artifact footprint | 467 MiB, below 3 GiB |
+
+## Final paired surface
+
+`A*` marks the authoritative A-P6-2 solo tier. Exact values are bounded
+frontiers. `>=` is a recorded-history lower bound. The TP4/MNS16 value is not a
+frontier because its feasibility sequence is non-monotonic.
+
+| Cell | Tier | v0.20 frontier | v0.24 authoritative result | Drift | Status |
+|---|---|---:|---:|---:|---|
+| TP1/MNS8 | A* solo | 2.1000 | **2.3833** | **+13.49%** | bounded up |
+| TP1/MNS16 | A* solo | 2.3500 | 2.3833 | +1.42% | bounded up |
+| TP1/MNS32 | A* solo | 2.2833 | 2.3833 | +4.38% | bounded up |
+| TP1/MNS64 | A* solo | 2.2833 | 2.3833 | +4.38% | bounded up |
+| TP2/MNS8 | A* solo | 2.2750 | 2.2417 | -1.47% | bounded down |
+| TP2/MNS16 | A* solo | 2.2750 | >=2.3000 | >=+1.10% | right-censored history edge |
+| TP2/MNS32 | A* solo | **3.2833** | **3.2583** | -0.76% | bounded down |
+| TP2/MNS64 | A* solo | 3.2583 | **2.3000** | **-29.41%** | bounded down |
+| TP4/MNS8 | A* solo | 1.2833 | 1.3208 | +2.92% | bounded up |
+| TP4/MNS16 | A* solo | 2.4417 | N/E; observed max feasible 2.5000 | N/E | **non-monotonic: L pass, peak fail, U pass** |
+| TP4/MNS32 | A* solo | 2.4417 | >=2.5000 | >=+2.39% | right-censored history edge |
+| TP4/MNS64 | A* solo | 2.4417 | >=2.5000 | >=+2.39% | right-censored history edge |
+
+The solo tier validates two material frontier changes under the predeclared 5%
+threshold: TP1/MNS8 improves by 13.49%, while TP2/MNS64 declines by 29.41%.
+The previous TP2/MNS64 decline is now quotable because both its passing 2.3000
+anchor and failing 3.2583 anchor were repeated solo.
+
+## Frozen decision-rule verdicts
+
+- **ARGMAX: INCONCLUSIVE.** TP2/MNS32 is the highest bounded observed frontier
+  at 3.2583 req/s/GPU, but the frozen rule requires a bounded 12-cell subset.
+  TP2/MNS16 and TP4/MNS32/64 terminate at passing history-edge anchors, and
+  TP4/MNS16 has no coherent frontier. The top floor bucket is therefore not
+  defined for all contenders.
+- **RANKING: INCONCLUSIVE.** Only 8/12 frontiers are bounded. Per protocol,
+  Kendall tau-b is **not evaluable** (`tau_b=null`) rather than computed from
+  censored lower bounds. The `tau-b>=0.8` survival rule cannot be applied.
+- **TRAP: INCONCLUSIVE.** TP4/MNS16 is non-monotonic and its TP4/MNS32 neighbor
+  is right-censored. Neither trap persistence nor escape satisfies the frozen
+  floor-bucket rule.
+
+## Solo versus co-located exact-anchor deltas
+
+The co-located column is the simultaneous-wave primary. The solo column is the
+accepted median after same-placement 2-of-3 adjudication. Delta is percentage
+points. `P/F` are SLO feasible/infeasible at 0.95.
+
+| Cell | Anchor | Co-located primary | Solo authoritative | Delta | State change |
+|---|---:|---:|---:|---:|---|
+| TP1/MNS8 | 0.21875000 | 0.992 | 0.992 | +0.00pp | P->P |
+| TP1/MNS8 | 0.22656250 | 0.206 | 0.992 | **+78.57pp** | F->P |
+| TP1/MNS16 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP1/MNS16 | 0.25000000 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP1/MNS32 | 0.24218750 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP1/MNS32 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP1/MNS64 | 0.24218750 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP1/MNS64 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP2/MNS8 | 0.49218750 | 0.691 | 1.000 | +30.86pp | F->P |
+| TP2/MNS8 | 0.49609375 | 0.092 | 0.581 | +48.90pp | F->F |
+| TP2/MNS16 | 0.49218750 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP2/MNS16 | 0.49609375 | 0.088 | 1.000 | **+91.21pp** | F->P |
+| TP2/MNS32 | 0.75000000 | 0.412 | 1.000 | +58.82pp | F->P |
+| TP2/MNS32 | 0.75390625 | 0.028 | 0.575 | +54.70pp | F->F |
+| TP2/MNS64 | 0.50000000 | 0.460 | 1.000 | +53.99pp | F->P |
+| TP2/MNS64 | 0.75000000 | 0.141 | 0.561 | +42.07pp | F->F |
+| TP4/MNS8 | 0.016055910008 | 1.000 | 1.000 | +0.00pp | P->P |
+| TP4/MNS8 | 0.016591107009 | 0.071 | 1.000 | **+92.86pp** | F->P |
+| TP4/MNS16 | 0.033182214016 | 0.609 | 1.000 | +39.13pp | F->P |
+| TP4/MNS16 | 0.033717411016 | 0.036 | 0.238 | +20.22pp | F->F |
+| TP4/MNS16 | 0.034252608017 | 0.847 | 1.000 | +15.33pp | F->P |
+
+The deltas range from 0 to +92.86pp across 21 exact pairs. Four TP1/MNS16-64
+pairs are unchanged, while several TP2/TP4 and TP1/MNS8 anchors flip. Thus
+co-location validity is **metric- and cell-dependent**, not a universal offset.
+Throughput-only values may co-locate, but SLO-frontier feasibility cannot be
+mixed across placement tiers.
+
+## Layer-1 mechanism notes for material drift
+
+Layer-1 is mechanism context, not a matched v0.20 causal decomposition. P3 P10
+is TP1 at a different workload/output contract. All 37 solo primaries have
+zero preemptions.
+
+| Cell/anchor | State | Waiting mean/max | Decode-B mean | KV mean | `NONE` graph share | Padding |
+|---|---|---:|---:|---:|---:|---:|
+| TP1/MNS8, 0.25 | frontier pass | 0.74 / 8 | 4.25 | 0.0407 | 3.26% | 0.34% |
+| TP1/MNS8, 0.50 | next fail | 11.66 / 27 | 7.19 | 0.0644 | 5.04% | 0.02% |
+| TP2/MNS64, 0.50 | frontier pass | 0.00 / 0 | 3.28 | 0.0084 | 0.00% | 18.88% |
+| TP2/MNS64, 0.75 | old-peak fail, primary | 0.00 / 0 | 18.05 | 0.0448 | 9.47% | 0.52% |
+| TP2/MNS64, 0.75 | old-peak fail, confirm | 0.0002 / 1 | 11.52 | 0.0285 | 4.57% | 1.13% |
+
+For TP1/MNS8, the new frontier remains feasible despite a larger decode batch
+and modest queueing; failure at 0.5 coincides with a 16x waiting-mean increase,
+higher KV usage, and more eager/non-full graph execution. For TP2/MNS64, the
+passing lower anchor has small decode batches and full/piecewise graphs despite
+high padding, whereas both old-peak trials fail with much larger decode batches
+and graph fallback. Zero preemptions and near-zero queueing rule out preemption
+and server waiting as the main TP2/MNS64 explanation, but the unmatched v0.20
+telemetry prevents assigning causality uniquely to graph mode or batch shape.
+
+## Operational findings and full attempt history
+
+1. The initial ownership-monitor false start charged 0.132918 H20-hours without
+   measurement. The repaired W1 charged 0.299393; A-P6-1 later accepted its
+   eight anchors from balanced checkpoint sidecars without rerun.
+2. Co-located W2/W3 completed 13 primary anchors and three confirmations but
+   exposed placement-sensitive SLO results. The 3.0-hour projection stopped W4
+   at 2.291173 total; those values remain indicative and preserved.
+3. A-P6-2 completed W4 and remeasured the full surface solo. Every solo server
+   used `--shutdown-timeout 120`, emitted a graceful footer, and passed Layer-1
+   accounting.
+4. Placement was not the only transient. TP4/MNS32 and TP4/MNS64 first full
+   replays failed at 0.031/0.114 but subsequent identical solo trials passed;
+   TP4/MNS16 produced an order-dependent non-monotonic sequence. The 16-request
+   A-P5-1 gate removes compile/capture cold start but not necessarily the first
+   full-trace transient. The predeclared 2-of-3 rule prevented single-trial
+   conclusions, but a future protocol should use a full-trace burn-in or
+   randomized/reversed anchor order.
+5. Two cleanup checks briefly observed 1-4 MiB at 0% with zero compute PIDs.
+   Memory decayed to 0 MiB within seconds, graceful footers validated, and no
+   rerun was needed. Cleanup now polls for 30 seconds rather than treating
+   transient driver bookkeeping as a failed experiment.
+
+## GPU total
+
+The complete campaign charged **5.6445441643 H20-hours** against the
+user-approved **6.0** cap:
+
+- original co-located attempts: 2.2911728495;
+- A-P6-2 authoritative solo tier: 3.3533713148;
+- remaining headroom: 0.3554558357 H20-hours.
+
+Final dash0 state is 0 MiB and 0% utilization on all eight H20s, with zero
+controller, client, vLLM, EngineCore, or worker processes.
+
+## Sanity block
+
+| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
+|---|---:|---:|---:|---:|---|
+| Historical selection checks | 92 | 66 requests | 600 requests | 34 | 92/92 exact |
+| Solo primary pass rates | 37 | 0.0307 | 1.0000 | 17 | Ratios in `[0,1]`; not identical |
+| Solo selected requests | 37 | 121 | 600 | 18 | Exact v0.20 count per anchor |
+| Layer-1 steps per primary | 37 | 343 | 12,103 | 37 | Contiguous; non-negative; zero drops |
+| Layer-1 records per cell | 12 | 14,174 | 58,725 | 12 | Footer/sidecar balanced |
+| Solo preemptions | 37 | 0 | 0 | 1 expected | Zero throughout |
+| Reported v0.24 frontier values/bounds | 11 | 1.3208 | 3.2583 | 6 | TP4/MNS16 omitted as non-monotonic |
+| Frontier drift values/bounds | 12 | -29.41% | +13.49% | 9 | 11 finite; one non-monotonic missing |
+| Bounded/censored cells | 12 | 8 bounded | 4 unbounded | 2 | Censoring never imputed |
+| Exact solo/co-located deltas | 21 | +0.00pp | +92.86pp | 13 | Same request set and anchor |
+| Solo confirmation pass rates | 29 | 0.4130 | 1.0000 | 10 | Frozen 2-of-3 applied |
+| Solo cell H20-hours | 12 | 0.0845 | 0.5534 | 12 | Non-negative; serialized placement |
+| Total campaign H20-hours | 1 | 5.644544 | 5.644544 | 1 | Below 6.0 hard cap |
+| Final GPU memory/utilization | 8 | 0 MiB / 0% | 0 MiB / 0% | 1 expected | Zero compute processes |
+
+Checked invariants: exact cap-before-threshold selection; raw-length SLO
+evaluation; exact 128-token completions or counted failures; nondecreasing
+arrivals; all selected outcomes represented in the denominator; one solo
+server/client at a time; compile/capture outside measured intervals; 12/12
+graceful footer balances; non-negative counters; pass ratios in range; no prompt
+text in public artifacts; complete attempt-history retention; authoritative
+tier separation; explicit censoring/non-monotonicity; tau-b suppression when
+the full surface is unbounded; hard-cap compliance; and complete GPU cleanup.

docs/opprof_campaign_state.md

@@ -0,0 +1,80 @@
+# OpProf campaign state — pattern-conditioned operator profiling (vLLM 0.24.0 / Qwen3-30B-A3B / H20)
+
+Started 2026-07-11. Orchestrator: Claude (session a42d23fb). Workers: codex via companion.
+Discipline inherited from SimFid campaign (replayserve/docs/simfid_campaign_state.md):
+pre-registered acceptance gates per phase, codex implement + independent review,
+orchestrator verifies, echo-before-expensive-action, GPU actions logged.
+
+## User decisions (2026-07-11)
+- vLLM pinned to **0.24.0** (latest community); record commit + sha256 at clone time.
+- GPU: **dash0 8×H20, approved for the whole campaign** (Phases 2-4). Orchestrator
+  still echoes load/duration before each launch. Single H20 sufficient for TP1.
+- Observation patch: **dual-layer** — always-on lightweight per-iteration composition
+  telemetry (<3% overhead budget) + sampled heavy torch.profiler/CUPTI windows.
+- L-C-A is REFERENCE ONLY in this campaign — pattern axes defined operationally
+  (input-len dist / output len / arrival shape / prefix sharing), no L-C-A dependency.
+
+## Phase plan (pre-registered)
+- P0 recon (local): clone v0.24.0, inventory built-in observability, patch design doc.
+  Gate: USER reviews patch design before P1 implementation.
+- P1 patch dev + no-GPU tests (local). Gate: strict review + orchestrator verify.
+- P2 single-H20 smoke on dash0: artifacts complete; measured overhead (on/off same
+  load) within budget; op-time sums ≈ iteration time; composition matches request-level
+  ground truth. Hard gate before P3.
+- P3 pattern matrix (~8-12 patterns × 2-3 configs, short replays, dash0): per-pattern
+  operator time breakdown + composition distributions + waste accounting
+  (padding% / graph-miss / ragged imbalance / mixed-batch interference).
+  Hypothesis go/no-go: does operator bottleneck ranking change across patterns?
+- P4 optimization proposal (ranked, measured bounds) + one cheap config-level
+  closed-loop validation (e.g., cudagraph capture sizes vs measured B distribution).
+- Non-goals: no new kernels; no L-C-A ablation (separate track); no production clusters.
+
+## Artifact layout
+- vLLM source: /home/gahow/phd/vllm-v0.24.0 (clone, pinned)
+- Patches: aituner patches/vllm-0.24.0-opprof/ (patch files + apply script + no-GPU tests)
+- Docs: aituner docs/opprof/ (phase0-recon.md, patch-design.md, later phase reports)
+
+## Job log
+- P0 recon dispatched: task-mrg3nm1f-spyl85 (2026-07-11T08:25:42Z). Scope: clone v0.24.0 to ~/phd/vllm-v0.24.0, observability inventory with file:line, dual-layer patch design doc. Gate: user reviews design before P1.
+- P0 ACCEPTED (task-mrg3nm1f, 34m54s): pin ee0da84a=v0.24.0 verified; VLLM_TORCH_PROFILER_DIR-absent claim verified; profiler moved to ProfilerConfig + entrypoints/serve/profile/api_router.py (verified); design doc 421 lines. USER APPROVED design with 5 delegated decisions (JSONL+msgspec+8192 queue; expert loads L2-only; 2+8 profiler window; 3% gate on 95% CI upper bound; reject --disable-log-stats) + checkpoint decision: community BF16, TP1 primary, TP2/4 counterpoints.
+- P1 dispatching: implement per approved design; branch in ~/phd/vllm-v0.24.0 + exported patches/vllm-0.24.0-opprof/ in aituner; no-GPU tests must pass locally (opprof core import-light).
+- P1 completed (task-mrg588j2, 26m36s): branch opprof tip c60e7eeb, +560/-1 across 5 files (within ±30% of design). Orchestrator pre-checks: pytest 11/11 reproduced; patch series applies cleanly on pristine v0.24.0 worktree and applied tree is byte-identical to branch (diff=0); scheduler/model_runner integration diff reviewed — surgical, matches design hooks. Strict review dispatching.
+- Strict review (task-mrg68puk, 14m59s): FAIL — 0 Blocking / 3 Major / 1 Minor. M1 writer-thread silent death + shutdown deadlock (opprof.py:106/145); M2 apply.sh already-applied bypasses base/dirty checks; M3 foreground raw lists violate no-raw-list hotspot contract (opprof.py:182). Minor: missing zero-token+None-stat test; true Scheduler integration test DEFERRED to P2 smoke gate (no local vllm install). Reviewer verified OK: fail-fast log_stats ValueError, pairing across sync/batch-queue/PP/spec-decode, schedule-time snapshots, DP-unique paths, patch-id equality 3/3. Orchestrator adjudication: ACCEPT all; fix worker dispatching (fresh task — resume-last would hit reviewer thread).
+- P1 fix round CLOSED (task-mrg6tnum, 13m34s): branch tip 668cfb7e (3 clean commits on ee0da84a). Orchestrator closure: 14/14 tests reproduced; M3 raw-lists-gone verified (bisect accumulation); BOTH apply.sh refusal cases independently reproduced (dirty exit=1; wrong-base-with-identical-patches exit=1); M1 writer failure handling inspected (one-time log, liveness check, bounded close, drop visibility). P1 ACCEPTED.
+- ECHO P2 (GPU, dash0): deploy patched vLLM 0.24.0 (python-only patch, VLLM_USE_PRECOMPILED path) on dash0, serve Qwen3-30B-A3B BF16 TP1 on ONE free H20; artifacts smoke + overhead gate (interleaved 5xON/5xOFF identical load, 3% gate on 95% CI upper bound) + one 2+8 Layer-2 profiler window + integration assertions (records==steps, no pending leak, footer accounting). Est 1.5-2.5h wall, single GPU. Deferred review item (real Scheduler integration) is covered by these assertions.
+- P2 dispatched: task-mrg7erq5 (dash0 smoke; ssh dash0 ONLY authorized; hard gates: artifacts/schema/footer/no-pending-leak, overhead 95%CI-upper <=3%, Layer-2 window loadable). Est 1.5-2.5h.
+- P2 first run (task-mrg7erq5, 1h37m): overhead gate FAIL honored (13.11%, CI [6.24,19.53]) — but measurement judged INVALID by orchestrator: OFF-arm spread 29% across identical runs, one pair NEGATIVE overhead, 10s window with cold-start per run, ON-first order confound (run 1 = coldest = ON). Physical bound: per-step cost is bisect+encode+enqueue at ~11 steps/s. Artifacts gate passed except PIECEWISE not exercised by smoke load. 14 remote tests pass; TRITON backend 12/12; cleanup verified. AMENDMENT A-P2-1 (pre-registered before rerun): warmup excluded, >=120s steady-state window, ignore_eos fixed output lengths, ABBA counterbalanced order + discard first pair, host load recorded, recorder microbenchmark as secondary evidence, one PIECEWISE-exercising artifact run. GATE UNCHANGED (3% on 95% CI upper).
+- P2 A-P2-1 rerun (task-mrgdbfju + predecessor, valid measurement): overhead gate REAL FAIL — 4.1816%, CI [3.1364, 4.7117], gate 3%. ON tight (28.18-28.26), OFF (28.80-29.64), pairs 2.07-4.74%. Artifacts PASS incl. PIECEWISE addendum (129 records, accounting balanced). Layer-2 correctly not run. LOCALIZATION PUZZLE: producer microbenchmark 29.1µs/step x 9.7 steps/s ≈ 0.04s/run vs observed ON-OFF gap 5.9s — direct recording cost explains <1% of overhead; structural suspect (capture scan @200 concurrency, writer-thread GIL, queue locking, flush I/O locus). GPU spend so far 117 H20-min. Diagnostic+fix round dispatching (stage-bisection first, minimal fix, gate rerun).
+- Bisection round (task-mrge67ke, 34m9s): recorder pipeline exonerated — all 4 stages ~0.5% vs stage-off; BUT all stages kept VLLM_OPPROF_DIR set → gpu_model_runner CUDAGraphStat construction/propagation path active in all; cross-attempt (attempt-2 true-OFF 29.45 vs bisection off 28.39) localizes ~3.6% to that path. Worker honored outside-OpProf stop boundary; temp diffs reversed; 14/14 tests both sides. Orchestrator hypothesis for next round: CUDAGraphStat in ModelRunnerOutput poisons msgspec IPC fast path (pickle fallback for whole output) OR upstream cudagraph_metrics feature carries inherent cost. Next: STATIC analysis first, then 3-trial confirm (true-baseline / upstream-metrics-only / env-set-recorder-off), then targeted fix.
+- P2 CLOSED: ALL GATES PASS (task-mrgfgedp, 2h17m). ROOT CAUSE of the 4.18%: VLLM_OPPROF_DIR entered compile_factors() (envs.py:2011-2105) → hashed into torch.compile cache path (backends.py:1024-1065) → every unique ON dir = cold graph/AOT cache (compile 36.41s vs 6.07s) and ~4% slower artifacts. Recording itself was NEVER the cost (recorder 29.1µs/step; serializer-poisoning hypothesis WRONG — TP1 in-process executor, msgspec handles CUDAGraphStat natively). 3-trial confirmation: baseline 29.72 / upstream-metrics-only 29.64 (0.25%) / env-set 28.45 (4.25%). Fix: +1 production line (ignore-list) + tests; tip bbfa717; 5-patch series; 15/15 tests x3 environments. FINAL GATE: overhead -0.042%, CI [-0.174, +0.046] — PASS. Layer-2 PASS (16.4MB Kineto, 2+8 window, 790,843 events, 7,367 kernel events; active-window perturbation 51.3% — confirms sampled-only design). Artifacts PASS (PIECEWISE 129/139). Orchestrator verified: fix commits, ignore-list diff, 15/15 local rerun, 5 patches. Caveat noted: attempt-4 JSONLs footerless (simultaneous shutdown) — excluded from accounting. UPSTREAM FINDING worth reporting: any per-run-unique env var hashed into compile factors silently causes cold-compile + slower artifacts.
+- P3 dispatching: protocol-first (worker drafts docs/opprof/phase3-protocol.md, STOPS for orchestrator review before execution).
+- USER DIRECTIVE (2026-07-12): parallelize P3 across 8xH20, one experiment per GPU (~8x speedup). Orchestrator caveat: no GPU contention but shared host CPU/memory-bandwidth (API server/tokenizer/scheduler/bench are host-side; P2 showed host-side effects can fabricate %-level differences; saturation-point calibration depends on absolute throughput). P3 protocol must include: pre-registered co-location validity check (same cell solo vs alongside 7 neighbors; throughput + op-share deltas <2-3% to authorize 8-way; fallback 4-way + revalidate), CPU affinity pinning per server/client pair, and host load recording per run.
+- P3 protocol (task-mrh6vfqp, 27m13s, 835 lines): ACCEPTED with orchestrator amendments. Dispositions on 6 open decisions: (1) fixed-duration client build+test APPROVED; (2) P10 private-trace transfer to dash0 CPFS wjh APPROVED (sampling_u<=0.125, input<=32768, output cap 256; no prompt text in artifacts; data returns to same org cluster it came from); (3) MNS{64,1024}xMBT{2048,8192} sparse factorial APPROVED; (4) P10/C00 sole TP2 counterpoint APPROVED; (5) 60% load / burst-8 / 240s / 4 windows APPROVED; (6) kernel mapping + 70% classifiability gate + thresholds APPROVED. AMENDMENT A-P3-1 (user directive): 8-way GPU parallelism, one cell per GPU; pre-registered co-location validity check (same cell solo vs with 7 neighbors, throughput AND op-share deltas <3% to authorize; fallback 4-way + revalidate); CPU affinity pinning (disjoint core sets per server+client); saturation calibration under the SAME co-location regime as measurement; host load + clocks per run; revised wall estimate ~1.5-2h. AMENDMENT A-P3-2 (E2 lesson): execution via detached resumable controller script on dash0 (setsid, --resume, state file) so runs survive codex turn deaths.
+- P3 E-a ACCEPTED (task-mrh7wd5x, 1h27m, 3.96 H20-h): co-location gate → 8-way FAIL (throughput Δ=0.000% but op-share shifts up to 4.2pp > 3% threshold), 4-way PASS (shares ≤0.075pp) — AUTHORIZED 4-WAY. Client+controller 12/12 no-GPU tests; provenance hashed. P10 transfer verified (4,011 rows, tokenizer parity 4011/4011 exact, 0600/0700 perms, no prompt text public). Orchestrator decisions: (1) 4-way verdict accepted despite validation-stream footer absence (shares from Kineto, unaffected); (2) shutdown-fix (API-parent-first) GPU quick-verify REQUIRED before E-b matrix; (3) A-P3-3: per-class drain budget — 240s for output-512 burst cells (P04-class), 120s others; (4) A-P3-4: P3 hard stop raised 8→16 H20-hours (user blanket dash0 approval; E-a consumed 3.96).
+- E-b step1 STOP (task-mrhb2dxw, 10m42s): shutdown-fix verification FAILED hard footer gate — vLLM 0.24.0 API shutdown selected mode=abort timeout=0s and SIGTERMed EngineCore before scheduler/opprof teardown; finally-footer never runs. Run itself healthy (5268 reqs, 43.9 req/s, 1701 records, 0 drops, step range contiguous). Matrix NOT launched (correct). Orchestrator decision: footer must not depend on graceful shutdown — (a) 10-min static check for an official graceful path in 0.24.0; if unreliable, (b) checkpoint-footer sidecar in opprof.py (atomic write each flush cadence; accounting gate accepts sidecar when stream footer absent; bounded loss = last flush interval), new commit + re-export + tests, GPU re-verify incl. abort-kill case, then matrix.
+- E-b round 2 (task-mrhbhj9j, ~1h5m): sidecar fix DONE (commit f8b68f24, +195/-7, patch tip 23450fb2, 7 patches, 18/18+12/12 tests; graceful path found: vllm serve --shutdown-timeout N; dual GPU verification PASS incl. SIGKILL with 24ms checkpoint sidecar balancing). Matrix launched, STOPPED at pre-registered drain gate: P10/C01 saturation drain 288.6s > 120s budget (clean window + accounting VALID; drain is post-measurement hang-watchdog only; 0.74 req/s saturated 32k-context queue drains slowly by nature). 3/4 first-wave runs PASS with plausible ordering. GPU 5.47/16. AMENDMENT A-P3-5: P10-class drain budget 600s; drain violations = quarantine-run-and-continue (stop only if >20% of runs violate); P10/C01 existing data re-adjudicated under amended rule if accounting balanced.
+- NETWORK OUTAGE (~19:2xZ 2026-07-12 onward): dash0 AND dash1 unreachable from local (pre-auth timeout) — ingress/network fault, not host failure. Detached matrix controller on dash0 unaffected by design (A-P3-2); worker holds 1/min read-only reachability probe with retained artifact hashes for post-recovery integrity check. On reconnect: verify controller state (expect matrix advanced or complete), resume worker thread if its turn died.
+- E-b round 3 blocked cleanly on ssh outage (task-mrhdu37w): resumable handoff at runs/opprof-phase3/phase3/access-blocker-20260712.json; last durable 8/52 accepted, 0 quarantine, 0 clean-failures, 6.82 H20-h; controller PID 2237019 untouched, likely still advancing locally. Also delivered mid-flight: P03 profile-control starvation client fix (13/13 tests) + frozen analyzer analyze_phase3.py (4/4 tests). Reconnection watcher armed.
+- NETWORK RESTORED (~21:1xZ per user). Controller advanced 8→12/52 during outage then FAILED ~2.2h ago (idle since). GPU 8.13/16. E-b worker resumed (task-mrhkbq8s): integrity check vs retained hashes → diagnose controller failure → minimal fix → --resume matrix (40 runs remain) → frozen analysis.
+- E-b round 4 (task-mrhkbq8s, ~2h): 40/52 accepted (20/24 cells), 0 clean-window failures, 542,350 L1 records, 72 L2 traces, GPU 13.31/16. Two aux-gate fixes shipped (+58/-11, 15/15+4/4 tests): profiler-control connection, clean-window failure boundary (post-clean disconnects wrongly counted), 3-sample GPU preflight. FINAL BLOCKER: P10/TP2 warm-up gate demands 32/32 completions; long-context prompts cannot finish in budget (15/32, 21/32) though clean windows are valid. Same failure class as drain budget: auxiliary gate miscalibrated for long-context pattern. AMENDMENT A-P3-6: P10-class warm-up gate = 32 completions OR (>=16 completions AND Layer-1-verified throughput stabilization over trailing warm-up steps), with mandatory post-hoc stability evidence in the report. Finish remaining cells within 16h cap; if cap hits, stop and analyze with documented gaps.
+- E-b round 5 (task-mrhnw1rm, 23m46s): P10/TP2 FAILED frozen A-P3-6 stabilization on fresh data (drift 36.76% vs 10%; bins 11/10/16 vs 16) — the cell genuinely does not stabilize; clean window itself valid. Worker correctly refused to relax. GPU 14.03/16. ORCHESTRATOR DECISION: freeze matrix at 40/52 runs / 20/24 cells; abandon remaining 12 runs. AMENDMENT A-P3-7: per-contrast evaluation — each frozen contrast evaluates iff both cells complete; missing → NOT EVALUABLE (never imputed); H1a/H1b use existential logic on completed cells (CONFIRM possible, REFUTE NOT possible at 20/24 — must be stated explicitly). P10/TP2 non-stabilization recorded as a pattern-conditioned operational FINDING.
+- PHASE 3 FINAL (task-mrhos386, 19m37s): **H1a INCONCLUSIVE / H1b PASS (5 of 6 evaluable contrasts) / compound PARTIAL** under A-P3-7. H1b effects large + Holm-corrected p≈0: P10 real trace R64 +44.79pp vs both long rectangular controls (efficiency −14.3%/−44.7%); P09 production mix R64 +39.62pp, padding +6.67pp (−8.3%); P06 bimodal burst R64 +23.0/+35.4pp (−11.6%/−22.8%). H1a inconclusive because only 1/9 patterns Layer-2 windows passed representativeness gates (descriptive: P06/P10 flip attention-led↔MoE-GEMM-led across load points). Orchestrator verified metrics.json per-contrast records match report exactly. GPU final 14.03/16 (1.97 unused). P4 dispatching.
+
+## Phase 5 — mechanism-decomposition ablations (USER-APPROVED 2026-07-12)
+- Motivation (user challenge): the sign of H1b is obvious; the contribution is the mechanism LEDGER. vLLM does no request-level padding (continuous batching); measured bucket padding (+5.4-6.7pp) explains ~1/6 of the 14-45% E_token gap; the remaining ~30pp attribution (ragged-attention SM imbalance / cudagraph bucket mismatch / chunked-prefill mix interference / MoE routing skew / scheduler batch-variance) is unmeasured in the literature.
+- Design skeleton (protocol to formalize): controlled ablations each isolating ONE mechanism on P10 (primary) + P09/P06 (secondary): (i) length-sorted/binned replay [kills intra-cohort raggedness, keeps content+totals]; (ii) capture sizes matched to measured decode-B distribution [kills bucket mismatch; doubles as the config-tier deliverable]; (iii) arrival-shape ablation [steady vs burst, same lengths]; (iv) prefix-caching on/off [C structure]. Accounting must include an explicit interaction/residual term — shares are NOT forced to sum to 100%.
+- Metric: E_token under the P3 discipline (C00-TP1, rho=0.60, 240s clean windows). New GPU cap: +6 H20-hours for P5 (P3 cap closed at 14.03/16).
+- Gates: protocol-first (orchestrator review before any GPU run); P4 acceptance re-scoped — speculative recovery claims in P4 must defer to P5 measured shares.
+- P5 protocol (task-mrhq3ud3, 13m49s): ACCEPTED. Data-grounded specifics verified by worker from P3 raw Layer-1: P10 decode-B support is {1..7} (17,941 pure-decode steps) → A2 capture sizes {3,5,6,7}; A1 = 32-request reorder blocks (R16 0.6417→0.4734, −26.2% rel), 142-request slice @0.4725 req/s, 64s fairness cap. Orchestrator dispositions on 5 open decisions: (1) BLOCKING → approve recorded-arrival BRIDGE ledger (production-faithful estimand; explicit not-literal-P3-decomposition limitation); (2) dual P03/P04 control ledgers, dominant must hold under both — approved; (3) A1 parameters — approved; (4) 3 replicates/arm, P3-control reuse behind 3% bridge gate, optional tier within 6.0 H20-h — approved; (5) Layer-1-only primary, routed-expert telemetry analysis-only no causal share — approved. Execution authorized.
+- P5 wave 1 (detached exec, 30min, 0.65 H20-h): HARD STOP correct — all 4 attempted arms failed frozen warm-up stabilization (drift 190.6-216.5% recorded arms, 13.2% uniform arm, gate 10%; completions 25-28/32). ROOT CAUSE: semantic mismatch — throughput-drift gate designed for saturation/steady loads is wrong for rate-following arms whose throughput follows a non-stationary recorded arrival BY DESIGN. Purpose of warm-up = no cold-start contamination, not stationarity. Bonus signal: 190% vs 13% drift gap is itself direct arrival-mechanism evidence. AMENDMENT A-P5-1: rate-following arms use cold-start-artifact gates (all compile/capture events before clean window per Layer-1 graph-mode series + >=16 warm-up completions incl >=1 long-context + zero first-occurrence capture events inside clean window); drift gate retained for saturation arms only. Budget 5.35 remaining.
+- P5 round 2 (detached, ~70min, 2.44/6.0 H20-h): 15/15 primary valid under A-P5-1, bridge PASS (0.269%). OFFICIAL ledger INCONCLUSIVE per frozen rules (only A1-under-P03 evaluable: share 0.038, CI [-0.20,0.23]; A2/A3/A4 manipulation-failed; P04 denominator unstable 45% nonpositive draws). REAL FINDINGS: (1) P3 P10-vs-P04 gap was largely a MATERIALIZATION ARTIFACT — P5 recorded-arrival base E=3.014 ≈ P04 control 3.055 (gap 0.041 CI spans 0), while uniformizing arrival (as P3 did) drops E to 2.626 (−12.9%, raw p=0.014, Holm 0.055): burstiness HELPS at low rate via batch formation (decode-B CV 0.87→0.66 under uniform but waiting CV rises); P5 A3 reproduces P3 base within 0.27% — bridge design caught our own artifact. (2) All four intuitive mechanisms ≈0 at rho=0.60: raggedness 3.8% n.s., capture-fix +1.6% n.s. (P4 independent validation +0.18%, padding bound matched within 0.002pp), prefix ~0. Remaining P10-vs-P03 gap (~36%) explained by NONE — workload physics, not recoverable waste at this regime. (3) P4 ranked list: #1 prefix affinity +82.14% ceiling (P08 vs P07), #4 pattern-specific MNS64 has a −24.27% P01 counterexample (sign flips by pattern). OpProf campaign phases P0-P5 all complete; ~16.5 H20-h total.
+
+## Phase 6 — cross-version churn quantification (USER-APPROVED 2026-07-13 "请你测试一下")
+- Question: does the vLLM 0.20.0 C1 12-cell TP×MNS optimum/ranking survive on 0.24.0? Paired-surface churn evidence for the paper motivation (matrix × real-eval cost × churn).
+- Ground truth: C1 recovered stores (verified): TP1 [2.10,2.35,2.283,2.283], TP2 [2.275,2.275,3.283,3.258], TP4 [1.283,2.442,2.442,2.442] req/s/GPU; best TP2/MNS32=3.2833; trap TP4/MNS16; per-cell recorded anchors + request counts in replayserve docs/assets/simfid_s2r/ground_truth.json; workload = materialized chat_w20260311_1000 (local + dash0 CPFS), output override 128, scale 0.1, pass>=0.95, stepped TTFT 2/4/6s, TPOT 50ms; caps 512 TP1/TP2, uncapped TP4.
+- Budget: 3.0 H20-hours hard cap → anchor-subset design required (TP4 cells cost 4 GPU-h per wall-hour). Covariate to note: C1 ran on dash1, P6 on dash0 (same H20 class).
+- Gate: protocol-first, orchestrator review before GPU.
+- P6 protocol ACCEPTED: 92/92 historical anchor counts reproduced locally pre-GPU (comparability verified); adaptive 25-anchor design (12 peak + 11 adjacent + 2 trap-bracket), 2.65 primary + 0.35 confirmation reserve = 3.0 cap exactly. Decision rules: feasibility flip / >5% drift / floor-bucket argmax / tau-b>=0.8 ranking survival / trap escape rule / 2-of-3 confirmation. All 5 open decisions APPROVED incl. upgrade-path estimand (resolved defaults = churn; dash1→dash0 + colocation = stated limitations) and warmup long-tier adaptation raw>4096. Execution dispatched.
+- P6 W1 hard stop (0.43 H20-h spent): 8 replays REJECTED for missing in-stream footers (wrong shutdown path) + redo projection 3.03 > 3.0 cap. Orchestrator diagnosis: P6 validator failed to apply the P5-ACCEPTED sidecar-footer accounting rule (records == sidecar counters within one flush interval). AMENDMENT A-P6-1: (a) sidecar footers are valid accounting per P5 rule — re-adjudicate W1 artifacts WITHOUT rerun if sidecars balance; (b) controller must use the graceful path (vllm serve --shutdown-timeout, found in P5) for subsequent waves; (c) IF W1 salvage fails, cap raised 3.0→3.5 as fallback (worker recommendation, within user posture).
+- P6 r2 partial (2.29/3.0 H20-h, hard stop before W4): 21/25 anchors, 10/12 cells. Signals: TP2/MNS64 >=29.4% DOWN (left-censored), TP1 cells right-censored UP, TP2 family down — surface shape moved opposite directions. CONFOUND (worker confirmation runs): co-location flips SLO-frontier pass rates (0.41→1.00, 0.03→0.96 solo; waiting 8.5→0.35) — co-location valid for throughput/op-shares (P3) but NOT for pass-rate cliffs; 0.20 baseline was solo. USER DECISION: cap raised to 6.0; A-P6-2 = frontier anchors SOLO placement; finish W4 + solo re-confirm key cells.
+- P6 FINAL (A-P6-2 solo tier complete, 5.64/6.0 H20-h, 66 authoritative trials, 12/12 cells): formal verdicts INCONCLUSIVE per frozen letter (4 cells censored/non-monotonic prevent bounded 12-cell surface; tau_b=null, never manufactured). SUBSTANTIVE PAIRED SURFACE (all solo-authoritative): TP2/MNS64 −29.41% CONFIRMED (3.2583→2.3000 bounded; mechanism: decode-B 11.5-18 + 4.6-9.5% NONE-graph at old anchor); TP1/MNS8 +13.49% and TP1 plateau converges to 2.3833; old best TP2/MNS32 −0.76% (held, highest bounded); TP4 family right-censored UP >=2.50; TP4/MNS16 frontier became NON-MONOTONIC in load. CO-LOCATION DELTA TABLE: up to +92.86pp pass-rate flips (21 exact-request anchor pairs) — co-location validity is metric-dependent (standalone methodological finding). Orchestrator verified −0.2941 drift in metrics.json. Campaign total incl. P6: ~22.2 H20-h.

docs/simulator-fidelity-frontier-20260711.md

@@ -0,0 +1,134 @@
+# Simulator-based tuning 保真度实验总结（Frontier / H20）— 供 review
+
+日期：2026-07-11。目的：判断现有实验数据是否**严格**支撑"simulator-based
+tuning 会导致错误的 rank 排序和最终性能 gap"这一论文主张。全部原始材料在
+`~/phd/replayserve`（见文末指针）；本文只做总结，不新增任何数字。
+
+## 一句话结论
+
+在给足模拟器一切有利条件（同模型同硬件的 H20 算子 profile、逐 token 精确的
+真实 workload、在独立数据上冻结的逐 TP 吞吐校准）之后，Frontier 在预注册判定
+规则下仍未通过：在真实调优器实测过的 12-cell TP×MNS 面上，模拟器选出的配置
+比真实最优差 **30.46%**（top-1 real-evaluated regret），关键交互关系只复现
+3/6。**若按模拟器结果部署，你会选 TP1/MNS64（真实 2.283 req/s/GPU），放弃
+TP2/MNS32（真实 3.283）——这就是"最终 gap"的操作性含义。**
+
+## 实验设置（三个不可辩驳性设计）
+
+- **Ground truth 是真实调优研究本身**：C1 交互研究（dash1 恢复的 store，
+  vLLM 0.20.0，H20，`chat_w20260311_1000` 窗口，scale 0.1），12 个
+  TP{1,2,4}×MNS{8,16,32,64} cell 的 SLO-feasible peak（pass≥0.95，阶梯
+  TTFT 2/4/6s，TPOT 50ms）。每个数字从原始 state.json/engine.log 独立复算。
+- **EXACT workload**：materialized JSONL（32,606 行，含真实 prompt 文本）
+  逐 token 重建——tokenize 后逐行断言等于 `input_length`（17,710/17,710 全
+  过），chat 模板开销恒为 +8 token 并分解到具体 token ID，block-16 前缀 hash
+  按 vLLM 0.20.0 的 `hash_block_tokens` 源码（vendored、逐字节比对）计算。
+  每个 cell 用它自己实测过的锚点（92 个锚点，选中请求数 92/92 与真实记录
+  相等），不插值、不补网格。
+- **预注册**：判定规则在任何模拟跑之前冻结于
+  `replayserve/docs/simfid_s2r_protocol.md` §6——(1) 最坏 top-1 regret ≤5%；
+  (2) TP4/MNS16 陷阱 6 条关系全复现；(3) 92 次留一锚点复检全过。四种读法中
+  只有"冻结校准 + 吞吐代理"承载结论，其余为诊断。校准系数
+  a_tp = {TP1: 0.7235, TP2: 0.4681, TP4: 0.3521} 来自独立的 3 配置切片
+  （S2-E），S2-R 数据上不重拟合。
+
+执行：184 次 Frontier CPU run（92 锚点 × 未校准/校准），184/184 通过、
+0 失败，sanity 无红旗。
+
+## 核心证据：12-cell 分数表（req/s/GPU）
+
+| Cell | 真实 SLO peak | 校准后 sim 吞吐 | 校准后 sim SLO |
+|---|---:|---:|---:|
+| tp1_mns8  | 2.100 | 2.173 | 1.717 |
+| tp1_mns16 | 2.350 | 3.242 | 2.383 |
+| tp1_mns32 | 2.283 | 4.297 | 2.383 |
+| tp1_mns64 | 2.283 | **4.357** ← sim 判为全局最优 | 2.383 |
+| tp2_mns8  | 2.275 | 2.039 | 1.742 |
+| tp2_mns16 | 2.275 | 2.244 | 2.300 |
+| tp2_mns32 | **3.283** ← 真实全局最优 | 3.650 | 3.750 |
+| tp2_mns64 | 3.258 | 3.650 | 3.750 |
+| tp4_mns8  | 1.283 | 1.545 | 1.321 |
+| tp4_mns16 | 2.442 | 2.437 | 2.500 |
+| tp4_mns32 | 2.442 | 2.462 | 2.500 |
+| tp4_mns64 | 2.442 | 2.462 | 2.500 |
+
+**失败机理**（不是尺度错，是形状错）：真实 TP1 在 MNS16 后随 SLO 边界饱和
+（2.35→2.28），sim 认为吞吐随 MNS 单调上升（3.24→4.30→4.36）。逐 TP 校准
+已消掉尺度误差，负载响应形状仍然错——来源是排队/尾延迟/调度开销，不是算子
+时间表。佐证：sim TTFT p95 仅为真实的 0.30–0.38，TPOT p95 0.63–0.79
+（S2-E 持出集）。
+
+排序质量：校准吞吐读法 τ-b = 0.448（未校准 0.236），成对方向正确率
+68–73%。远低于可用水平。
+
+## 什么已被严格证明（本文主张的边界内）
+
+1. **在被测面上，simulator-only 的吞吐排序会给出 30.46% 的真实 gap**——
+   端到端、预注册、workload 逐 token 精确、同硬件 profile、校准冻结。链条
+   里没有"我们没给模拟器机会"的空隙。
+2. **未做延迟校准的模拟器做 SLO 判断不可信**：S2-E 直接反例（sim TPOT p95
+   46.43ms vs 真实 71.38ms，横跨 50ms SLO → sim 判可行、真实不可行）；
+   S2-R-b 校准 SLO 读法在 92 锚点上有 21 假可行 / 7 假不可行。
+3. **覆盖缺口独立于精度成立**：MoE EP>1 无 profile 支持、GMU 惰性、
+   CUDA-graph 未接、scheduler-delay 不可表达——这些 knob 模拟器根本无法
+   评估（replayserve/docs/simfid_inventory.md 旋钮矩阵）。
+
+## 什么还没被严格证明（review 时请重点判断这三条）
+
+1. **跨引擎版本 profile**（最大攻击面）：H20 profile 来自 vLLM 0.11.1 时代
+   的对齐工作，ground truth 是 0.20.0。防守方可以说"按 0.20.0 重新 profile
+   就好了"。我们的回应有二：(a) 冻结校准已吸收逐 TP 尺度误差，剩余的是形状
+   误差，其来源（排队/调度/CUDA-graph）不在算子表里；(b) 反身性论证——若
+   每个引擎版本 × 硬件都要重 profile + 重校准，profiling 本身占用同款 GPU
+   跑真实负载，模拟器的成本优势即被churn吃掉。但 (a) 目前是机理论证 +
+   间接证据，不是实验闭环。
+2. **SLO 门控读法的意外成功**：校准 + SLO 门控（预注册为仅诊断）达到
+   regret 0–0.76%、τ-b 0.967、陷阱 6/6。审稿人可主张"你选错了读法"。
+   我们的回应：该读法建立在错误的逐锚点判定上（21 假可行/7 假不可行），
+   正确性来自误差在逐 cell 峰值处的部分抵消，无法保证泛化；且它是事后
+   观察，预注册规则不允许换读法。**但要诚实：这条把可主张的结论从
+   "模拟器必然错排"弱化为"模拟器排序不可信、必须真实验证"。**
+3. **单面、单模型、单硬件、单 workload 窗口**：无跨 workload 复制。且论文
+   审稿人点名的多半是 Vidur，我们测的是 Frontier（同类：算子 profile +
+   事件驱动调度模拟；需论证类代表性或列为 limitation）。
+
+## 建议的论文主张口径（可辩护版本）
+
+> 不主张"simulator 总是错排"，主张：在我们的真实调优器必须绕开局部陷阱的
+> 那个交互面上，一个被给足条件的模拟器（同硬件 profile + 吞吐校准 + 精确
+> workload）在其预注册的最优读法下产生 30% 的部署 gap，且其 SLO 判定在
+> 无延迟校准时存在跨边界的假可行；因此 simulator-only tuning 的结果不经
+> 真实评估不可信——而真实评估的成本控制正是 AITuner 的贡献。混合设计
+> （模拟器粗筛 + 真实终判）是被我们的诊断数据支持的未来方向，且"何时必须
+> 真实评估"仍由相似度度量回答。
+
+## 可选补强（按闭环价值排序）
+
+1. **负载响应形状分析**（零新模拟，复用 184 run + 真实 probe history）：
+   逐 cell 对比归一化吞吐/延迟-锚点曲线。若形状系统性不匹配，则证明任意
+   逐配置尺度校准（= 任意精度的算子 profile）原理上无法恢复排序，直接
+   封死上面第 1 条攻击面的一半。
+2. **vLLM 0.20.0 重 profile**（需 GPU 时间 + 审批）：实验性封死跨版本
+   质疑，但注意这同时演示了 churn 成本，输赢都有叙事价值。
+3. **第二个 workload 窗口**（如 coder 或 2200 slot）：复制性。
+
+## 数据 sanity block
+
+- 12-cell 真实向量：n=12，min/max=1.283/3.283，8 个 distinct 值。
+- 校准 sim 吞吐向量：n=12，min/max=1.545/4.357，10 个 distinct。
+- 执行：n=184，失败 0，请求数 min/max=66/600（34 distinct）。
+- regret=30.456853% 由 1−2.2833/3.2833 独立复算精确一致；a_tp 三值与
+  S2-E 冻结清单一致；所有比率在 [0,1]；无逐配置全同向量。
+- 已知异常（保留未修饰）：真实 TP2/MNS32 与 MNS64 的 pass-rate 非单调，
+  但 0.95 可行性截断有序。
+
+## 材料指针（全部在 ~/phd/replayserve）
+
+- 最终综合报告：`docs/simfid_s3_fidelity_report.md`
+- 12-cell 结果与全部指标：`docs/simfid_s2rb_results.md`、
+  `runs/simfid_s2rb/results/metrics.json`
+- 预注册协议 + 修正案：`docs/simfid_s2r_protocol.md`
+- 吞吐校准与延迟反例：`docs/simfid_s2e_report.md`
+- 数据清单与旋钮覆盖矩阵：`docs/simfid_inventory.md`
+- 全程决策/验收台账：`docs/simfid_campaign_state.md`
+- 恢复的 EXACT workload：`~/phd/aituner/trace_windows/traces/chat_w20260311_1000.jsonl`

docs/telemetry-residual-tuning-roadmap-20260714.md

@@ -0,0 +1,286 @@
+# Telemetry-conditioned residual tuning roadmap
+
+Status: **R0 COMPLETE / FAILED; R1 AND R2 CLOSED FOR THIS MODEL**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Research question and claim boundary
+
+The question is whether a small number of real engine observations can correct
+a simulator's task-specific error over **unmeasured configurations**, and
+whether that correction reduces the real-GPU cost of finding a high
+SLO-goodput serving configuration.
+
+The intended headline claim, if the evidence supports it, is:
+
+> An engine-state-conditioned residual model turns a simulator prediction into
+> a task-specific posterior over unmeasured serving configurations, allowing a
+> sequential tuner to reach near-oracle SLO-goodput with materially fewer
+> H20-hours than simulator-only and outcome-only tuning.
+
+Classification accuracy, simulator-error diagnosis, and telemetry overhead are
+supporting evidence.  None is an end-to-end tuning contribution by itself.
+
+The following method is closed and will not be revived under another name:
+per-candidate five-second accept/reject as the headline contribution.  The P1
+result showed only 1.426% cost reduction in the frozen `k=2` workflow.
+
+## Two models, one evaluation
+
+Both branches use the same legal candidate set, real measurements, task split,
+cost accounting, and acquisition function.
+
+### Simulator-residual branch (primary)
+
+For measured anchor `c_t` and unmeasured candidate `c'`:
+
+```text
+y_hat(c') = y_real(c_t)
+          + [y_sim(c') - y_sim(c_t)]
+          + f(state_real(c_t) - state_sim(c_t), c' - c_t, workload, SLO)
+```
+
+The simulator delta is the prior.  The learned model may correct it only with
+training-supported state/config transitions; uncertainty or distribution shift
+must shrink the correction back toward the simulator prior.
+
+### Telemetry-only branch (mandatory)
+
+```text
+y_hat(c') = y_real(c_t)
+          + g(state_real(c_t), c' - c_t, workload, SLO)
+```
+
+This branch tests whether the simulator is actually necessary.  It does not
+use a hand-authored bottleneck-to-knob rule.
+
+### Search policy
+
+Legal configurations are enumerated independently of telemetry.  A generic
+cost-aware acquisition rule ranks candidates from predicted improvement,
+uncertainty, and measured H20 cost.  The current production harness's
+bottleneck scores, topology-first ordering, and hand-set relief constants are
+not consumed by either branch.  The validator may enforce legality,
+full-config no-repeat, failure accounting, and resource caps only.
+
+## Hypotheses
+
+| ID | Hypothesis | Direct test | Failure meaning |
+|---|---|---|---|
+| H0 | Existing artifacts can express a common, direct-measurement state without heuristic labels. | Engine/simulator extractor coverage and invariants. | Route is not currently implementable. |
+| H1 | Simulator errors are predictable from engine/simulator state discrepancy at measured anchors. | Task-held-out pairwise inversion correction and new-inversion rate. | Telemetry is diagnostic but cannot correct the surface. |
+| H2 | Telemetry alone predicts useful config transitions beyond outcome-only history. | Telemetry-only versus real-outcome-only sequential replay. | Direct telemetry-guided tuning has no independent value. |
+| H3 | Residual correction changes actual tuning decisions and cost. | H20-hours to 95% oracle and regret AUC against the strongest safe baseline. | No system contribution even if H1/H2 prediction metrics improve. |
+
+## Common-state contract
+
+Only directly observed or exactly reconstructed quantities are admitted.
+
+| Quantity | vLLM Layer-1 | Frontier | R0 status |
+|---|---|---|---|
+| Scheduled requests / batch size | Per scheduler step | Existing per-batch metric, disabled in P1 output | Common after CPU replay |
+| Scheduled prefill/decode tokens | Per scheduler step | Existing per-batch metrics | Common after CPU replay |
+| Scheduler/batch rate | Monotonic step timestamps | Batch count / simulated duration | Common after CPU replay |
+| Waiting queue area | Time-weighted queue gauge | Sum of request waiting times | Common aggregate |
+| Running request area | Time-weighted running gauge | Sum of E2E minus waiting time | Common aggregate, semantics audited |
+| Preemption count | Per step | Per request | Common |
+| KV usage/headroom | Exact blocks and ratio | Not in committed output | Engine-only until exact reconstruction exists |
+| CUDA graph mode/padding | Exact per step | Not modeled | Engine-only omitted-mechanism signal |
+| Request TTFT/TPOT/pass rate | Exact real outcomes | Exact simulated request metrics | Common outcome, not state |
+
+Unavailable fields remain null.  They cannot be imputed from a human
+`prefill/decode/queueing` label.
+
+Frontier already contains the required detailed batch and timestamped
+stage-batch ledger output.  P1 disabled it for artifact size.  R0 replays the
+same immutable fixtures with the existing output flags enabled; it does not
+change the simulator model or calibration.
+
+## Data separation
+
+- Phase 6 / `chat_w20260311_1000`: development only.
+- P1 / `chat_w20260312_1000`: development only.
+- R1 / `chat_w20260313_1000`: new development surface.
+- R2: trace windows not used for feature, model, threshold, candidate-space,
+  cutoff, or acquisition decisions.
+- Splits are by complete workload/SLO task.  Anchor- or pair-level random
+  splits are prohibited.
+- Sequential-policy seeds measure algorithmic variability; they are not
+  counted as independent system tasks.
+
+The two existing development tasks have an important limitation: the now-
+available SLO-gated simulator reading already retains the real oracle at its
+top rank/tie.  They therefore cannot establish a positive end-to-end ranking
+claim.  They are used for plumbing, known false-feasible cases, and negative
+evidence.  R1 must be run as an unbiased complete surface, not selected after
+observing simulator success or failure.
+
+## Step-by-step roadmap
+
+### R0.1 — Inventory and roadmap
+
+Deliverables:
+
+- this roadmap;
+- rolling untracked `ONGOING.md`;
+- exact engine/simulator field and artifact inventory.
+
+Gate: every claimed input has an authoritative file path and provenance.
+
+### R0.2 — Common-state plumbing
+
+Deliverables:
+
+- `runs/telemetry-residual/common_state.py`;
+- synthetic correctness tests;
+- one exact P1 Frontier replay with individual batch metrics and the full
+  stage-batch ledger enabled;
+- paired engine/simulator state summary for the same fixture.
+
+Gate:
+
+- replay request count and SLO scorer exactly agree with the committed replay;
+- batch/ledger outputs are non-empty;
+- all counters are non-negative, ratios bounded, times monotonic;
+- no GPU is visible to Frontier;
+- output volume is practical before expanding to twelve replays.
+
+### R0.3 — Development residual/headroom audit
+
+Use all frozen P1 primary fixtures and corresponding engine intervals.  Produce:
+
+- common-state residuals per anchor;
+- simulator-error labels and continuous SLO/goodput residuals;
+- ordered source/target diagnostic that removes both config identities from
+  both roles in every training fold;
+- oracle upper bound for cross-candidate correction;
+- explicit comparison with simulator+outcome and telemetry-only features.
+
+R0 is a feasibility gate, not headline evidence.  Proceed to R1 only if:
+
+1. state features are collected with the measured source anchor, vary across
+   cells, and are available before any target config is evaluated;
+2. at least one known simulator error has a state discrepancy not exposed by
+   the matched external prefix outcome;
+3. a prior-preserving model can correct development errors without introducing
+   a larger number of new errors under regularization sensitivity;
+4. an oracle cross-candidate correction has at least 15% sequential tuning-cost
+   headroom under full startup/warm-up accounting.
+
+### R0 result and decision
+
+R0 completed without a data-validity red flag, but failed condition 3.  The
+decision is **STOP_BEFORE_R1**; no H20 job was launched for this route.
+
+- All 12 detailed Frontier CPU replays exactly reproduced their committed SLO
+  scorers.  Runtime was 23.943--54.786 seconds per replay, detailed artifacts
+  were 4.12--13.53 MB, CUDA visibility was empty, and there were zero failures.
+- The paired surface contains 12 real/sim anchors, two known simulator
+  false-feasible anchors, and 120 legal cross-config ordered transitions.  A
+  fold removes both the source and target TP/MNS identity from source and
+  target roles; the two offered-load anchors remain part of the same task.
+- Raw Frontier feasibility is 83.33% on the repeated transition view.  The
+  structurally correct hybrid model uses
+  `r_target = r_source + delta_r`; the direct model uses
+  `y_target = y_source + delta_y` and never reads simulator fields.
+- Direct telemetry is not robust relative to real-outcome-only: its accuracy
+  delta over L2 `{0.1,1,10,100}` is `{-0.83,+1.67,0,-4.17}` percentage points,
+  and its best absolute accuracy is 54.17%, below the raw simulator's 83.33%.
+- Hybrid telemetry raises classification accuracy over the corresponding
+  simulator+outcome transition regression by 1.67--4.17 percentage points,
+  but worsens pass-rate RMSE by 0.141--0.201 and MAE by 0.084--0.125.  Its full
+  correction reaches only 46.67--53.33% absolute accuracy.
+- Across 24 nonzero `(L2, raw-simulator-prior weight)` combinations, no model
+  both corrects an existing simulator error without more new errors and avoids
+  worsening RMSE/MAE.  Whenever a correction fixes at least one error, it
+  corrupts at least 11 previously correct transitions.
+- A perfect correction could skip the frozen simulator rank-2 real final and
+  save 0.043469 H20-hours: 15.45% of the prospective online `k=2` cost, or
+  14.40% when the prior failed launch is charged.  On this development task the
+  simulator top-1 already is the real oracle with zero regret, so headroom
+  versus the observed-safe top-1 baseline is 0%.
+
+The result does not prove that engine telemetry is useless.  It shows that the
+current one-task anchor-transition evidence cannot support either a safe
+simulator-residual tuner or a simulator-free telemetry tuner.  A larger model
+or an R1 run would add capacity/data after a failed gate and is therefore not
+authorized under this roadmap.
+
+### R1 — New development surface
+
+Status: **NOT LAUNCHED; CLOSED BY R0**.
+
+Frozen starting setup:
+
+- host: dash0, eight NVIDIA H20 GPUs;
+- cells run solo; no co-location for SLO verdicts;
+- patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16;
+- trace: `chat_w20260313_1000`;
+- output tokens: exactly 128;
+- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, pass rate at least 0.95;
+- config surface: TP `{1,2,4}` × MNS `{8,16,32,64}`;
+- hard campaign cap: 4 H20-hours.
+
+The load ladder, repetitions, randomized order, exact commands, expected wall
+time, and artifact paths are frozen only after R0.  A resolved echo is required
+before launch.
+
+R1 passes only if a frozen sequential replay shows at least 15% E2E H20-hour
+headroom over the strongest safe baseline with final regret at most 5%.  R1 is
+development evidence and cannot be reported as the held-out result.
+
+### R2 — Held-out sequential tuning
+
+Status: **NOT LAUNCHED; CLOSED BY R0**.
+
+Required baselines:
+
+1. random search;
+2. real-outcome-only Bayesian/sequential search;
+3. Frontier ranking plus real top-k final;
+4. simulator plus real-outcome residual;
+5. telemetry-only transition tuner;
+6. simulator plus telemetry residual tuner;
+7. complete real surface as oracle, not as a cost competitor.
+
+Primary metric: end-to-end H20-hours to first reach 95% of the real full-surface
+SLO-goodput oracle.  Secondary metrics are cost-normalized regret AUC, final
+regret at fixed budgets, oracle false-prune, wall time, and per-task regressions.
+
+The route is successful only if the winning telemetry method reduces the
+primary cost by at least 20% versus the strongest safe baseline and ends within
+5% regret on every headline task.  If hybrid beats telemetry-only by at least
+10%, simulator residual correction is the primary method.  If telemetry-only
+is within 5% or better, the simulator dependency is removed.  If neither clears
+the contribution bar, the route is closed and telemetry remains a diagnostic
+facility only.
+
+## Cost discipline
+
+- R0 simulator work is CPU-only and must set empty CUDA visibility.
+- R1 cannot exceed 4 H20-hours.
+- R2 receives no budget until R1 passes.
+- Startup, warm-up, burn-in, failed launches, real probes, continuation, and
+  final validation are charged.  Benchmark-only annotation repeats are
+  reported separately and cannot disappear from campaign accounting.
+
+## Final R0 sanity block
+
+| Data | n | Min | Max | Distinct | Checked invariant |
+|---|---:|---:|---:|---:|---|
+| Phase 6 cells | 12 | TP1/MNS8 | TP4/MNS64 | 12 | Surface not identical; solo SLO tier authoritative |
+| Phase 6 Layer-1 primary steps | 37 streams | 343 | 12,103 | 37 | Contiguous; zero drops |
+| P1 primary anchors | 12 | infeasible | feasible | 2 labels | 7 feasible / 5 infeasible |
+| P1 Frontier runtime | 12 | 24.093 s | 54.575 s | 12 | CPU-only; zero failures |
+| Detailed Frontier replay runtime | 12 | 23.943 s | 54.786 s | 12 | Exact committed scorers; CUDA hidden |
+| Detailed artifact bytes | 12 | 4,123,724 | 13,527,776 | 12 | Non-negative; practical CPU replay size |
+| Cross-config transitions | 120 | real pass 0.1067 | real pass 1.0 | 6 outcomes | Both endpoint config identities held out |
+| State residual vectors | 12 | 16 fields | 16 fields | 12 vectors | Finite; no missing common field |
+| R0 E2E cost values | 4 | 0.237914 | 0.301935 H20-h | 4 | Non-negative; `k=1/2`, online/conservative |
+
+Checked invariants: non-negative counts and costs; pass rates in `[0,1]`;
+simulator results not all identical; exact request count/hash agreement; Layer-1
+step continuity and zero drops; no co-resident SLO measurements; no calibration
+or evaluation split reuse for a future headline claim.  No current red flag
+invalidates R0 plumbing.  The R0 tuning gate itself failed because safe
+prior-preserving correction was absent.

docs/tuning-core-challenges-cost-audit-20260715.md

@@ -0,0 +1,391 @@
+# AITuner tuning：核心挑战、统一成本口径与研究路线
+
+日期：2026-07-15（Asia/Singapore）
+
+状态：**问题定义与历史成本审计完成；新的 tuner 贡献尚未建立。**
+
+## 结论先行
+
+我们不应该把 tuning 定义成“根据当前 telemetry 判断哪个 cap 满了，再调对应 knob”。这个定义同时遗漏了 knob interaction、反事实识别、实验成本和跨任务失配。更准确的问题是：
+
+> 给定模型、engine version、hardware、workload、SLO 和一个声明好的合法配置空间，tuner 如何用最少的真实 GPU 成本，依次选择可能包含多个 knob 的 intervention，找到 SLO-goodput regret 不超过 `epsilon` 的配置？
+
+AITuner 可以形成的系统贡献应当是：
+
+> **一个 intervention-calibrated、action-conditioned、cost-aware 的 tuner：它从真实 engine trajectory 和已测 intervention 中学习联合 config action 的反事实收益分布，并以 cost-to-oracle 而非规则命中率作为目标。Harness 只负责实验语义、合法性、配对、记账和可复现性，不负责用人工 bottleneck rule 决定 action。**
+
+现有结果支持这个问题值得做，但不支持宣称它已经解决：
+
+- 在真实 `TP x MNS` surface 上，one-knob-at-a-time 会停在比 oracle 低 **25.6%** 的 coordinate-wise local optimum。
+- 在 action-aware pilot 中，增加 MBBT 在“几乎从未独占打满 MBBT cap”的情况下仍把 source goodput 提高 **48.0%--77.1%**；因此 `cap -> knob` 不是完整模型。
+- 同一 dash0 任务上，当前 guided harness 到 5% empirical regret 只比纯 LLM 少 **5.85%** H20-hours；到 2% regret 则少 **61.09%**。这说明必须比较完整 cost--regret curve，不能只比较最终最好值。
+- Frontier 的 decision-bearing throughput top-1 在 12-cell surface 上有 **30.46%** real regret。Simulator 本身的边际 GPU cost 是 0，但通过 real-final 恢复 oracle 需要 tie-expanded 4 个真实 cell，即 **0.7828 reconstructed H20-hours**。
+
+## 1. Tuning 问题和成功标准
+
+固定 task context：
+
+```text
+T = {model, engine build, hardware, workload/trace, SLO, legal config space C}
+```
+
+每个完整配置 `c in C` 的目标为：
+
+```text
+f_T(c) = max request_rate_per_gpu
+         subject to request SLO pass rate >= target
+```
+
+有限空间 oracle 为：
+
+```text
+f*_T = max_{c in C} f_T(c)
+regret(c) = 1 - f_T(c) / f*_T
+```
+
+顺序 tuner 在第 `t` 步基于历史 `D_t` 选择一个完整 config intervention：
+
+```text
+a_t = c_t -> c_{t+1}
+```
+
+成功不是“最后找到一个不错的值”，而是同时满足：
+
+1. `regret(best_t) <= epsilon`；
+2. 达到该点之前的 all-in H20-hours 最小；
+3. launch、correctness、SLO 和失败率约束不退化；
+4. 结论在 held-out task 上成立，而不是在用于设计规则的 task 上成立。
+
+### 1.1 GPU cost 的统一定义
+
+未来实验的 task-marginal cost 应定义为：
+
+```text
+C_task = sum_j allocated_GPU_count_j
+                 * (GPU_idle_or_release_time_j - allocation_start_time_j)
+```
+
+它包括 method 实际触发的 startup、warm-up、prefix/full replay、confirmation、failure、cleanup；如果 LLM 思考期间 GPU 仍被占用，也计入。Simulator/模型的一次性 onboarding 成本单独报告：
+
+```text
+C_e2e(N tasks) = C_profile_or_training / N + C_task
+```
+
+另外报告 CPU-hours、LLM API latency/cost，但不把它们伪装成 GPU-hours。构建 benchmark oracle 的 exhaustive annotation cost 是公共评测成本，单独报告，不计入任何方法；同时可给一个将其等量加回所有方法的 conservative view。
+
+历史记录没有 allocation start/release timestamp。本次只能从每个 `engine.log` 的首末时间戳重建：
+
+```text
+C_engine_lower_bound = parallel_size * engine_log_span / 3600
+```
+
+因此下面所有历史 H20-hour 数字都是 **engine-lifetime lower bound**，不是 all-in cost。尤其 simulator 的一次性 H20 operator profiling 成本没有记录，不能称为完全免费。
+
+### 1.2 两种 oracle 必须分开
+
+- **Exact finite-surface oracle**：声明好的 12-cell `TP x MNS` 空间全部真实测量，oracle 是 `TP2/MNS32 = 3.2833 req/s/GPU`。
+- **Broader empirical reference**：dash0 两个 sequential run 中观察到的最好值 `3.35 req/s/GPU`。它包含 surface 外的 MBBT/chunk/GMU action，但只是 best observed，不是全局 oracle。
+
+不能把 empirical best 写成 global oracle，也不能让每个方法使用不同的 oracle 定义。
+
+## 2. 现有方案的 cost-to-oracle 审计
+
+可复算输入和完整结果在：
+
+- `runs/tuning-cost/manifest.json`
+- `runs/tuning-cost/analyze.py`
+- `runs/tuning-cost/metrics.json`
+
+### 2.1 严格同任务对照：纯 LLM vs 当前 guided harness
+
+两组均为 dash0、Qwen3-30B-A3B、community-vLLM 0.20.0、8xH20 可见、`chat_w20260311_1000`、input 0--8k、output 128、replay scale 0.1、TTFT 2/4/6s、TPOT 50ms、pass rate 0.95。除 tuner method 和服务端口外，固定 task spec 相同。
+
+Reference 是两组中 best observed `3.35 req/s/GPU`：
+
+| Method | 到 <=5% regret | 到 <=2% regret | 到 <=1% regret | 完整 run 成本 | 最终 best |
+|---|---:|---:|---:|---:|---:|
+| Pure LLM, no harness | 0.2847 H20h，trial 2，regret 2.736% | 1.1458，trial 6，regret 1.493% | 1.3719，trial 7，regret 0% | 2.2825 | 3.35 |
+| Guided harness v2 | 0.2681 H20h，trial 2，regret 2.736% | 0.4458，trial 3，regret 1.990% | 未达到 | 0.6231 | 3.30，regret 1.493% |
+
+直接结论：
+
+- 5% endpoint：guided 比 pure LLM 少 **5.85%**，不是 material contribution。
+- 2% endpoint：guided 比 pure LLM 少 **61.09%**，有明显 headroom signal，但只有一个 task，不能外推。
+- Pure LLM 在 trial 7 已找到 best observed，之后又花了 `2.2825 - 1.3719 = 0.9106 H20h` 而没有改进，说明 trustworthy stopping 本身就是成本来源。
+- Pure LLM 的 trial 3 使用当前 binary 不支持的 `--expert-parallel-size` 并在 launch 前失败。当前 harness 的 legality/version contract 有实际价值，但它仍不是性能 action-ranking 贡献。
+
+### 2.2 Simulator：零边际 GPU cost 不等于零 tuning cost
+
+Frontier fidelity suite 在 CPU 上执行 184 个 simulation，耗时 **2.055 CPU-hours**，simulation 本身为 0 marginal H20-hours。其对应的 exact dash1 12-cell real surface annotation lower bound 为 **3.5953 H20-hours**。
+
+Decision-bearing `frozen-calibrated/throughput-proxy`：
+
+| Policy | Real cells evaluated | Real-final H20h lower bound | Selected real regret |
+|---|---:|---:|---:|
+| Simulator-only top-1 | 0 | 0 | **30.46%**，选 TP1/MNS64 |
+| Throughput top-1 + real final | 1 | 0.1353 | **30.46%** |
+| Throughput top-2 + real final | 2 | 0.2672 | **30.46%** |
+| Throughput nominal top-3 + real final | tie-expanded 4 | 0.7828 | 0%，找到 TP2/MNS32 |
+
+Post-hoc `SLO-gated` reading 把 `{TP2/MNS32, TP2/MNS64}` 放在 top tie bucket；测两个 cell 需 **0.5156 H20h** 并能找到 oracle。但它不是 preregistered decision-bearing policy，而且 anchor verdict 中有 21 个 false-feasible、7 个 false-infeasible，只能作为诊断上界，不能反写成 prospective simulator 结果。
+
+Pure LLM/harness 数据来自 dash0，simulator exact surface 来自 dash1。模型、engine、trace、GPU type 匹配，但 host 和 campaign 不同。因此两块内部可以直接比较，跨块只能做 development-level 指示；paper 结论必须在同 host、同 task execution protocol 下重跑。
+
+### 2.3 我们要达到的成本目标
+
+在当前 reconstructed lower-bound 口径下，一个有意义的单任务 development bar 是：
+
+| Endpoint | 当前最强同任务 baseline | 20% reduction bar | 兼顾 post-hoc sim+real 的 30% bar | 暂定目标 |
+|---|---:|---:|---:|---:|
+| <=5% empirical regret | guided 0.2681 | 0.2144 | 0.3609 | **<=0.2144 H20h** |
+| <=2% empirical regret | guided 0.4458 | 0.3567 | 0.3609 | **<=0.3567 H20h** |
+
+这两个数字不是 paper result，只用于检查 proposed method 是否有足够 headroom：
+
+- 5% endpoint 已经由 baseline + TP2 两个完整 trial 达到。任何必须先跑 source 再跑 target 的 telemetry tuner 都不能靠减少 trial count 获得 20% 优势；它必须能够 one-shot warm-start、跳过 baseline，或安全地缩短其中一次测量。
+- 2% endpoint 有更合理的结构性空间：从一个 source 直接选择 joint `TP2 + MBBT/chunk` target，可能跳过当前中间 trial；如果仍按当前三次完整 trial 顺序执行，就不会达到 bar。
+
+Paper-facing gate 不使用这些跨 campaign 绝对数，而使用 prospective same-host all-in cost：在每个 held-out task 上 regret <=5%，相对最强 safe outcome-only/current harness 至少省 20%，相对 frozen simulator+real 至少省 30%，并报告 task-level paired confidence interval。
+
+## 3. 四个最核心的 tuning challenge
+
+### Challenge 1：响应面是联合、条件化且 regime-dependent 的
+
+#### 问题本质
+
+一般情况下：
+
+```text
+f(c) != base + sum_k effect_k(c_k)
+```
+
+一个 knob 的 effect 是当前完整 context 的函数：
+
+```text
+Delta_x(c, workload, engine state)
+```
+
+它可能随 topology、另一个 runtime knob、load、SLO 或 engine version 改变大小甚至改变符号。因此不能先分别求每个 knob 的最优值再 merge，也不能固定一个低质量 context 去判断另一个 knob。
+
+#### 已有真实证据
+
+在 C1 12-cell real surface：
+
+- `MNS 8 -> 32` 在 TP1/TP2/TP4 下分别提升约 **8.7% / 44.3% / 90.3%**。
+- 从同一 `TP1/MNS8` 起点，先 tune MNS 再 TP 会停在 `TP4/MNS16 = 2.4417`；该点沿任一单维都没有 strictly improving move，但 joint/global surface oracle `TP2/MNS32 = 3.2833` 高 **34.5% relative to the local point**，即 local point 对 oracle 有 **25.6% regret**。
+- C3 中 `MBT 256 -> 384` 的 effect 根据 topology/MNS 从 0 到约 -9.2%；`MNS 64 -> 128` 从 0 到约 +10.1%。
+- Action-aware Regime A 中 MBBT 几乎从不作为 exclusive cap，但 MBBT action 仍把 source goodput 提高 48.0%--77.1%。它通过 chunk size、prefill packing 和 scarce MNS slot residency 的联合变化获得收益。
+
+这直接否定两类通用策略：OAT/coordinate greedy，以及 `which cap is full -> tune that knob`。
+
+#### Tuner 必须具备的能力
+
+- Action 的基本单位是完整 `config delta`，允许 sparse joint action，而不是孤立 knob/value。
+- 对 topology/runtime family 使用 crossed anchors 或信息增益设计，主动测 interaction；不是默认所有 interaction 都强。
+- 能从数据判断 task 是 topology-dominant、runtime-interaction-dominant 还是 flat/noisy，并据此分配实验，而不是把固定 search order 写进规则。
+
+### Challenge 2：当前状态是 observational signal，tuning 需要 counterfactual identification
+
+#### 问题本质
+
+一次 telemetry trace 只能告诉我们：
+
+```text
+P(engine trajectory | current config, workload)
+```
+
+Tuning 真正需要的是：
+
+```text
+P(Delta SLO-goodput, failure, cost
+  | source trajectory, proposed full-config action)
+```
+
+Queue、KV、padding、split prefill 等状态既可能是原因，也可能是 workload/config 的结果。看见某种状态，不等于知道哪个 action 能修复它。一个 action 也可能同时改变多条机制；例如 MBBT 同时改变总 token budget、per-request chunk 和 multi-request packing，现有 telemetry 的解释是 mechanism-consistent，不是已完成的 causal decomposition。
+
+#### 已有真实证据
+
+- 5/10 秒 telemetry 确实太短；300 秒 phase-aware experiment 中，MNS action 的 queue/padding 机制直到 replay 75%--100% 才稳定出现。
+- 但 external TTFT outcome 在 25% 已完美区分该 action 是否修复 SLO。Telemetry 解释了 why，却没有比 outcome 更早或更可靠地指导 tuning。
+- 3.125 req/s/GPU 的 source 无法在 timeout 内 drain；另一组 source 已达 offered ceiling 的 99.1%--100%，数学上不可能通过 10% improvement gate。没有 exposure/headroom 和 censoring control，模型学到的不是 action response。
+- Same-config repeats 与 matched intervention 的波动不可忽略；只比较两个未经配对的 run 会混入 arrival/order/warm-state noise。
+
+#### Tuner 必须具备的能力
+
+- 训练样本必须是 exact-workload paired intervention：`(source trajectory, action) -> target delta`，保留失败和 censoring。
+- 使用 phase-binned continuous trajectory，而不是人工 bottleneck label 或 threshold rule。
+- 输出 response distribution 和 uncertainty；证据不足时 abstain，而不是强行给 diagnosis。
+- Telemetry 的价值必须通过同 cutoff、同 model capacity 的 outcome-only ablation 证明。若不能降低 end-to-end H20-hours，instrumentation 只保留为 debugging/解释工具。
+
+### Challenge 3：这是异构成本下的 sequential experimental design，不是静态 ranking
+
+#### 问题本质
+
+每个 trial 的成本不同：TP4 是 TP1 的四倍 GPU multiplier，startup/warm-up 可能主导短 probe，失败也有成本；同时 tuner 不知道 oracle，只能在 exploitation、information gain 和 cost 之间权衡。选对 top-1 的 accuracy 不能代表 tuning 效果。
+
+必须回答三个连续问题：
+
+1. 下一次测哪个联合 action？
+2. 测多久，何时 continuation/confirmation？
+3. 什么证据允许停止，并声称 best 已在 `epsilon` 内？
+
+#### 已有真实证据
+
+- Pure LLM 达到 best observed 后仍浪费 0.9106 reconstructed H20h。
+- Simulator top-1 虽然 0 marginal GPUh，却因 rank error 损失 30.46%；real-final 的 k 增大又迅速增加 H20h。
+- 5% endpoint 上两个方法都只需两个 trial，selection-count headroom 很小；2% endpoint 才暴露 action quality 和 stopping 的巨大差异。
+- Prefix 不是天然便宜：如果 startup、warm-up 和稳定状态形成占主要成本，缩短 replay window 未必带来等比例 H20h reduction。
+
+#### Tuner 必须具备的能力
+
+- Acquisition 直接优化 expected regret reduction / predicted H20 cost，并把 failure probability 纳入约束。
+- 在 run 前做与 tuning policy 分离的 workload admissibility check：避免 outcome ceiling、无法 drain、无请求或 measurement cap。
+- 使用 uncertainty-aware continuation 和 stop；stop criterion 针对声明的 candidate set 中“仍存在 >epsilon improvement 的概率”，而不是连续几次没提升。
+- 主结果报告 H20-hours-to-5%/2%/1%、fixed-budget regret 和 cost-normalized regret AUC，不 metric shopping。
+
+### Challenge 4：任何 mechanism model 都有 fidelity 和 transfer boundary
+
+#### 问题本质
+
+Simulator、learned surrogate、LLM prior 都是近似。Workload、SLO、model、hardware、engine version 改变后，operator cost、scheduler state transition、合法 flag 和 response surface 都可能变化。模型在 calibration task 上解释得好，不表示能在 held-out task 上排序正确。
+
+#### 已有真实证据
+
+- Frontier throughput reading 在完全匹配的 12-cell task 上仍把 real oracle 排错，top-1 regret 30.46%。这说明预测绝对 throughput 还不够，局部 rank fidelity 才是 tuning 关键。
+- Post-hoc SLO reading 的 top bucket 正确，但有大量 anchor feasibility error，也没有 prospective policy status。
+- Pure LLM 提出了当前 community-vLLM binary 不支持的 flag；engine/API version knowledge 本身会漂移。
+- 已有 cross-version experiment 中 vLLM 0.20 的强配置在 0.24 上出现大幅退化，说明 response prior 不能无条件迁移。
+
+#### Tuner 必须具备的能力
+
+- Simulator 只能作为 prior mean 或 candidate prior；真实 outcome 是 authoritative update。
+- 学习 simulator residual：把 `sim prediction + source state + action` 映射到 real response，而不是用 telemetry 重新实现另一个无校准 simulator。
+- 对 task-level OOD 显式提高 uncertainty/abstain；train/test 按完整 task 分割，不能按 request、anchor 或同一 surface cell 随机分割。
+- 分开报告 cold-start profile/training cost 与 per-task marginal cost，并在 N=1/10/100 等 amortization horizon 下展示。
+
+## 4. 对应的系统设计
+
+### 4.1 Harness：从 rule-based tuner 收缩成 experimental control plane
+
+Harness 保留以下确定性职责：
+
+- engine-version-aware config schema、合法性和资源约束；
+- 完整 config/action canonicalization，禁止隐式 merge 和重复试验；
+- exact trace/request/arrival/length hash，配对、随机化和 counter-rotation；
+- engine trajectory、external outcome、failure/censoring 的统一时间轴；
+- all-in GPU cost ledger、oracle annotation 分账、budget enforcement；
+- data sanity、coverage、SLO/correctness 和 stop-proof audit。
+
+Harness **不**包含 `queue > N -> increase MNS`、`cap full -> tune knob` 或人工 diagnosis-to-action mapping。这里的规则是实验语义和安全 invariant，不是性能决策 heuristic。
+
+### 4.2 Action-conditioned response model
+
+每条学习记录为：
+
+```text
+x = {source full config,
+     workload/SLO context,
+     source external outcome,
+     phase-binned engine trajectory}
+a = normalized full-config delta
+y = {Delta SLO-goodput, target feasibility/failure, measured H20 cost}
+```
+
+学习：
+
+```text
+p_theta(y | x, a, optional simulator prediction)
+```
+
+第一版应使用适合小数据且有 uncertainty 的 action-conditioned Gaussian-process/bootstrapped surrogate；kernel/feature ablation包括：
+
+1. config + external outcome；
+2. 同样输入 + telemetry trajectory；
+3. simulator + config + outcome；
+4. 同样输入 + telemetry residual features。
+
+Telemetry 保留 continuous phase distributions：queue/running residency、MNS/token slack、prefill/decode composition、partial/split prefill、step duration、KV、graph/padding。模型学习它们与 action 的 interaction；不先压成 bottleneck label。
+
+### 4.3 Cost-aware policy
+
+在合法的 single/joint candidate set 上选择：
+
+```text
+a* = argmax_a
+       expected constrained improvement(a)
+       / expected all-in H20 cost(a)
+```
+
+探索项来自 posterior uncertainty/information gain；launch/SLO failure 有显式 penalty。Simulator 可提供 prior mean，但 simulator 与 real discrepancy 会被 posterior residual 更新。一次 target measurement 后更新 response model，并重新计算下一步 action 或停止概率。
+
+LLM 在这个 tuning core 中不是 telemetry classifier。它最多作为可移除的 candidate/prior source，提出 schema 内的 sparse joint actions 或检索 engine mechanism；每个 proposal 都由同一个 response model、cost acquisition 和 real validator 评分。只有 `with LLM` 相对 `same tuner without LLM` 在 held-out tasks 上继续降低 cost-to-oracle，才能讨论 LLM 必要性。
+
+### 4.4 Stop 条件
+
+对一个预先声明的有限 candidate set，满足以下条件才 stop：
+
+```text
+P(exists c: f(c) > best_observed / (1 - epsilon) | D_t) < alpha
+```
+
+并且 best config 通过独立 confirmation、SLO/correctness gate，remaining candidate 的 cost-aware value of information 低于阈值。停止原因、posterior coverage 和未测区域必须写入 audit。
+
+## 5. 下一阶段如何证明，而不是再次构造 heuristic
+
+### R0：已有数据 retrospective premise check
+
+- 用 C1/C3 response surfaces 检查 joint model 是否能避免 OAT trap。
+- 用 action-aware paired records 比较 outcome-only 与 +telemetry 的 action-delta calibration。
+- 用 SimFid surface 比较 direct model 与 simulator-residual model 的 rank/regret。
+- 所有 feature、kernel、candidate encoding 在 held-out task 结果之前冻结。
+
+R0 只能筛选 model family，不能作为 paper result，因为现有 tasks 已参与路线设计。
+
+### R1：prospective same-host cost-to-oracle pilot
+
+- dash0 8xH20，固定 engine build/model；serialized placement，禁止共置干扰。
+- 至少一个未参与 feature/threshold 选择的新 trace window；选择非 ceiling、可 drain 的 offered load。
+- 声明一个可穷举的小 surface，至少包含 topology/runtime crossed actions，而不是只有一个 MNS ladder。
+- Oracle annotation 与 tuner online actions 分开记账；method 只能看到当时可用的数据。
+- 运行 random/search、OAT、纯 LLM、当前 guided harness、frozen simulator+real、outcome-only response、+telemetry response、sim-residual +telemetry。
+- 比较完整 H20 cost-to-regret curve，而不是 action classification accuracy。
+
+Pilot opening gate：
+
+1. telemetry model 相对相同 response model 去掉 telemetry，确实改变至少一个正确的 prospective action ranking；
+2. 最终 regret <=5%，无 false-safe accept；
+3. all-in H20-hours 相对 strongest safe outcome-only 至少下降 20%；
+4. 如果使用 simulator，需相对 frozen simulator+real 至少下降 30%；
+5. instrumentation overhead <=1%，所有成本和失败均计入。
+
+若 1--5 任一失败，就不能把 telemetry/harness 写成 tuning contribution；保留其 debugging/measurement 价值即可。
+
+### R2：task-held-out replication
+
+至少 3 个 workload window x 2 个 SLO regime，按完整 task 做 leave-one-task-out 或固定 train/test split。报告每个 task 的 regret、安全和成本，以及 task-level paired bootstrap CI。只有 R2 通过，才能把单 task 的 61.09% lower-bound saving 升级为项目贡献。
+
+## 6. 当前能说与不能说的贡献
+
+当前能说：
+
+- 我们有真实反例证明 OAT 和 cap-to-knob mapping 不是通用 tuning strategy。
+- Harness 的 legality、exact replay、failure/cost accounting 有必要的实验基础设施价值。
+- 当前 guided sequence 在一个严格同任务比较中显著减少了达到 2% empirical regret 的 reconstructed engine cost。
+- Simulator 的边际计算便宜，但 rank error 会转化成显著 real regret 或更多 real-final 成本。
+
+当前不能说：
+
+- telemetry 已经对 end-to-end tuning 提供独立增益；现有 direct pilot 对此为 negative。
+- 当前 harness 的 heuristic action ranking 是系统贡献；5% endpoint 只省 5.85%。
+- LLM 是必要组件；尚无同 policy 的 with/without LLM held-out ablation。
+- simulator 总 tuning cost 是 0；profile GPU cost 未审计，real verification 不能忽略。
+- 3.35 是 global oracle，或 dash0 与 dash1 数字是完全 controlled comparison。
+
+## Data sanity
+
+- Dash0 sequential numeric scores：n=9，min/max `1.1042/3.35`，distinct=7；两组 config outcome 不全相同。
+- Exact surface scores：n=12，min/max `1.2833/3.2833`，distinct=8；12 cells 完整且与 simulator metrics 中的 real scores 一致。
+- Reconstructed trial/cell attempts 包括 4 个无 engine timestamp 的失败：n=32，min/max `0/0.49778 H20h`，distinct=26；所有可重建成本均非负。
+- Sequential regret observations：n=16，min/max `0/0.34328`，distinct=6，全部在 `[0,1]`。
+- Checked invariants：dash0 fixed task contexts 相同（除 method/port）；trial counts 与 manifest 相符；engine log timestamps monotonic；surface cell 唯一且 MBT=8192；simulator 无失败且 predictions 不全相同；scores/results 不全相同；cost 非负；regret bounded。
+- Measurement limitation：primary 12-cell campaign 的 4 个 TP4 pre-ready failure 没有 engine timestamp，随后由 companion campaign 完整重跑；其失败成本在 engine-lifetime reconstruction 中为 0。因此 `3.5953 H20h` 是 completed annotation lower bound，不能作为 all-in annotation cost。这个缺口已显式保留，没有在其上建立 total-cost claim。

patches/vllm-0.24.0-opprof/0001-Add-lightweight-per-step-OpProf-telemetry.patch

@@ -0,0 +1,487 @@
+From f6f1cacbce0e39992d04843f652c1adda373ae43 Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sat, 11 Jul 2026 17:29:02 +0800
+Subject: [PATCH 1/5] Add lightweight per-step OpProf telemetry
+
+Assisted-by: OpenAI Codex
+---
+ vllm/envs.py                       |   4 +
+ vllm/v1/core/sched/scheduler.py    |  28 +++
+ vllm/v1/opprof.py                  | 337 +++++++++++++++++++++++++++++
+ vllm/v1/worker/gpu_model_runner.py |   6 +-
+ 4 files changed, 374 insertions(+), 1 deletion(-)
+ create mode 100644 vllm/v1/opprof.py
+
+diff --git a/vllm/envs.py b/vllm/envs.py
+index 27a85bb..b3093e9 100755
+--- a/vllm/envs.py
++++ b/vllm/envs.py
+@@ -45,6 +45,7 @@ if TYPE_CHECKING:
+     VLLM_LOGGING_COLOR: str = "auto"
+     NO_COLOR: bool = False
+     VLLM_LOG_STATS_INTERVAL: float = 10.0
++    VLLM_OPPROF_DIR: str = ""
+     VLLM_TRACE_FUNCTION: int = 0
+     VLLM_USE_FLASHINFER_SAMPLER: bool = True
+     VLLM_PP_LAYER_PARTITION: str | None = None
+@@ -786,6 +787,9 @@ environment_variables: dict[str, Callable[[], Any]] = {
+         if (val := float(os.getenv("VLLM_LOG_STATS_INTERVAL", "10."))) > 0.0
+         else 10.0
+     ),
++    # Directory for per-step OpProf JSONL telemetry.
++    # Empty disables OpProf.
++    "VLLM_OPPROF_DIR": lambda: os.getenv("VLLM_OPPROF_DIR", ""),
+     # Trace function calls
+     # If set to 1, vllm will trace function calls
+     # Useful for debugging
+diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py
+index 90d93a1..303c562 100644
+--- a/vllm/v1/core/sched/scheduler.py
++++ b/vllm/v1/core/sched/scheduler.py
+@@ -7,6 +7,7 @@ from collections.abc import Iterable
+ from dataclasses import replace
+ from typing import Any
+ 
++import vllm.envs as envs
+ from vllm.compilation.cuda_graph import CUDAGraphStat
+ from vllm.config import VllmConfig
+ from vllm.distributed.ec_transfer.ec_connector.base import (
+@@ -55,6 +56,7 @@ from vllm.v1.engine import EngineCoreEventType, EngineCoreOutput, EngineCoreOutp
+ from vllm.v1.kv_cache_interface import KVCacheConfig
+ from vllm.v1.metrics.perf import ModelMetrics, PerfStats
+ from vllm.v1.metrics.stats import PrefixCacheStats, SchedulerStats
++from vllm.v1.opprof import OpProfRecorder
+ from vllm.v1.outputs import DraftTokenIds, KVConnectorOutput, ModelRunnerOutput
+ from vllm.v1.request import Request, RequestStatus, StreamingUpdate
+ from vllm.v1.spec_decode.dynamic.utils import build_dynamic_sd_schedule_lookup
+@@ -271,6 +273,12 @@ class Scheduler(SchedulerInterface):
+         if self.connector is not None:
+             self.connector.bind_gpu_block_pool(self.kv_cache_manager.block_pool)
+ 
++        self.opprof = OpProfRecorder.create(
++            output_dir=envs.VLLM_OPPROF_DIR,
++            dp_rank=self.parallel_config.data_parallel_index,
++            log_stats=self.log_stats,
++        )
++
+         self.use_pp = self.parallel_config.pipeline_parallel_size > 1
+         self.use_v2_model_runner = vllm_config.use_v2_model_runner
+         # Scheduler iteration counter. Drives the V2+PP+async decode-throttle
+@@ -386,6 +394,9 @@ class Scheduler(SchedulerInterface):
+         return num_new_tokens
+ 
+     def schedule(self, throttle_prefills: bool = False) -> SchedulerOutput:
++        opprof_start = (
++            self.opprof.capture_start(self) if self.opprof is not None else None
++        )
+         self.current_step += 1
+         # NOTE(woosuk) on the scheduling algorithm:
+         # There's no "decoding phase" nor "prefill phase" in the scheduler.
+@@ -1090,6 +1101,14 @@ class Scheduler(SchedulerInterface):
+             )
+             scheduler_output.ec_connector_metadata = ec_meta
+ 
++        if self.opprof is not None:
++            assert opprof_start is not None
++            self.opprof.begin(
++                scheduler=self,
++                output=scheduler_output,
++                start=opprof_start,
++            )
++
+         # Advance the fence only for non-empty steps (those that actually
+         # write KV and have their output processed later in update_from_output).
+         if self.defer_block_free and total_num_scheduled_tokens > 0:
+@@ -1800,6 +1819,12 @@ class Scheduler(SchedulerInterface):
+                 engine_core_outputs[0] = eco = EngineCoreOutputs()
+             eco.scheduler_stats = stats
+ 
++        if self.opprof is not None:
++            self.opprof.finalize(
++                output=scheduler_output,
++                cudagraph_stat=cudagraph_stats,
++            )
++
+         return engine_core_outputs
+ 
+     @staticmethod
+@@ -2292,6 +2317,9 @@ class Scheduler(SchedulerInterface):
+         if self.ec_connector is not None:
+             self.ec_connector.shutdown()
+ 
++        if self.opprof is not None:
++            self.opprof.close()
++
+         logger.debug_once("[shutdown] Scheduler: complete")
+ 
+     ########################################################################
+diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py
+new file mode 100644
+index 0000000..f0330d0
+--- /dev/null
++++ b/vllm/v1/opprof.py
+@@ -0,0 +1,337 @@
++# SPDX-License-Identifier: Apache-2.0
++# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
++import atexit
++import logging
++import os
++import queue
++import threading
++import time
++from bisect import bisect_left
++from pathlib import Path
++from typing import Any
++
++import msgspec
++
++logger = logging.getLogger(__name__)
++
++SCHEMA_VERSION = 1
++CONTEXT_LENGTH_EDGES = tuple(1 << exponent for exponent in range(7, 18))
++CHUNK_SIZE_EDGES = tuple(1 << exponent for exponent in range(4, 12))
++DEFAULT_QUEUE_CAPACITY = 8192
++_CLOSE_TIMEOUT_SECONDS = 1.0
++_STOP = object()
++_PREFIX_FIELDS = (  # noqa: SIM905
++    "requests queries hits preempted_requests preempted_queries preempted_hits"
++).split()
++
++
++ScheduleStart = tuple[int, int, dict[str, dict[str, int] | None]]
++
++
++def classify_chunk(was_chunk: bool, end: int, target: int) -> str:
++    assert 0 <= end <= target
++    if was_chunk:
++        return "middle" if end < target else "final"
++    return "first" if end < target else "unsplit"
++
++
++def _prefix_values(stats: Any | None) -> dict[str, int] | None:
++    if stats is None:
++        return None
++    return {name: int(getattr(stats, name, 0)) for name in _PREFIX_FIELDS}
++
++
++def _prefix_snapshot(scheduler: Any) -> dict[str, dict[str, int] | None]:
++    return {
++        "local": _prefix_values(scheduler.kv_cache_manager.prefix_cache_stats),
++        "external": _prefix_values(scheduler.connector_prefix_cache_stats),
++    }
++
++
++def _prefix_delta(
++    before: dict[str, int] | None, after: dict[str, int] | None
++) -> dict[str, int] | None:
++    if before is None and after is None:
++        return None
++    before = before or dict.fromkeys(_PREFIX_FIELDS, 0)
++    after = after or dict.fromkeys(_PREFIX_FIELDS, 0)
++    delta = {name: after[name] - before[name] for name in _PREFIX_FIELDS}
++    assert all(value >= 0 for value in delta.values())
++    return delta
++
++
++class JSONLWriter:
++    def __init__(
++        self,
++        path: Path,
++        capacity: int = DEFAULT_QUEUE_CAPACITY,
++        start: bool = True,
++    ) -> None:
++        self._queue: queue.Queue[Any] = queue.Queue(capacity)
++        self._encoder = msgspec.json.Encoder()
++        self._file = path.open("xb", buffering=1 << 20)
++        self._thread = threading.Thread(target=self._run, daemon=True)
++        self._failure_lock = threading.Lock()
++        self._started = self._closed = False
++        self.failed = False
++        self.failure: Exception | None = None
++        self.encoded_records = self.written_records = 0
++        self.dropped_records = self._unreported_drops = 0
++        if start:
++            self.start()
++
++    def start(self) -> None:
++        if not self._started:
++            self._started = True
++            self._thread.start()
++
++    def _record_failure(self, error: Exception) -> None:
++        with self._failure_lock:
++            if self.failed:
++                return
++            self.failed = True
++            self.failure = error
++            logger.error("OpProf writer failed: %s", error)
++
++    def _writer_unavailable(self) -> bool:
++        if self.failed:
++            return True
++        if self._started and not self._thread.is_alive():
++            self._record_failure(RuntimeError("writer thread stopped unexpectedly"))
++            return True
++        return False
++
++    def _drop(self, pending: int) -> bool:
++        self.dropped_records += 1
++        self._unreported_drops = pending + 1
++        return False
++
++    def submit(self, record: dict[str, Any]) -> bool:
++        if self._closed:
++            raise RuntimeError("OpProf writer is closed")
++        pending = self._unreported_drops
++        record["dropped_records_before"] = pending
++        if self._writer_unavailable():
++            return self._drop(pending)
++        payload = self._encoder.encode(record) + b"\n"
++        self.encoded_records += 1
++        if self._writer_unavailable():
++            return self._drop(pending)
++        try:
++            self._queue.put_nowait(payload)
++        except queue.Full:
++            return self._drop(pending)
++        self._unreported_drops = 0
++        return True
++
++    def _run(self) -> None:
++        buffered = 0
++        last_flush = time.monotonic()
++        try:
++            while True:
++                try:
++                    item = self._queue.get(timeout=1.0)
++                except queue.Empty:
++                    self._file.flush()
++                    buffered = 0
++                    last_flush = time.monotonic()
++                    continue
++                try:
++                    if item is _STOP:
++                        break
++                    self._file.write(item)
++                    buffered += len(item)
++                    self.written_records += 1
++                    now = time.monotonic()
++                    if buffered >= 1 << 20 or now - last_flush >= 1.0:
++                        self._file.flush()
++                        buffered = 0
++                        last_flush = now
++                finally:
++                    self._queue.task_done()
++        except Exception as error:
++            self._record_failure(error)
++        finally:
++            footer = dict(
++                schema=SCHEMA_VERSION,
++                record_type="footer",
++                encoded_records=self.encoded_records,
++                written_records=self.written_records,
++                dropped_records=self.dropped_records,
++            )
++            try:
++                self._file.write(self._encoder.encode(footer) + b"\n")
++                self._file.flush()
++            except Exception as error:
++                self._record_failure(error)
++            finally:
++                try:
++                    self._file.close()
++                except Exception as error:
++                    self._record_failure(error)
++
++    def close(self) -> None:
++        if self._closed:
++            return
++        self._closed = True
++        self.start()
++        if not self._thread.is_alive():
++            return
++        try:
++            self._queue.put(_STOP, timeout=_CLOSE_TIMEOUT_SECONDS)
++        except queue.Full:
++            if not self._thread.is_alive():
++                return
++            try:
++                self._queue.put_nowait(_STOP)
++            except queue.Full:
++                self._record_failure(
++                    TimeoutError("timed out enqueueing writer stop sentinel")
++                )
++                return
++        self._thread.join(timeout=_CLOSE_TIMEOUT_SECONDS)
++        if self._thread.is_alive():
++            self._record_failure(TimeoutError("timed out joining writer thread"))
++
++
++class OpProfRecorder:
++    def __init__(self, engine_id: str, writer: JSONLWriter) -> None:
++        self.engine_id = engine_id
++        self.writer = writer
++        self._next_step = 0
++        self._pending: dict[int, dict[str, Any]] = {}
++        atexit.register(self.close)
++
++    @classmethod
++    def create(
++        cls, output_dir: str, dp_rank: int, log_stats: bool
++    ) -> "OpProfRecorder | None":
++        if not output_dir:
++            return None
++        if not log_stats:
++            raise ValueError("VLLM_OPPROF_DIR requires log stats to be enabled")
++        directory = Path(output_dir).expanduser()
++        if not directory.is_absolute():
++            raise ValueError("VLLM_OPPROF_DIR must be an absolute path")
++        directory.mkdir(parents=True, exist_ok=True)
++        engine_id = f"dp{dp_rank}-pid{os.getpid()}"
++        name = f"opprof-v{SCHEMA_VERSION}-{engine_id}-{time.time_ns()}.jsonl"
++        return cls(engine_id, JSONLWriter(directory / name))
++
++    @staticmethod
++    def capture_start(scheduler: Any) -> ScheduleStart:
++        return time.time_ns(), time.monotonic_ns(), _prefix_snapshot(scheduler)
++
++    def begin(self, scheduler: Any, output: Any, start: ScheduleStart) -> None:
++        key = id(output)
++        assert key not in self._pending, "duplicate OpProf begin"
++        new_ids = {request.req_id for request in output.scheduled_new_reqs}
++        prefill_requests = prefill_tokens = decode_requests = decode_tokens = 0
++        context_length_hist = [0] * (len(CONTEXT_LENGTH_EDGES) + 1)
++        chunk_size_hist = [0] * (len(CHUNK_SIZE_EDGES) + 1)
++        chunks: dict[str, Any] = dict.fromkeys(
++            ("first", "middle", "final", "unsplit"), 0
++        )
++        for req_id, num_tokens in output.num_scheduled_tokens.items():
++            request = scheduler.requests[req_id]
++            end = request.num_computed_tokens + num_tokens
++            assert end >= 0
++            context_length_hist[bisect_left(CONTEXT_LENGTH_EDGES, end)] += 1
++            is_prefill = (
++                req_id in new_ids
++                or output.scheduled_cached_reqs.is_context_phase(req_id)
++            )
++            if is_prefill:
++                prefill_requests += 1
++                prefill_tokens += num_tokens
++                assert num_tokens >= 0
++                chunk_size_hist[bisect_left(CHUNK_SIZE_EDGES, num_tokens)] += 1
++                target = request.num_tokens + request.num_output_placeholders
++                chunks[classify_chunk(request.is_prefill_chunk, end, target)] += 1
++            else:
++                decode_requests += 1
++                decode_tokens += num_tokens
++        prefix_after = _prefix_snapshot(scheduler)
++        block_pool = scheduler.kv_cache_manager.block_pool
++        total_blocks = block_pool.num_gpu_blocks - 1
++        free_blocks = block_pool.get_num_free_blocks()
++        assert 0 <= free_blocks <= total_blocks
++        chunks["tokens"] = prefill_tokens
++        chunks["chunk_size_hist"] = chunk_size_hist
++        values = dict(
++            schema=SCHEMA_VERSION,
++            engine_id=self.engine_id,
++            step_index=self._next_step,
++            submit_wall_ns=start[0],
++            submit_mono_ns=start[1],
++            model_executed=output.total_num_scheduled_tokens > 0,
++            scheduled_requests=len(output.num_scheduled_tokens),
++            decode_batch_size=decode_requests,
++            prefill_requests=prefill_requests,
++            prefill_tokens=prefill_tokens,
++            decode_tokens=decode_tokens,
++            chunked_prefill=chunks,
++            context_length_hist=context_length_hist,
++            preemptions=len(output.preempted_req_ids or ()),
++            queues=dict(
++                running=len(scheduler.running),
++                waiting=len(scheduler.waiting),
++                deferred=len(scheduler.skipped_waiting),
++            ),
++            kv=dict(
++                total_blocks=total_blocks,
++                free_blocks=free_blocks,
++                used_blocks=total_blocks - free_blocks,
++                usage=scheduler.kv_cache_manager.usage,
++            ),
++            prefix=dict(
++                local=_prefix_delta(start[2]["local"], prefix_after["local"]),
++                external=_prefix_delta(start[2]["external"], prefix_after["external"]),
++            ),
++        )
++        assert prefill_tokens + decode_tokens == output.total_num_scheduled_tokens
++        self._pending[key] = values
++        self._next_step += 1
++
++    def finalize(self, output: Any, cudagraph_stat: Any | None) -> bool:
++        try:
++            values = self._pending.pop(id(output))
++        except KeyError:
++            raise AssertionError("missing or already finalized OpProf step") from None
++        if cudagraph_stat is None:
++            assert not values["model_executed"]
++            cudagraph = dict(
++                hit=False,
++                runtime_mode="NONE",
++                unpadded_tokens=0,
++                bucket_tokens=0,
++                padding_tokens=0,
++            )
++        else:
++            mode = str(cudagraph_stat.runtime_mode).rsplit(".", 1)[-1]
++            cudagraph = dict(
++                hit=mode != "NONE",
++                runtime_mode=mode,
++                unpadded_tokens=cudagraph_stat.num_unpadded_tokens,
++                bucket_tokens=cudagraph_stat.num_padded_tokens,
++                padding_tokens=cudagraph_stat.num_paddings,
++            )
++        record = dict(
++            values,
++            complete_mono_ns=time.monotonic_ns(),
++            cudagraph=cudagraph,
++            moe_expert_load=None,
++            dropped_records_before=0,
++        )
++        return self.writer.submit(record)
++
++    def close(self) -> None:
++        self.writer.close()
++
++    @property
++    def failed(self) -> bool:
++        return self.writer.failed
++
++    @property
++    def failure(self) -> Exception | None:
++        return self.writer.failure
+diff --git a/vllm/v1/worker/gpu_model_runner.py b/vllm/v1/worker/gpu_model_runner.py
+index 74938a8..c11d773 100644
+--- a/vllm/v1/worker/gpu_model_runner.py
++++ b/vllm/v1/worker/gpu_model_runner.py
+@@ -437,6 +437,7 @@ class GPUModelRunner(
+         self.scheduler_config = vllm_config.scheduler_config
+         self.speculative_config = vllm_config.speculative_config
+         self.observability_config = vllm_config.observability_config
++        self.opprof_enabled = bool(envs.VLLM_OPPROF_DIR)
+ 
+         model_config = self.model_config
+         cache_config = self.cache_config
+@@ -3917,7 +3918,10 @@ class GPUModelRunner(
+                 assert batch_descriptor.num_tokens == num_tokens_padded
+ 
+         cudagraph_stats = None
+-        if self.vllm_config.observability_config.cudagraph_metrics:
++        if (
++            self.vllm_config.observability_config.cudagraph_metrics
++            or self.opprof_enabled
++        ):
+             cudagraph_stats = CUDAGraphStat(
+                 num_unpadded_tokens=num_tokens,
+                 num_padded_tokens=batch_descriptor.num_tokens,
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0002-Add-standalone-OpProf-telemetry-tests.patch

@@ -0,0 +1,417 @@
+From 4f4ee674f217698436b00c3ab6357f59a792477a Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sat, 11 Jul 2026 17:29:02 +0800
+Subject: [PATCH 2/5] Add standalone OpProf telemetry tests
+
+Assisted-by: OpenAI Codex
+---
+ tests/v1/core/test_opprof.py | 397 +++++++++++++++++++++++++++++++++++
+ 1 file changed, 397 insertions(+)
+ create mode 100644 tests/v1/core/test_opprof.py
+
+diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
+new file mode 100644
+index 0000000..9bfbfcc
+--- /dev/null
++++ b/tests/v1/core/test_opprof.py
+@@ -0,0 +1,397 @@
++# SPDX-License-Identifier: Apache-2.0
++# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
++"""Standalone tests: this file intentionally does not import the vllm package."""
++
++import errno
++import importlib.util
++import logging
++import sys
++import threading
++from pathlib import Path
++from types import SimpleNamespace
++
++import msgspec
++import pytest
++
++_ROOT = Path(__file__).parents[3]
++_SPEC = importlib.util.spec_from_file_location(
++    "opprof_standalone", _ROOT / "vllm" / "v1" / "opprof.py"
++)
++assert _SPEC is not None and _SPEC.loader is not None
++opprof = importlib.util.module_from_spec(_SPEC)
++sys.modules[_SPEC.name] = opprof
++_SPEC.loader.exec_module(opprof)
++
++
++class CachedRequests:
++    def __init__(self, context_ids=()):
++        self.context_ids = set(context_ids)
++
++    def is_context_phase(self, req_id):
++        return req_id in self.context_ids
++
++
++def prefix_stats(**overrides):
++    values = dict.fromkeys(opprof._PREFIX_FIELDS, 0)
++    values.update(overrides)
++    return SimpleNamespace(**values)
++
++
++def request(computed, total, was_chunk=False, placeholders=0):
++    return SimpleNamespace(
++        num_computed_tokens=computed,
++        num_tokens=total,
++        num_output_placeholders=placeholders,
++        is_prefill_chunk=was_chunk,
++    )
++
++
++def scheduler(requests, local=None, external=None):
++    block_pool = SimpleNamespace(
++        num_gpu_blocks=101,
++        get_num_free_blocks=lambda: 40,
++    )
++    kv_manager = SimpleNamespace(
++        prefix_cache_stats=local or prefix_stats(),
++        block_pool=block_pool,
++        usage=0.6,
++    )
++    return SimpleNamespace(
++        requests=requests,
++        kv_cache_manager=kv_manager,
++        connector_prefix_cache_stats=external,
++        running=list(range(len(requests))),
++        waiting=[0, 1],
++        skipped_waiting=[0],
++    )
++
++
++def schedule_output(tokens, context_ids=(), new_ids=(), preempted=()):
++    return SimpleNamespace(
++        scheduled_new_reqs=[SimpleNamespace(req_id=req_id) for req_id in new_ids],
++        scheduled_cached_reqs=CachedRequests(context_ids),
++        num_scheduled_tokens=tokens,
++        total_num_scheduled_tokens=sum(tokens.values()),
++        preempted_req_ids=set(preempted),
++    )
++
++
++def graph(mode="FULL", unpadded=1, padded=1):
++    return SimpleNamespace(
++        runtime_mode=mode,
++        num_unpadded_tokens=unpadded,
++        num_padded_tokens=padded,
++        num_paddings=padded - unpadded,
++    )
++
++
++def recorder(tmp_path, *, capacity=8192, start=True):
++    path = tmp_path / "opprof.jsonl"
++    writer = opprof.JSONLWriter(path, capacity=capacity, start=start)
++    return opprof.OpProfRecorder("dp0-pid1", writer), path
++
++
++def emit(rec, sched, output, cg=None):
++    start = rec.capture_start(sched)
++    rec.begin(sched, output, start)
++    return rec.finalize(output, cg or graph())
++
++
++def read_jsonl(path):
++    return [msgspec.json.decode(line) for line in path.read_bytes().splitlines()]
++
++
++def test_import_light_and_approved_constants():
++    assert "torch" not in sys.modules
++    assert "vllm" not in sys.modules
++    assert opprof.DEFAULT_QUEUE_CAPACITY == 8192
++    assert tuple(1 << i for i in range(7, 18)) == opprof.CONTEXT_LENGTH_EDGES
++    assert tuple(1 << i for i in range(4, 12)) == opprof.CHUNK_SIZE_EDGES
++
++
++def test_schema_and_invariants(tmp_path):
++    sched = scheduler(
++        {
++            "first": request(0, 100),
++            "final": request(64, 100, was_chunk=True),
++            "decode": request(1024, 1025),
++        }
++    )
++    output = schedule_output(
++        {"first": 64, "final": 36, "decode": 1},
++        context_ids={"final"},
++        new_ids={"first"},
++        preempted={"old"},
++    )
++    rec, path = recorder(tmp_path)
++    start = rec.capture_start(sched)
++    sched.kv_cache_manager.prefix_cache_stats = prefix_stats(
++        requests=1, queries=100, hits=64
++    )
++    rec.begin(sched, output, start)
++    assert rec.finalize(output, graph("FULL", 101, 128))
++    rec.close()
++
++    record, footer = read_jsonl(path)
++    assert record["schema"] == 1
++    assert record["scheduled_requests"] == 3
++    assert record["prefill_requests"] == 2
++    assert record["decode_batch_size"] == 1
++    assert record["prefill_tokens"] + record["decode_tokens"] == 101
++    assert sum(record["context_length_hist"]) == 3
++    assert len(record["context_length_hist"]) == 12
++    assert sum(record["chunked_prefill"]["chunk_size_hist"]) == 2
++    assert len(record["chunked_prefill"]["chunk_size_hist"]) == 9
++    assert record["chunked_prefill"]["first"] == 1
++    assert record["chunked_prefill"]["final"] == 1
++    assert record["preemptions"] == 1
++    assert record["kv"] == {
++        "total_blocks": 100,
++        "free_blocks": 40,
++        "used_blocks": 60,
++        "usage": 0.6,
++    }
++    assert record["prefix"]["local"]["hits"] == 64
++    assert record["moe_expert_load"] is None
++    assert record["complete_mono_ns"] >= record["submit_mono_ns"]
++    assert footer["record_type"] == "footer"
++    assert footer["written_records"] == 1
++
++
++def test_capture_record_matches_pre_refactor_golden(tmp_path, monkeypatch):
++    sched = scheduler(
++        {
++            "edge": request(0, 128),
++            "after": request(65, 129, was_chunk=True),
++            "decode": request(256, 257),
++        }
++    )
++    output = schedule_output(
++        {"edge": 128, "after": 64, "decode": 1},
++        context_ids={"after"},
++        new_ids={"edge"},
++        preempted={"old"},
++    )
++    rec, path = recorder(tmp_path)
++    zero_prefix = dict.fromkeys(opprof._PREFIX_FIELDS, 0)
++    start = (100, 200, {"local": zero_prefix, "external": None})
++    monkeypatch.setattr(opprof.time, "monotonic_ns", lambda: 300)
++
++    rec.begin(sched, output, start)
++    assert rec.finalize(output, graph("FULL", 193, 256))
++    rec.close()
++
++    record = read_jsonl(path)[0]
++    assert record == {
++        "schema": 1,
++        "engine_id": "dp0-pid1",
++        "step_index": 0,
++        "submit_wall_ns": 100,
++        "submit_mono_ns": 200,
++        "model_executed": True,
++        "scheduled_requests": 3,
++        "decode_batch_size": 1,
++        "prefill_requests": 2,
++        "prefill_tokens": 192,
++        "decode_tokens": 1,
++        "chunked_prefill": {
++            "first": 0,
++            "middle": 0,
++            "final": 1,
++            "unsplit": 1,
++            "tokens": 192,
++            "chunk_size_hist": [0, 0, 1, 1, 0, 0, 0, 0, 0],
++        },
++        "context_length_hist": [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
++        "preemptions": 1,
++        "queues": {"running": 3, "waiting": 2, "deferred": 1},
++        "kv": {
++            "total_blocks": 100,
++            "free_blocks": 40,
++            "used_blocks": 60,
++            "usage": 0.6,
++        },
++        "prefix": {"local": zero_prefix, "external": None},
++        "complete_mono_ns": 300,
++        "cudagraph": {
++            "hit": True,
++            "runtime_mode": "FULL",
++            "unpadded_tokens": 193,
++            "bucket_tokens": 256,
++            "padding_tokens": 63,
++        },
++        "moe_expert_load": None,
++        "dropped_records_before": 0,
++    }
++
++
++@pytest.mark.parametrize(
++    ("was_chunk", "end", "target", "expected"),
++    [
++        (False, 64, 100, "first"),
++        (True, 80, 100, "middle"),
++        (True, 100, 100, "final"),
++        (False, 100, 100, "unsplit"),
++    ],
++)
++def test_chunk_classification(was_chunk, end, target, expected):
++    assert opprof.classify_chunk(was_chunk, end, target) == expected
++
++
++def test_async_pairing_out_of_order_and_double_finalize(tmp_path):
++    sched = scheduler({"a": request(10, 11), "b": request(200, 201)})
++    first = schedule_output({"a": 1})
++    second = schedule_output({"b": 1})
++    rec, path = recorder(tmp_path)
++    rec.begin(sched, first, rec.capture_start(sched))
++    rec.begin(sched, second, rec.capture_start(sched))
++    assert rec.finalize(second, graph())
++    assert rec.finalize(first, graph("NONE"))
++    with pytest.raises(AssertionError, match="already finalized"):
++        rec.finalize(second, graph())
++    assert not rec._pending
++    rec.close()
++    records = read_jsonl(path)[:-1]
++    assert [record["step_index"] for record in records] == [1, 0]
++    assert records[0]["context_length_hist"][1] == 1
++    assert records[1]["context_length_hist"][0] == 1
++
++
++def test_disabled_noop_and_log_stats_fail_fast(tmp_path):
++    assert opprof.OpProfRecorder.create("", dp_rank=0, log_stats=False) is None
++    with pytest.raises(ValueError, match="requires log stats"):
++        opprof.OpProfRecorder.create(str(tmp_path), dp_rank=0, log_stats=False)
++    assert not list(tmp_path.iterdir())
++
++
++def test_bounded_queue_drop_accounting(tmp_path):
++    sched = scheduler({str(i): request(i, i + 1) for i in range(3)})
++    rec, path = recorder(tmp_path, capacity=1, start=False)
++    assert emit(rec, sched, schedule_output({"0": 1}))
++    assert not emit(rec, sched, schedule_output({"1": 1}))
++    rec.writer.start()
++    rec.writer._queue.join()
++    assert emit(rec, sched, schedule_output({"2": 1}))
++    rec.close()
++
++    first, after_drop, footer = read_jsonl(path)
++    assert first["step_index"] == 0
++    assert after_drop["step_index"] == 2
++    assert after_drop["dropped_records_before"] == 1
++    assert footer["encoded_records"] == 3
++    assert footer["written_records"] == 2
++    assert footer["dropped_records"] == 1
++
++
++def test_writer_enospc_is_exposed_and_shutdown_is_bounded(
++    tmp_path, monkeypatch, caplog
++):
++    sched = scheduler(
++        {
++            "first": request(0, 1),
++            "after_failure": request(1, 2),
++        }
++    )
++    rec, _ = recorder(tmp_path, capacity=1, start=False)
++    assert emit(rec, sched, schedule_output({"first": 1}))
++
++    real_file = rec.writer._file
++
++    def fail_enospc(*_args, **_kwargs):
++        raise OSError(errno.ENOSPC, "No space left on device")
++
++    failing_file = SimpleNamespace(
++        write=fail_enospc,
++        flush=fail_enospc,
++        close=real_file.close,
++    )
++    monkeypatch.setattr(rec.writer, "_file", failing_file)
++    caplog.set_level(logging.ERROR, logger=opprof.__name__)
++
++    rec.writer.start()
++    rec.writer._thread.join(timeout=1.0)
++    assert not rec.writer._thread.is_alive()
++
++    producer_result = emit(
++        rec, sched, schedule_output({"after_failure": 1})
++    )
++
++    closer = threading.Thread(target=rec.close, daemon=True)
++    closer.start()
++    closer.join(timeout=1.0)
++    assert not closer.is_alive(), "OpProf close blocked after writer failure"
++    assert not producer_result
++    assert rec.writer.dropped_records == 1
++    assert rec.failed
++    assert isinstance(rec.failure, OSError)
++    assert rec.failure.errno == errno.ENOSPC
++    errors = [
++        record
++        for record in caplog.records
++        if "OpProf writer failed" in record.getMessage()
++    ]
++    assert len(errors) == 1
++
++
++def test_shutdown_flush_is_idempotent(tmp_path):
++    sched = scheduler({"decode": request(8, 9)})
++    rec, path = recorder(tmp_path)
++    assert emit(rec, sched, schedule_output({"decode": 1}))
++    rec.close()
++    rec.close()
++    record, footer = read_jsonl(path)
++    assert record["step_index"] == 0
++    assert footer["written_records"] == 1
++    assert path.stat().st_size > 0
++
++
++def test_piecewise_cudagraph_record_preserved(tmp_path):
++    sched = scheduler({"decode": request(4096, 4097)})
++    output = schedule_output({"decode": 1})
++    rec, path = recorder(tmp_path)
++    assert emit(rec, sched, output, graph("PIECEWISE", 513, 520))
++    rec.close()
++    record = read_jsonl(path)[0]
++    assert record["cudagraph"] == {
++        "hit": True,
++        "runtime_mode": "PIECEWISE",
++        "unpadded_tokens": 513,
++        "bucket_tokens": 520,
++        "padding_tokens": 7,
++    }
++
++
++def test_zero_scheduled_tokens_finalize_without_cudagraph(tmp_path):
++    sched = scheduler({})
++    output = schedule_output({})
++    rec, path = recorder(tmp_path)
++    rec.begin(sched, output, rec.capture_start(sched))
++    assert rec._pending
++
++    assert rec.finalize(output, cudagraph_stat=None)
++    assert not rec._pending
++    rec.close()
++
++    record = read_jsonl(path)[0]
++    assert record["model_executed"] is False
++    assert record["scheduled_requests"] == 0
++    assert record["prefill_requests"] == 0
++    assert record["prefill_tokens"] == 0
++    assert record["decode_batch_size"] == 0
++    assert record["decode_tokens"] == 0
++    assert record["context_length_hist"] == [0] * 12
++    assert record["chunked_prefill"] == {
++        "first": 0,
++        "middle": 0,
++        "final": 0,
++        "unsplit": 0,
++        "tokens": 0,
++        "chunk_size_hist": [0] * 9,
++    }
++    assert record["cudagraph"] == {
++        "hit": False,
++        "runtime_mode": "NONE",
++        "unpadded_tokens": 0,
++        "bucket_tokens": 0,
++        "padding_tokens": 0,
++    }
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0003-Log-the-OpProf-output-path-at-startup.patch

@@ -0,0 +1,52 @@
+From 668cfb7e27e488454dbf09a4927b8a60d6d49b40 Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sat, 11 Jul 2026 17:32:27 +0800
+Subject: [PATCH 3/5] Log the OpProf output path at startup
+
+Assisted-by: OpenAI Codex
+---
+ tests/v1/core/test_opprof.py    | 1 +
+ vllm/v1/core/sched/scheduler.py | 2 ++
+ vllm/v1/opprof.py               | 1 +
+ 3 files changed, 4 insertions(+)
+
+diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
+index 9bfbfcc..79c1fae 100644
+--- a/tests/v1/core/test_opprof.py
++++ b/tests/v1/core/test_opprof.py
+@@ -336,6 +336,7 @@ def test_writer_enospc_is_exposed_and_shutdown_is_bounded(
+ def test_shutdown_flush_is_idempotent(tmp_path):
+     sched = scheduler({"decode": request(8, 9)})
+     rec, path = recorder(tmp_path)
++    assert rec.writer.path == path
+     assert emit(rec, sched, schedule_output({"decode": 1}))
+     rec.close()
+     rec.close()
+diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py
+index 303c562..769a02a 100644
+--- a/vllm/v1/core/sched/scheduler.py
++++ b/vllm/v1/core/sched/scheduler.py
+@@ -278,6 +278,8 @@ class Scheduler(SchedulerInterface):
+             dp_rank=self.parallel_config.data_parallel_index,
+             log_stats=self.log_stats,
+         )
++        if self.opprof is not None:
++            logger.info("OpProf telemetry enabled: %s", self.opprof.writer.path)
+ 
+         self.use_pp = self.parallel_config.pipeline_parallel_size > 1
+         self.use_v2_model_runner = vllm_config.use_v2_model_runner
+diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py
+index f0330d0..75f63de 100644
+--- a/vllm/v1/opprof.py
++++ b/vllm/v1/opprof.py
+@@ -68,6 +68,7 @@ class JSONLWriter:
+         start: bool = True,
+     ) -> None:
+         self._queue: queue.Queue[Any] = queue.Queue(capacity)
++        self.path = path
+         self._encoder = msgspec.json.Encoder()
+         self._file = path.open("xb", buffering=1 << 20)
+         self._thread = threading.Thread(target=self._run, daemon=True)
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0004-Exclude-OpProf-output-path-from-compile-cache-key.patch

@@ -0,0 +1,64 @@
+From 335da4abe60e0177872e0b7751e86eeec6756a2b Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sat, 11 Jul 2026 22:32:30 +0800
+Subject: [PATCH 4/5] Exclude OpProf output path from compile cache key
+
+---
+ tests/v1/core/test_opprof.py | 21 ++++++++++++++++++++-
+ vllm/envs.py                 |  1 +
+ 2 files changed, 21 insertions(+), 1 deletion(-)
+
+diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
+index 79c1fae..a820e7e 100644
+--- a/tests/v1/core/test_opprof.py
++++ b/tests/v1/core/test_opprof.py
+@@ -8,7 +8,7 @@ import logging
+ import sys
+ import threading
+ from pathlib import Path
+-from types import SimpleNamespace
++from types import ModuleType, SimpleNamespace
+ 
+ import msgspec
+ import pytest
+@@ -23,6 +23,25 @@ sys.modules[_SPEC.name] = opprof
+ _SPEC.loader.exec_module(opprof)
+ 
+ 
++def test_output_dir_is_not_a_compile_factor(monkeypatch: pytest.MonkeyPatch):
++    spec = importlib.util.spec_from_file_location(
++        "envs_standalone", _ROOT / "vllm" / "envs.py"
++    )
++    assert spec is not None and spec.loader is not None
++    envs = importlib.util.module_from_spec(spec)
++    monkeypatch.setitem(sys.modules, spec.name, envs)
++    spec.loader.exec_module(envs)
++
++    monkeypatch.setitem(sys.modules, "vllm", ModuleType("vllm"))
++    monkeypatch.setitem(sys.modules, "vllm.config", ModuleType("vllm.config"))
++    config_utils = ModuleType("vllm.config.utils")
++    config_utils.__dict__["normalize_value"] = lambda value: value
++    monkeypatch.setitem(sys.modules, "vllm.config.utils", config_utils)
++    monkeypatch.setenv("VLLM_OPPROF_DIR", "/tmp/opprof")
++
++    assert "VLLM_OPPROF_DIR" not in envs.compile_factors()
++
++
+ class CachedRequests:
+     def __init__(self, context_ids=()):
+         self.context_ids = set(context_ids)
+diff --git a/vllm/envs.py b/vllm/envs.py
+index b3093e9..5634708 100755
+--- a/vllm/envs.py
++++ b/vllm/envs.py
+@@ -2044,6 +2044,7 @@ def compile_factors() -> dict[str, object]:
+         "VLLM_LOGGING_CONFIG_PATH",
+         "VLLM_LOGGING_COLOR",
+         "VLLM_LOG_STATS_INTERVAL",
++        "VLLM_OPPROF_DIR",
+         "VLLM_DEBUG_LOG_API_SERVER_RESPONSE",
+         "VLLM_TUNED_CONFIG_FOLDER",
+         "VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR",
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0005-Keep-compile-factor-regression-import-light.patch

@@ -0,0 +1,24 @@
+From bbfa7176a6a3686a88ee66696f1ad8d754559d96 Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sat, 11 Jul 2026 22:38:08 +0800
+Subject: [PATCH 5/5] Keep compile-factor regression import-light
+
+---
+ tests/v1/core/test_opprof.py | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
+index a820e7e..0b8a8a1 100644
+--- a/tests/v1/core/test_opprof.py
++++ b/tests/v1/core/test_opprof.py
+@@ -37,6 +37,7 @@ def test_output_dir_is_not_a_compile_factor(monkeypatch: pytest.MonkeyPatch):
+     config_utils = ModuleType("vllm.config.utils")
+     config_utils.__dict__["normalize_value"] = lambda value: value
+     monkeypatch.setitem(sys.modules, "vllm.config.utils", config_utils)
++    monkeypatch.setitem(sys.modules, "torch", ModuleType("torch"))
+     monkeypatch.setenv("VLLM_OPPROF_DIR", "/tmp/opprof")
+ 
+     assert "VLLM_OPPROF_DIR" not in envs.compile_factors()
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0006-Checkpoint-OpProf-accounting-across-hard-kills.patch

@@ -0,0 +1,306 @@
+From f8b68f2452c424d22de4a69527a427207dcfbca5 Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sun, 12 Jul 2026 12:56:39 +0800
+Subject: [PATCH] Checkpoint OpProf accounting across hard kills
+
+---
+ tests/v1/core/test_opprof.py | 118 +++++++++++++++++++++++++++++++++++
+ vllm/v1/opprof.py            |  84 ++++++++++++++++++++++---
+ 2 files changed, 195 insertions(+), 7 deletions(-)
+
+diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
+index 0b8a8a1..007c5bb 100644
+--- a/tests/v1/core/test_opprof.py
++++ b/tests/v1/core/test_opprof.py
+@@ -5,6 +5,8 @@
+ import errno
+ import importlib.util
+ import logging
++import os
++import subprocess
+ import sys
+ import threading
+ from pathlib import Path
+@@ -121,6 +123,12 @@ def read_jsonl(path):
+     return [msgspec.json.decode(line) for line in path.read_bytes().splitlines()]
+ 
+ 
++def read_sidecar(path):
++    return msgspec.json.decode(
++        path.with_name(f"{path.name}.footer.json").read_bytes()
++    )
++
++
+ def test_import_light_and_approved_constants():
+     assert "torch" not in sys.modules
+     assert "vllm" not in sys.modules
+@@ -366,6 +374,116 @@ def test_shutdown_flush_is_idempotent(tmp_path):
+     assert path.stat().st_size > 0
+ 
+ 
++def test_sidecar_updates_are_atomic(tmp_path, monkeypatch):
++    writer = opprof.JSONLWriter(tmp_path / "atomic.jsonl", start=False)
++    writer._write_sidecar(
++        encoded_records=1,
++        written_records=1,
++        dropped_records=0,
++        last_step_index=0,
++        final=False,
++    )
++    original = writer.sidecar_path.read_bytes()
++    real_replace = os.replace
++    replacements = []
++
++    def inspect_replace(source, destination):
++        assert Path(destination) == writer.sidecar_path
++        assert writer.sidecar_path.read_bytes() == original
++        candidate = msgspec.json.decode(Path(source).read_bytes())
++        assert candidate["written_records"] == 2
++        replacements.append(candidate)
++        real_replace(source, destination)
++
++    monkeypatch.setattr(opprof.os, "replace", inspect_replace)
++    writer._write_sidecar(
++        encoded_records=3,
++        written_records=2,
++        dropped_records=1,
++        last_step_index=2,
++        final=False,
++    )
++    monkeypatch.setattr(opprof.os, "replace", real_replace)
++
++    assert len(replacements) == 1
++    assert read_sidecar(writer.path)["encoded_records"] == 3
++    assert not list(tmp_path.glob(".*.tmp-*"))
++    writer.close()
++
++
++def test_hard_kill_sidecar_balances_last_flush(tmp_path):
++    path = tmp_path / "hard-kill.jsonl"
++    child = """
++import importlib.util
++import sys
++import time
++from pathlib import Path
++
++source, output = sys.argv[1:]
++spec = importlib.util.spec_from_file_location("opprof_hard_kill", source)
++module = importlib.util.module_from_spec(spec)
++sys.modules[spec.name] = module
++spec.loader.exec_module(module)
++writer = module.JSONLWriter(Path(output))
++for step in range(3):
++    assert writer.submit({"schema": 1, "step_index": step})
++writer._queue.join()
++while not writer.sidecar_path.exists():
++    time.sleep(0.01)
++print("READY", flush=True)
++time.sleep(60)
++"""
++    process = subprocess.Popen(
++        [
++            sys.executable,
++            "-c",
++            child,
++            str(_ROOT / "vllm/v1/opprof.py"),
++            str(path),
++        ],
++        stdout=subprocess.PIPE,
++        text=True,
++    )
++    try:
++        assert process.stdout is not None
++        assert process.stdout.readline().strip() == "READY"
++        process.kill()
++        assert process.wait(timeout=5) < 0
++    finally:
++        if process.poll() is None:
++            process.kill()
++            process.wait(timeout=5)
++
++    records = read_jsonl(path)
++    sidecar = read_sidecar(path)
++    assert len(records) == 3
++    assert all(record.get("record_type") != "footer" for record in records)
++    assert sidecar["final"] is False
++    assert sidecar["written_records"] == len(records)
++    assert sidecar["encoded_records"] == (
++        sidecar["written_records"] + sidecar["dropped_records"]
++    )
++    assert sidecar["last_step_index"] == records[-1]["step_index"] == 2
++
++
++def test_clean_footer_and_final_sidecar_agree(tmp_path):
++    sched = scheduler({"decode": request(8, 9)})
++    rec, path = recorder(tmp_path)
++    assert emit(rec, sched, schedule_output({"decode": 1}))
++    rec.close()
++
++    record, footer = read_jsonl(path)
++    sidecar = read_sidecar(path)
++    assert sidecar["record_type"] == "footer_checkpoint"
++    assert sidecar["stream"] == path.name
++    assert sidecar["final"] is True
++    assert sidecar["last_step_index"] == record["step_index"] == 0
++    assert sidecar["checkpoint_wall_ns"] > 0
++    assert sidecar["flush_interval_seconds"] == opprof.FLUSH_INTERVAL_SECONDS
++    for counter in ("encoded_records", "written_records", "dropped_records"):
++        assert sidecar[counter] == footer[counter]
++
++
+ def test_piecewise_cudagraph_record_preserved(tmp_path):
+     sched = scheduler({"decode": request(4096, 4097)})
+     output = schedule_output({"decode": 1})
+diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py
+index 75f63de..28d9635 100644
+--- a/vllm/v1/opprof.py
++++ b/vllm/v1/opprof.py
+@@ -7,6 +7,7 @@ import queue
+ import threading
+ import time
+ from bisect import bisect_left
++from contextlib import suppress
+ from pathlib import Path
+ from typing import Any
+ 
+@@ -18,6 +19,7 @@ SCHEMA_VERSION = 1
+ CONTEXT_LENGTH_EDGES = tuple(1 << exponent for exponent in range(7, 18))
+ CHUNK_SIZE_EDGES = tuple(1 << exponent for exponent in range(4, 12))
+ DEFAULT_QUEUE_CAPACITY = 8192
++FLUSH_INTERVAL_SECONDS = 1.0
+ _CLOSE_TIMEOUT_SECONDS = 1.0
+ _STOP = object()
+ _PREFIX_FIELDS = (  # noqa: SIM905
+@@ -69,6 +71,7 @@ class JSONLWriter:
+     ) -> None:
+         self._queue: queue.Queue[Any] = queue.Queue(capacity)
+         self.path = path
++        self.sidecar_path = path.with_name(f"{path.name}.footer.json")
+         self._encoder = msgspec.json.Encoder()
+         self._file = path.open("xb", buffering=1 << 20)
+         self._thread = threading.Thread(target=self._run, daemon=True)
+@@ -78,6 +81,9 @@ class JSONLWriter:
+         self.failure: Exception | None = None
+         self.encoded_records = self.written_records = 0
+         self.dropped_records = self._unreported_drops = 0
++        self._checkpoint_encoded_records = 0
++        self._checkpoint_dropped_records = 0
++        self._last_written_step_index: int | None = None
+         if start:
+             self.start()
+ 
+@@ -119,33 +125,90 @@ class JSONLWriter:
+         if self._writer_unavailable():
+             return self._drop(pending)
+         try:
+-            self._queue.put_nowait(payload)
++            self._queue.put_nowait(
++                (
++                    payload,
++                    self.encoded_records,
++                    self.dropped_records,
++                    int(record["step_index"]),
++                )
++            )
+         except queue.Full:
+             return self._drop(pending)
+         self._unreported_drops = 0
+         return True
+ 
++    def _write_sidecar(
++        self,
++        *,
++        encoded_records: int,
++        written_records: int,
++        dropped_records: int,
++        last_step_index: int | None,
++        final: bool,
++    ) -> None:
++        sidecar = dict(
++            schema=SCHEMA_VERSION,
++            record_type="footer_checkpoint",
++            stream=self.path.name,
++            encoded_records=encoded_records,
++            written_records=written_records,
++            dropped_records=dropped_records,
++            last_step_index=last_step_index,
++            checkpoint_wall_ns=time.time_ns(),
++            flush_interval_seconds=FLUSH_INTERVAL_SECONDS,
++            final=final,
++        )
++        temporary_path = self.sidecar_path.with_name(
++            f".{self.sidecar_path.name}.tmp-{os.getpid()}-{threading.get_ident()}"
++        )
++        try:
++            with temporary_path.open("wb") as temporary_file:
++                temporary_file.write(self._encoder.encode(sidecar) + b"\n")
++                temporary_file.flush()
++            os.replace(temporary_path, self.sidecar_path)
++        finally:
++            with suppress(FileNotFoundError):
++                temporary_path.unlink()
++
++    def _flush_checkpoint(self) -> None:
++        self._file.flush()
++        self._write_sidecar(
++            encoded_records=self._checkpoint_encoded_records,
++            written_records=self.written_records,
++            dropped_records=self._checkpoint_dropped_records,
++            last_step_index=self._last_written_step_index,
++            final=False,
++        )
++
+     def _run(self) -> None:
+         buffered = 0
+         last_flush = time.monotonic()
+         try:
+             while True:
+                 try:
+-                    item = self._queue.get(timeout=1.0)
++                    item = self._queue.get(timeout=FLUSH_INTERVAL_SECONDS)
+                 except queue.Empty:
+-                    self._file.flush()
++                    self._flush_checkpoint()
+                     buffered = 0
+                     last_flush = time.monotonic()
+                     continue
+                 try:
+                     if item is _STOP:
+                         break
+-                    self._file.write(item)
+-                    buffered += len(item)
++                    payload, encoded, dropped, step_index = item
++                    self._file.write(payload)
++                    buffered += len(payload)
+                     self.written_records += 1
++                    self._checkpoint_encoded_records = encoded
++                    self._checkpoint_dropped_records = dropped
++                    self._last_written_step_index = step_index
+                     now = time.monotonic()
+-                    if buffered >= 1 << 20 or now - last_flush >= 1.0:
+-                        self._file.flush()
++                    if (
++                        buffered >= 1 << 20
++                        or now - last_flush >= FLUSH_INTERVAL_SECONDS
++                    ):
++                        self._flush_checkpoint()
+                         buffered = 0
+                         last_flush = now
+                 finally:
+@@ -163,6 +226,13 @@ class JSONLWriter:
+             try:
+                 self._file.write(self._encoder.encode(footer) + b"\n")
+                 self._file.flush()
++                self._write_sidecar(
++                    encoded_records=footer["encoded_records"],
++                    written_records=footer["written_records"],
++                    dropped_records=footer["dropped_records"],
++                    last_step_index=self._last_written_step_index,
++                    final=True,
++                )
+             except Exception as error:
+                 self._record_failure(error)
+             finally:
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/0007-Recreate-scheduled-torch-profiler-between-windows.patch

@@ -0,0 +1,27 @@
+From 23450fb21ac255b0cf710f4ee965ee694921975d Mon Sep 17 00:00:00 2001
+From: Gahow Wang <gahow.wang@gmail.com>
+Date: Sun, 12 Jul 2026 13:12:52 +0800
+Subject: [PATCH] Recreate scheduled torch profiler between windows
+
+---
+ vllm/v1/worker/gpu_worker.py | 4 ++++
+ 1 file changed, 4 insertions(+)
+
+diff --git a/vllm/v1/worker/gpu_worker.py b/vllm/v1/worker/gpu_worker.py
+index 5e266a3..0058f96 100644
+--- a/vllm/v1/worker/gpu_worker.py
++++ b/vllm/v1/worker/gpu_worker.py
+@@ -978,6 +978,10 @@ class Worker(WorkerBase):
+                 logger.warning("Profiler was not started, nothing to stop.")
+                 return
+             self.profiler.stop()
++            # A scheduled torch.profiler.profile does not reset its schedule
++            # after stop(). Recreate it for the next /start_profile window.
++            if isinstance(self.profiler, TorchProfilerWrapper):
++                self.profiler = None
+ 
+     def execute_dummy_batch(self) -> None:
+         num_tokens = getattr(self.model_runner, "uniform_decode_query_len", 1)
+-- 
+2.43.0
+

patches/vllm-0.24.0-opprof/README.md

@@ -0,0 +1,119 @@
+# vLLM 0.24.0 OpProf patch series
+
+## Goal
+
+Apply the accepted OpProf Layer-1 instrumentation to exactly vLLM `v0.24.0`
+at base commit `ee0da84ab9e04ac7610e28580af62c365e898389`. The series adds one
+scheduler-owned composition record per step without installing new runtime
+dependencies or changing GPU kernels.
+
+## Contents
+
+- `0001-Add-lightweight-per-step-OpProf-telemetry.patch`: adds the environment
+  switch, import-light JSONL recorder/writer, scheduler hooks, and reuse of the
+  existing CUDA-graph stat. Writer failures are exposed without blocking
+  producers or shutdown, and request histograms are accumulated in-place.
+- `0002-Add-standalone-OpProf-telemetry-tests.patch`: adds CPU-only tests that
+  load the recorder directly without importing or installing vLLM or torch,
+  including ENOSPC, golden-record, and zero-token regressions.
+- `0003-Log-the-OpProf-output-path-at-startup.patch`: logs the resolved JSONL
+  output path and covers it in the standalone shutdown test.
+- `0004-Exclude-OpProf-output-path-from-compile-cache-key.patch`: prevents the
+  per-run telemetry destination from invalidating vLLM's torch.compile/AOT
+  cache and adds an import-light regression test.
+- `0005-Keep-compile-factor-regression-import-light.patch`: isolates the new
+  regression from torch in full vLLM test environments.
+- `0006-Checkpoint-OpProf-accounting-across-hard-kills.patch`: atomically
+  checkpoints balanced writer counters beside each JSONL stream once per
+  flush interval, with clean-close and hard-kill regressions.
+- `0007-Recreate-scheduled-torch-profiler-between-windows.patch`: discards a
+  stopped scheduled torch-profiler wrapper so each subsequent official profile
+  endpoint call receives a fresh 2+8 schedule and emits its own trace.
+- `apply.sh`: verifies the exact base, refuses dirty/wrong revisions, applies
+  all numbered patches with `git am`, and exits successfully only when the
+  exact series is already applied directly on the required base.
+- `pytest-evidence.txt`: exact isolated test command, dependency versions, and
+  all-pass summary.
+
+The source branch tip used to generate the patches is
+`23450fb21ac255b0cf710f4ee965ee694921975d` (`opprof`).
+
+## Apply
+
+Prerequisite: a clean checkout whose `HEAD` is the exact base commit.
+
+```bash
+./patches/vllm-0.24.0-opprof/apply.sh /path/to/vllm-v0.24.0
+```
+
+Running the command again is a no-op only when the five matching patch commits
+are rooted directly at the required base. A partially applied series, dirty
+tree, unrelated commit, or any other `HEAD` is rejected instead of being
+guessed around.
+
+## Enable and output
+
+Set an absolute output directory before starting vLLM:
+
+```bash
+export VLLM_OPPROF_DIR=/absolute/path/to/run/opprof
+```
+
+Unset or empty disables the feature before recorder construction. Combining it
+with `--disable-log-stats` fails fast, as approved.
+
+Each EngineCore/DP scheduler writes one file named approximately
+`opprof-v1-dp0-pid1234-<start_ns>.jsonl`. Records contain schema/engine/step and
+timestamps; scheduled prefill/decode composition; first/middle/final/unsplit
+prefill chunks; 12-bin context and 9-bin chunk-size histograms; preemptions;
+running/waiting/deferred queues; KV blocks/usage; local/external prefix deltas;
+CUDA-graph hit/mode/bucket/padding; explicit null Layer-1 MoE load; and drop-gap
+accounting. A clean close writes a final writer-count footer in the stream.
+
+Every JSONL flush also atomically replaces
+`<stream>.footer.json` through a same-directory temporary file. The sidecar
+contains the encoded, written, and dropped counts through that durable flush,
+the last written step index, a wall-clock timestamp, the one-second flush
+interval, and whether it is final. Queue entries carry their submission
+ordinal and cumulative drops, so a periodic checkpoint always satisfies
+`encoded = written + dropped` without decoding records in the writer thread.
+On clean close the in-stream footer is authoritative and the final sidecar must
+agree with all three counters. If a hard kill prevents the in-stream footer,
+the latest sidecar is authoritative: the decoded data-line count must equal
+its `written_records`, its final data-line step must equal
+`last_step_index`, and its counters must balance. Data after that checkpoint
+may be lost, bounded by at most the configured one-second flush interval.
+
+The bounded queue holds 8192 encoded records. Producers never wait for disk;
+full queues or a failed writer drop the new record and report the gap on the
+next successful record. A writer I/O failure is exposed through recorder state,
+logged once, and cannot make shutdown wait indefinitely. The writer flushes at
+1 MiB, one second, or shutdown.
+
+## Test
+
+Only pytest and msgspec are required. `--confcutdir` prevents vLLM's global
+test configuration from importing its full dependency stack.
+
+```bash
+cd /path/to/vllm-v0.24.0
+uv run --no-project --with pytest --with msgspec \
+  pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q
+```
+
+Expected summary:
+
+```text
+18 passed in 1.09s
+```
+
+## Caveats
+
+- Layer 1 intentionally records no expert-load arrays. Exact routed experts
+  remain a separate Layer-2 run.
+- `PIECEWISE` means graph-wrapped compiled regions, not full-step graph replay.
+- Phase 2 must measure the always-on overhead; acceptance requires the upper
+  bound of the 95% confidence interval to remain below 3% for every primary
+  serving metric.
+- Primary campaign topology is TP1 on community BF16 Qwen3-30B-A3B, with TP2
+  and TP4 counterpoints. Record the selected MoE backend log every run.

patches/vllm-0.24.0-opprof/apply.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+base_commit=ee0da84ab9e04ac7610e28580af62c365e898389
+script_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
+repo=${1:-.}
+patches=("$script_dir"/0*.patch)
+
+git -C "$repo" rev-parse --git-dir >/dev/null
+if [[ -n $(git -C "$repo" status --porcelain) ]]; then
+    echo "Refusing to apply to a dirty worktree." >&2
+    exit 1
+fi
+if ((${#patches[@]} == 0)) || [[ ! -f ${patches[0]} ]]; then
+    echo "No numbered patch files found in $script_dir" >&2
+    exit 1
+fi
+
+head_commit=$(git -C "$repo" rev-parse HEAD)
+mapfile -t recent_commits < <(
+    git -C "$repo" rev-list --max-count="${#patches[@]}" --reverse HEAD
+)
+patches_match=true
+((${#recent_commits[@]} == ${#patches[@]})) || patches_match=false
+for i in "${!patches[@]}"; do
+    $patches_match || break
+    patch_id=$(git patch-id --stable <"${patches[$i]}" | awk '{print $1}')
+    commit_id=$(
+        git -C "$repo" show --pretty=format: "${recent_commits[$i]}" |
+            git patch-id --stable | awk '{print $1}'
+    )
+    [[ $patch_id == "$commit_id" ]] || patches_match=false
+done
+first_parent=""
+if ((${#recent_commits[@]} > 0)); then
+    first_parent=$(
+        git -C "$repo" rev-parse --verify "${recent_commits[0]}^" 2>/dev/null || true
+    )
+fi
+if $patches_match && [[ $first_parent == "$base_commit" ]]; then
+    echo "OpProf patch series is already applied."
+    exit 0
+fi
+
+if [[ $head_commit == "$base_commit" ]]; then
+    git -C "$repo" am "${patches[@]}"
+    echo "Applied OpProf patch series to $repo"
+    exit 0
+fi
+if $patches_match; then
+    echo "Refusing to treat OpProf patches as already applied:" >&2
+    echo "first patch parent is $first_parent, expected $base_commit" >&2
+    exit 1
+fi
+
+echo "Refusing to apply: HEAD is $head_commit; expected $base_commit" >&2
+echo "or the exact OpProf patch series rooted at that base." >&2
+exit 1

patches/vllm-0.24.0-opprof/pytest-evidence.txt

@@ -0,0 +1,16 @@
+OpProf standalone pytest evidence
+Date: 2026-07-12
+Source branch: opprof
+Source tip: 23450fb21ac255b0cf710f4ee965ee694921975d
+Base: ee0da84ab9e04ac7610e28580af62c365e898389 (v0.24.0)
+Environment: Python 3.11.13, pytest 9.1.1, msgspec 0.21.1
+vLLM installed: no
+torch installed in isolated test environment: no
+GPU/remote access: no
+
+Command:
+uv run --no-project --with pytest --with msgspec pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q
+
+Output:
+..................                                                       [100%]
+18 passed in 1.09s

runs/action-aware-v0/action_aware_client.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Add explicit MBBT/config provenance to the accepted Phase-6 replay client."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+
+PHASE6 = Path(__file__).resolve().parents[1] / "opprof-phase6"
+sys.path.insert(0, str(PHASE6))
+
+import opprof_phase6_client as base  # noqa: E402
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("command", choices=("warmup", "run-anchor"))
+    result.add_argument("--study", required=True)
+    result.add_argument("--cell", required=True)
+    result.add_argument("--anchor", type=float, required=True)
+    result.add_argument("--tp", type=int, required=True)
+    result.add_argument("--mns", type=int, required=True)
+    result.add_argument("--mbbt", type=int, required=True)
+    result.add_argument("--base-url", required=True)
+    result.add_argument("--result-dir", required=True)
+    result.add_argument("--disable-slo-early-stop", action="store_true")
+    return result
+
+
+def main() -> None:
+    args = parser().parse_args()
+    result = base.run_replay(args, warmup=args.command == "warmup")
+    result.update(
+        {
+            "schema": "action-aware-pilot-result-v0",
+            "config_id": args.cell,
+            "mbbt": args.mbbt,
+        }
+    )
+    base.atomic_json(Path(args.result_dir) / "result.json", result)
+    print(
+        json.dumps(
+            {
+                key: result[key]
+                for key in (
+                    "config_id",
+                    "mns",
+                    "mbbt",
+                    "kind",
+                    "pass_rate",
+                    "feasible",
+                )
+            },
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/action-aware-v0/analyze_pilot.py

@@ -0,0 +1,697 @@
+#!/usr/bin/env python3
+"""Audit source-only constraint signals against crossed real interventions."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+import os
+import statistics
+import sys
+from pathlib import Path
+from typing import Any, Iterable, Mapping
+
+
+HERE = Path(__file__).resolve().parent
+COMMON_STATE = HERE.parent / "telemetry-residual"
+sys.path.insert(0, str(COMMON_STATE))
+
+from common_state import summarize_engine  # noqa: E402
+
+
+SCHEMA = "action-aware-constraint-pilot-audit-v0"
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix(path.suffix + ".tmp")
+    temporary.write_text(
+        json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+    )
+    os.replace(temporary, path)
+
+
+def numeric(values: Iterable[float]) -> dict[str, Any]:
+    finite = [float(value) for value in values]
+    if not finite:
+        raise ValueError("numeric summary requires values")
+    if any(not math.isfinite(value) for value in finite):
+        raise ValueError("numeric summary received non-finite values")
+    return {
+        "n": len(finite),
+        "min": min(finite),
+        "max": max(finite),
+        "distinct_n": len(set(finite)),
+    }
+
+
+def distribution(values: Iterable[float]) -> dict[str, Any]:
+    finite = [float(value) for value in values]
+    summary = numeric(finite)
+    return {
+        **summary,
+        "mean": statistics.fmean(finite),
+        "p50": quantile(finite, 0.50),
+        "p95": quantile(finite, 0.95),
+        "p99": quantile(finite, 0.99),
+    }
+
+
+def quantile(values: Iterable[float], probability: float) -> float:
+    ordered = sorted(float(value) for value in values)
+    if not ordered:
+        raise ValueError("quantile requires values")
+    position = probability * (len(ordered) - 1)
+    lower = math.floor(position)
+    upper = math.ceil(position)
+    if lower == upper:
+        return ordered[lower]
+    weight = position - lower
+    return ordered[lower] * (1.0 - weight) + ordered[upper] * weight
+
+
+def load_jsonl(path: Path) -> list[dict[str, Any]]:
+    records = []
+    with path.open(encoding="utf-8") as source:
+        for line_number, line in enumerate(source, 1):
+            try:
+                records.append(json.loads(line))
+            except json.JSONDecodeError as error:
+                raise ValueError(f"{path}:{line_number}: invalid JSON") from error
+    return records
+
+
+def binding_summary(
+    records: list[Mapping[str, Any]], *, mns: int, mbbt: int
+) -> dict[str, Any]:
+    if not records:
+        raise ValueError("binding summary requires scheduler records")
+    counts = {
+        "mns_exclusive": 0,
+        "mbbt_exclusive": 0,
+        "both": 0,
+        "waiting_unresolved": 0,
+        "waiting": 0,
+    }
+    running_utilization = []
+    token_utilization = []
+    kv_usage = []
+    preemptions = 0
+    for record in records:
+        waiting = int(record["queues"]["waiting"]) + int(
+            record["queues"]["deferred"]
+        )
+        running = int(record["queues"]["running"])
+        scheduled_tokens = int(record["prefill_tokens"]) + int(
+            record["decode_tokens"]
+        )
+        if running > mns:
+            raise ValueError("running requests exceed configured MNS")
+        if scheduled_tokens > mbbt:
+            raise ValueError("scheduled tokens exceed configured MBBT")
+        mns_hit = waiting > 0 and running == mns
+        mbbt_hit = waiting > 0 and scheduled_tokens == mbbt
+        if waiting > 0:
+            counts["waiting"] += 1
+            if mns_hit and mbbt_hit:
+                counts["both"] += 1
+            elif mns_hit:
+                counts["mns_exclusive"] += 1
+            elif mbbt_hit:
+                counts["mbbt_exclusive"] += 1
+            else:
+                counts["waiting_unresolved"] += 1
+        running_utilization.append(running / mns)
+        token_utilization.append(scheduled_tokens / mbbt)
+        kv_usage.append(float(record["kv"]["usage"]))
+        preemptions += int(record["preemptions"])
+    count = len(records)
+    return {
+        "records": count,
+        **{f"{name}_count": value for name, value in counts.items()},
+        **{f"{name}_fraction": value / count for name, value in counts.items()},
+        "running_utilization_mean": statistics.fmean(running_utilization),
+        "running_utilization_max": max(running_utilization),
+        "token_utilization_mean": statistics.fmean(token_utilization),
+        "token_utilization_max": max(token_utilization),
+        "kv_usage_mean": statistics.fmean(kv_usage),
+        "kv_usage_max": max(kv_usage),
+        "preemptions": preemptions,
+    }
+
+
+def telemetry_coverage(
+    records: list[Mapping[str, Any]], *, start_ns: int, end_ns: int
+) -> tuple[dict[str, float], bool]:
+    if not records:
+        raise ValueError("telemetry coverage requires records")
+    submit_gaps = [
+        (int(right["submit_mono_ns"]) - int(left["submit_mono_ns"])) / 1e9
+        for left, right in zip(records, records[1:], strict=False)
+    ]
+    uncovered_gaps = [
+        max(
+            0,
+            int(right["submit_mono_ns"]) - int(left["complete_mono_ns"]),
+        )
+        / 1e9
+        for left, right in zip(records, records[1:], strict=False)
+    ]
+    coverage = {
+        "start_gap_s": (int(records[0]["submit_mono_ns"]) - start_ns) / 1e9,
+        "end_gap_s": (end_ns - int(records[-1]["submit_mono_ns"])) / 1e9,
+        "max_internal_submit_gap_s": max(submit_gaps, default=0.0),
+        "max_uncovered_gap_s": max(uncovered_gaps, default=0.0),
+    }
+    covered = (
+        0.0 <= coverage["start_gap_s"] <= 1.0
+        and 0.0 <= coverage["end_gap_s"] <= 1.0
+        and 0.0 <= coverage["max_uncovered_gap_s"] <= 1.0
+    )
+    return coverage, covered
+
+
+def mechanism_summary(records: list[Mapping[str, Any]]) -> dict[str, Any]:
+    executed = [record for record in records if bool(record["model_executed"])]
+    if not executed:
+        raise ValueError("mechanism summary requires executed steps")
+    prefill = [record for record in executed if int(record["prefill_tokens"]) > 0]
+    decode_only = [
+        record for record in executed if int(record["prefill_tokens"]) == 0
+    ]
+    if not prefill or not decode_only:
+        raise ValueError("mechanism summary requires prefill and decode-only steps")
+
+    def durations_ms(selected: list[Mapping[str, Any]]) -> list[float]:
+        values = [
+            (int(record["complete_mono_ns"]) - int(record["submit_mono_ns"]))
+            / 1e6
+            for record in selected
+        ]
+        if any(value < 0.0 for value in values):
+            raise ValueError("engine step duration must be non-negative")
+        return values
+
+    chunk_keys = ("first", "middle", "final", "unsplit", "tokens")
+    chunks = {
+        key: sum(int(record["chunked_prefill"][key]) for record in executed)
+        for key in chunk_keys
+    }
+    prefill_tokens = [int(record["prefill_tokens"]) for record in prefill]
+    prefill_requests = sum(int(record["prefill_requests"]) for record in prefill)
+    prefix_queries = sum(
+        int(record["prefix"]["local"]["queries"]) for record in executed
+    )
+    prefix_hits = sum(
+        int(record["prefix"]["local"]["hits"]) for record in executed
+    )
+    invariants = {
+        "nonnegative_counts": all(
+            value >= 0
+            for value in (
+                *chunks.values(),
+                prefill_requests,
+                prefix_queries,
+                prefix_hits,
+            )
+        ),
+        "chunk_tokens_match_prefill_tokens": chunks["tokens"]
+        == sum(prefill_tokens),
+        "prefix_hits_bounded": 0 <= prefix_hits <= prefix_queries,
+    }
+    return {
+        "executed_steps": len(executed),
+        "step_duration_ms": distribution(durations_ms(executed)),
+        "prefill_steps": len(prefill),
+        "prefill_step_duration_ms": distribution(durations_ms(prefill)),
+        "decode_only_steps": len(decode_only),
+        "decode_only_step_duration_ms": distribution(durations_ms(decode_only)),
+        "prefill": {
+            "requests": prefill_requests,
+            "requests_per_step": prefill_requests / len(prefill),
+            "tokens": sum(prefill_tokens),
+            "tokens_per_step": distribution(prefill_tokens),
+            "chunks": chunks,
+        },
+        "prefix": {
+            "queries": prefix_queries,
+            "hits": prefix_hits,
+            "hit_rate": prefix_hits / prefix_queries if prefix_queries else 0.0,
+        },
+        "sanity": {"invariants": invariants},
+    }
+
+
+def request_summary(path: Path, expected_count: int) -> dict[str, Any]:
+    rows = load_jsonl(path)
+    if len(rows) != expected_count:
+        raise ValueError(f"request row count mismatch: {path}")
+    ttft = [float(row["ttft_ms"]) for row in rows if row["ttft_ms"] is not None]
+    tpot = [float(row["tpot_ms"]) for row in rows if row["tpot_ms"] is not None]
+    if not ttft or not tpot:
+        raise ValueError(f"missing request latency values: {path}")
+    return {
+        "ttft_ms": {f"p{int(p * 100)}": quantile(ttft, p) for p in (0.5, 0.95, 0.99)},
+        "tpot_ms": {f"p{int(p * 100)}": quantile(tpot, p) for p in (0.5, 0.95, 0.99)},
+    }
+
+
+def load_stream(session_root: Path) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+    streams = sorted((session_root / "opprof").glob("*.jsonl"))
+    sidecars = sorted((session_root / "opprof").glob("*.jsonl.footer.json"))
+    if len(streams) != 1 or len(sidecars) != 1:
+        raise ValueError(f"expected one OpProf stream and sidecar: {session_root}")
+    decoded = load_jsonl(streams[0])
+    records = [row for row in decoded if "step_index" in row]
+    footers = [row for row in decoded if row.get("record_type") == "footer"]
+    sidecar = json.loads(sidecars[0].read_text(encoding="utf-8"))
+    indexes = [int(row["step_index"]) for row in records]
+    invariants = {
+        "one_footer_last": len(footers) == 1 and decoded[-1] is footers[0],
+        "sidecar_final": sidecar.get("final") is True,
+        "zero_drops": sidecar.get("dropped_records") == 0,
+        "written_matches_records": sidecar.get("written_records") == len(records),
+        "contiguous_step_indexes": indexes == list(range(len(indexes))),
+        "monotonic_timestamps": all(
+            int(right["submit_mono_ns"]) >= int(left["submit_mono_ns"])
+            for left, right in zip(records, records[1:], strict=False)
+        ),
+    }
+    return records, {
+        "stream": str(streams[0]),
+        "stream_sha256": sha256_file(streams[0]),
+        "records": len(records),
+        "invariants": invariants,
+    }
+
+
+def analyze_run(
+    *,
+    run_root: Path,
+    config: Mapping[str, Any],
+    repetition: int,
+    expected: Mapping[str, Any],
+    stream_records: list[Mapping[str, Any]],
+    duration_s: float,
+    phase_fractions: list[float],
+) -> dict[str, Any]:
+    result_root = run_root / "sessions" / str(config["id"]) / f"rep{repetition}"
+    result_path = result_root / "result.json"
+    result = json.loads(result_path.read_text(encoding="utf-8"))
+    selection = result["selection"]
+    invariants = {
+        "result_schema": result.get("schema") == "action-aware-pilot-result-v0",
+        "config_id": result.get("config_id") == config["id"],
+        "tp": int(result.get("tp", -1)) == 4,
+        "mns": int(result.get("mns", -1)) == int(config["mns"]),
+        "mbbt": int(result.get("mbbt", -1)) == int(config["mbbt"]),
+        "uncensored": not bool(result.get("early_stopped", True)),
+        "slo_early_stop_disabled": result.get("slo_early_stop_disabled") is True,
+        "selection_count": int(selection["count"]) == int(expected["selected_count"]),
+        "request_accounting": int(result["observed_count"])
+        == int(expected["selected_count"]),
+        "request_hash": selection["request_id_order_sha256"]
+        == expected["request_id_order_sha256"],
+        "arrival_hash": selection["arrival_order_sha256"]
+        == expected["arrival_order_sha256"],
+        "length_hash": selection["raw_length_order_sha256"]
+        == expected["input_length_order_sha256"],
+    }
+    start_ns = int(result["interval"]["start_mono_ns"])
+    arrival_end_ns = start_ns + round(duration_s * 1e9)
+    full_records = [
+        record
+        for record in stream_records
+        if start_ns <= int(record["submit_mono_ns"]) <= arrival_end_ns
+    ]
+    if not full_records:
+        raise ValueError(f"no telemetry records in measured window: {result_path}")
+    coverage, invariants["telemetry_coverage"] = telemetry_coverage(
+        full_records, start_ns=start_ns, end_ns=arrival_end_ns
+    )
+    binding = binding_summary(
+        full_records, mns=int(config["mns"]), mbbt=int(config["mbbt"])
+    )
+    mechanism = mechanism_summary(full_records)
+    invariants["mechanism_summary"] = all(
+        mechanism["sanity"]["invariants"].values()
+    )
+    phases = {}
+    for fraction in phase_fractions:
+        phase_end = start_ns + round(duration_s * fraction * 1e9)
+        phase_records = [
+            record
+            for record in full_records
+            if int(record["submit_mono_ns"]) <= phase_end
+        ]
+        phases[f"{fraction:.2f}"] = binding_summary(
+            phase_records, mns=int(config["mns"]), mbbt=int(config["mbbt"])
+        )
+    state = summarize_engine(
+        full_records,
+        start_ns=start_ns,
+        end_ns=arrival_end_ns,
+        request_count=int(result["observed_count"]),
+    )
+    latency = request_summary(
+        result_root / "requests.jsonl", int(result["observed_count"])
+    )
+    return {
+        "config_id": config["id"],
+        "mns": int(config["mns"]),
+        "mbbt": int(config["mbbt"]),
+        "repetition": repetition,
+        "result_path": str(result_path),
+        "result_sha256": sha256_file(result_path),
+        "selection": {
+            "count": int(selection["count"]),
+            "request_id_order_sha256": selection["request_id_order_sha256"],
+            "arrival_order_sha256": selection["arrival_order_sha256"],
+            "raw_length_order_sha256": selection["raw_length_order_sha256"],
+        },
+        "outcome": {
+            "pass_rate": float(result["pass_rate"]),
+            "feasible": bool(result["feasible"]),
+            "slo_pass_count": int(result["slo_pass_count"]),
+            "slo_goodput_req_s": int(result["slo_pass_count"]) / duration_s,
+            "elapsed_s": float(result["interval"]["elapsed_s"]),
+            **latency,
+        },
+        "binding": binding,
+        "mechanism": mechanism,
+        "phases": phases,
+        "state": state,
+        "coverage": coverage,
+        "invariants": invariants,
+    }
+
+
+def median(values: Iterable[float]) -> float:
+    return float(statistics.median(float(value) for value in values))
+
+
+def evaluate_decisions(
+    runs: list[Mapping[str, Any]], manifest: Mapping[str, Any]
+) -> dict[str, Any]:
+    by_key = {
+        (str(run["config_id"]), int(run["repetition"])): run for run in runs
+    }
+    repetitions = sorted(int(key) for key in manifest["repetitions"])
+    regime_results = {}
+    all_predictions = []
+    crossed_pass = True
+    binding_pass = True
+    material_ambiguity = False
+    for regime_name, regime in manifest["regimes"].items():
+        rows = []
+        source_runs = []
+        for repetition in repetitions:
+            source = by_key[(str(regime["source"]), repetition)]
+            mns_target = by_key[(str(regime["actions"]["mns"]), repetition)]
+            mbbt_target = by_key[(str(regime["actions"]["mbbt"]), repetition)]
+            source_runs.append(source)
+            source_goodput = float(source["outcome"]["slo_goodput_req_s"])
+            mns_goodput = float(mns_target["outcome"]["slo_goodput_req_s"])
+            mbbt_goodput = float(mbbt_target["outcome"]["slo_goodput_req_s"])
+            observed = (
+                "mns"
+                if mns_goodput > mbbt_goodput
+                else "mbbt"
+                if mbbt_goodput > mns_goodput
+                else "tie"
+            )
+            mns_score = float(source["binding"]["mns_exclusive_fraction"])
+            mbbt_score = float(source["binding"]["mbbt_exclusive_fraction"])
+            predicted = (
+                "mns"
+                if mns_score > mbbt_score
+                else "mbbt"
+                if mbbt_score > mns_score
+                else "tie"
+            )
+            phase_predictions = {}
+            for phase, summary in source["phases"].items():
+                left = float(summary["mns_exclusive_fraction"])
+                right = float(summary["mbbt_exclusive_fraction"])
+                phase_predictions[phase] = (
+                    "mns" if left > right else "mbbt" if right > left else "tie"
+                )
+            margin = (
+                abs(mns_goodput - mbbt_goodput) / source_goodput
+                if source_goodput > 0
+                else None
+            )
+            row = {
+                "repetition": repetition,
+                "source_goodput_req_s": source_goodput,
+                "mns_target_goodput_req_s": mns_goodput,
+                "mbbt_target_goodput_req_s": mbbt_goodput,
+                "observed_winner": observed,
+                "predicted_winner": predicted,
+                "prediction_correct": predicted == observed,
+                "relative_winner_margin_over_source": margin,
+                "mns_exclusive_fraction": mns_score,
+                "mbbt_exclusive_fraction": mbbt_score,
+                "phase_predictions": phase_predictions,
+                "phase_stable": all(value == predicted for value in phase_predictions.values()),
+            }
+            rows.append(row)
+            all_predictions.append(row)
+
+        expected_winner = "mns" if regime_name == "A" else "mbbt"
+        minimum_margin = float(manifest["gates"]["minimum_relative_winner_margin"])
+        regime_crossed = all(
+            row["observed_winner"] == expected_winner
+            and row["relative_winner_margin_over_source"] is not None
+            and row["relative_winner_margin_over_source"] >= minimum_margin
+            for row in rows
+        )
+        crossed_pass &= regime_crossed
+        winning_key = f"{expected_winner}_exclusive_fraction"
+        losing_key = (
+            "mbbt_exclusive_fraction" if expected_winner == "mns" else "mns_exclusive_fraction"
+        )
+        winning_median = median(row[winning_key] for row in rows)
+        losing_median = median(row[losing_key] for row in rows)
+        ratio_pass = winning_median >= float(
+            manifest["gates"]["minimum_exclusive_ratio"]
+        ) * losing_median
+        regime_binding = (
+            all(row["prediction_correct"] and row["phase_stable"] for row in rows)
+            and winning_median
+            >= float(manifest["gates"]["minimum_exclusive_fraction"])
+            and ratio_pass
+        )
+        binding_pass &= regime_binding
+        ambiguity_median = median(
+            float(run["binding"]["both_fraction"])
+            + float(run["binding"]["waiting_unresolved_fraction"])
+            for run in source_runs
+        )
+        score_gap_median = median(
+            abs(
+                float(run["binding"]["mns_exclusive_fraction"])
+                - float(run["binding"]["mbbt_exclusive_fraction"])
+            )
+            for run in source_runs
+        )
+        kv_max_median = median(
+            float(run["binding"]["kv_usage_max"]) for run in source_runs
+        )
+        any_preemption = any(
+            int(run["binding"]["preemptions"]) > 0 for run in source_runs
+        )
+        regime_material = (
+            ambiguity_median >= score_gap_median
+            or kv_max_median >= float(manifest["gates"]["material_kv_usage"])
+            or any_preemption
+        )
+        material_ambiguity |= regime_material
+        regime_results[regime_name] = {
+            "source": regime["source"],
+            "actions": regime["actions"],
+            "expected_winner": expected_winner,
+            "crossed_response_pass": regime_crossed,
+            "binding_pass": regime_binding,
+            "winning_exclusive_median": winning_median,
+            "losing_exclusive_median": losing_median,
+            "exclusive_ratio_pass": ratio_pass,
+            "ambiguity_median": ambiguity_median,
+            "exclusive_gap_median": score_gap_median,
+            "kv_usage_max_median": kv_max_median,
+            "any_preemption": any_preemption,
+            "material_ambiguity": regime_material,
+            "repetitions": rows,
+        }
+
+    if not crossed_pass:
+        decision = "STOP_WORKLOAD_NOT_CROSSED"
+    elif not binding_pass:
+        decision = "STOP_BINDING_NOT_PREDICTIVE"
+    elif material_ambiguity:
+        decision = "OPEN_EXACT_ATTRIBUTION_ABLATION"
+    else:
+        decision = "STOP_NO_NEW_INSTRUMENTATION_NEEDED"
+    correct = sum(int(row["prediction_correct"]) for row in all_predictions)
+    return {
+        "decision": decision,
+        "crossed_response_pass": crossed_pass,
+        "binding_pass": binding_pass,
+        "material_ambiguity": material_ambiguity,
+        "regimes": regime_results,
+        "baselines": {
+            "always_mns_correct": sum(
+                int(row["observed_winner"] == "mns") for row in all_predictions
+            ),
+            "always_mbbt_correct": sum(
+                int(row["observed_winner"] == "mbbt") for row in all_predictions
+            ),
+            "binding_correct": correct,
+            "decision_count": len(all_predictions),
+        },
+    }
+
+
+def analyze(run_root: Path, manifest_path: Path) -> dict[str, Any]:
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    if manifest.get("schema") not in {
+        "action-aware-constraint-pilot-manifest-v0",
+        "action-aware-constraint-pilot-manifest-v1",
+    }:
+        raise ValueError("unexpected manifest schema")
+    duration_s = float(manifest["engine"]["duration_s"])
+    phase_fractions = [float(value) for value in manifest["gates"]["phase_fractions"]]
+    runs = []
+    stream_audits = []
+    for config in manifest["configs"]:
+        session_root = run_root / "sessions" / str(config["id"])
+        stream_records, stream_audit = load_stream(session_root)
+        stream_audit["config_id"] = config["id"]
+        stream_audits.append(stream_audit)
+        for repetition in sorted(int(key) for key in manifest["repetitions"]):
+            runs.append(
+                analyze_run(
+                    run_root=run_root,
+                    config=config,
+                    repetition=repetition,
+                    expected=manifest["repetitions"][str(repetition)]["selection"],
+                    stream_records=stream_records,
+                    duration_s=duration_s,
+                    phase_fractions=phase_fractions,
+                )
+            )
+    invariants = {
+        "fifteen_runs": len(runs) == 15,
+        "five_streams": len(stream_audits) == 5,
+        "all_run_invariants": all(
+            all(bool(value) for value in run["invariants"].values()) for run in runs
+        ),
+        "all_stream_invariants": all(
+            all(bool(value) for value in stream["invariants"].values())
+            for stream in stream_audits
+        ),
+        "nonnegative_counters": all(
+            all(
+                float(run["binding"][key]) >= 0
+                for key in (
+                    "mns_exclusive_count",
+                    "mbbt_exclusive_count",
+                    "both_count",
+                    "waiting_unresolved_count",
+                    "preemptions",
+                )
+            )
+            for run in runs
+        ),
+        "ratios_bounded": all(
+            all(
+                0.0 <= float(run["binding"][key]) <= 1.0
+                for key in (
+                    "mns_exclusive_fraction",
+                    "mbbt_exclusive_fraction",
+                    "both_fraction",
+                    "waiting_unresolved_fraction",
+                    "kv_usage_mean",
+                    "kv_usage_max",
+                )
+            )
+            for run in runs
+        ),
+        "per_config_results_not_all_identical": len(
+            {float(run["outcome"]["pass_rate"]) for run in runs}
+        )
+        > 1,
+    }
+    red_flags = [name for name, passed in invariants.items() if not passed]
+    decisions = (
+        evaluate_decisions(runs, manifest)
+        if not red_flags
+        else {
+            "decision": "STOP_DATA_INVALID",
+            "crossed_response_pass": False,
+            "binding_pass": False,
+            "material_ambiguity": False,
+            "regimes": {},
+            "baselines": {},
+        }
+    )
+    payload = {
+        "schema": SCHEMA,
+        "decision": decisions["decision"],
+        "manifest": str(manifest_path),
+        "manifest_sha256": sha256_file(manifest_path),
+        "run_root": str(run_root),
+        "runs": runs,
+        "streams": stream_audits,
+        "decision_audit": decisions,
+        "sanity": {
+            "runs": len(runs),
+            "pass_rate": numeric(run["outcome"]["pass_rate"] for run in runs),
+            "slo_goodput_req_s": numeric(
+                run["outcome"]["slo_goodput_req_s"] for run in runs
+            ),
+            "telemetry_records_per_run": numeric(
+                run["binding"]["records"] for run in runs
+            ),
+            "mns_values": numeric(run["mns"] for run in runs),
+            "mbbt_values": numeric(run["mbbt"] for run in runs),
+            "invariants": invariants,
+            "red_flags": red_flags,
+        },
+    }
+    return payload
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    payload = analyze(args.run_root, args.manifest)
+    atomic_json(args.output, payload)
+    print(
+        json.dumps(
+            {
+                "decision": payload["decision"],
+                "sanity": payload["sanity"],
+                "decision_audit": payload["decision_audit"],
+            },
+            indent=2,
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/action-aware-v0/pilot-manifest-v1.json

@@ -0,0 +1,227 @@
+{
+  "budget": {
+    "expected_h20_hours": [
+      6.0,
+      7.2
+    ],
+    "expected_wall_minutes": [
+      90,
+      110
+    ],
+    "global_hard_cap_h20_hours": 8.0,
+    "hard_cap_h20_hours": 7.614013100465138,
+    "prior_attempt_artifact": "/home/admin/cpfs/wjh/action-aware-constraint-v0-20260714/operational-stop-v0.json",
+    "prior_attempt_h20_hours": 0.38598689953486126,
+    "safety_h20_hours": 0.25,
+    "session_estimate_h20_hours": 1.35
+  },
+  "burnin": {
+    "anchor": 0.18919793755240089,
+    "arrival_order_sha256": "6c0ac4cb9a30ef501eeeacc8e6cc631c345e976db5ccf530ea5a1ec706d62a24",
+    "input_length_order_sha256": "7939cc20e1a00d1031d27d71508789f38decbbbb6ea59a1df18b2ec342fd2ef8",
+    "offered_req_s": 8.5,
+    "offered_req_s_per_gpu": 2.125,
+    "request_id_order_sha256": "84f4809acbc8acd3b1d14dfa357134a1dc0b9287341624b33f598dafeef54dc7",
+    "selected_count": 510,
+    "study": "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/studies/burnin-tp4.json",
+    "study_sha256": "5d6c2098042909a863efd3112818fbee9bafe96f22898ac98b66846dbe1fef0f"
+  },
+  "configs": [
+    {
+      "id": "b_base",
+      "mbbt": 2048,
+      "mns": 64,
+      "repetition_order": [
+        1,
+        2,
+        3
+      ]
+    },
+    {
+      "id": "a_base",
+      "mbbt": 8192,
+      "mns": 16,
+      "repetition_order": [
+        2,
+        3,
+        1
+      ]
+    },
+    {
+      "id": "shared",
+      "mbbt": 8192,
+      "mns": 64,
+      "repetition_order": [
+        3,
+        1,
+        2
+      ]
+    },
+    {
+      "id": "b_mns",
+      "mbbt": 2048,
+      "mns": 128,
+      "repetition_order": [
+        1,
+        3,
+        2
+      ]
+    },
+    {
+      "id": "a_mbbt",
+      "mbbt": 16384,
+      "mns": 16,
+      "repetition_order": [
+        2,
+        1,
+        3
+      ]
+    }
+  ],
+  "engine": {
+    "burnin_max_elapsed_s": 90.0,
+    "client_timeout_s": 450.0,
+    "disable_slo_early_stop": true,
+    "duration_s": 300.0,
+    "tp": 4
+  },
+  "gates": {
+    "material_kv_usage": 0.9,
+    "minimum_exclusive_fraction": 0.1,
+    "minimum_exclusive_ratio": 5.0,
+    "minimum_relative_winner_margin": 0.1,
+    "phase_fractions": [
+      0.25,
+      0.5,
+      0.75,
+      1.0
+    ]
+  },
+  "regimes": {
+    "A": {
+      "actions": {
+        "mbbt": "a_mbbt",
+        "mns": "shared"
+      },
+      "source": "a_base"
+    },
+    "B": {
+      "actions": {
+        "mbbt": "shared",
+        "mns": "b_mns"
+      },
+      "source": "b_base"
+    }
+  },
+  "repetitions": {
+    "1": {
+      "merged_trace": {
+        "bytes": 337429767,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep1.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9420,
+        "sha256": "68983266aa0e66aa589562f7c08edbd966f9ba4405e20c105adb43777d2dfbf5",
+        "source_sha256": [
+          "b242d1d9086df3accab57b4c92445d5edd581e12f47e12cea227aa63964c6930",
+          "d23b549f7b69af3647308677bbf76f818a3c226a1c98f9a9f93f09ceee46be87"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low1.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high1.jsonl"
+        ]
+      },
+      "selection": {
+        "anchor": 0.48686986110831465,
+        "arrival_order_sha256": "c2ad99986ce558da5901a9c5ec0a00bd69f198c981d8779235f2773a5c87f1c0",
+        "input_length_order_sha256": "9442bfebdc3fab5062dc1f4d688dc28c02afe3fd806c56dd8159f0ac7e6d0b94",
+        "offered_req_s": 8.5,
+        "offered_req_s_per_gpu": 2.125,
+        "request_id_order_sha256": "0bb61dbc9c26875e991d0d4f984134910d37463e5063f86ee960cf4f8aafb771",
+        "selected_count": 2550,
+        "target_count": 2550,
+        "target_req_s_per_gpu": 2.125
+      },
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep1-tp4.json",
+      "study_sha256": "ecfff96e33d458eb1e3b9a6d24386f00cc6f1b19ff926e2ec6320b3f671a7ae3"
+    },
+    "2": {
+      "merged_trace": {
+        "bytes": 337509330,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep2.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9457,
+        "sha256": "f38e8938f6a481fc6725b71b21aa04ff7eaf79783cdfd6e41aa2f074156f00c2",
+        "source_sha256": [
+          "4cbb0baac082bd54af562ce2f39104c5c23b4671672da365a67b1e8c146adf9f",
+          "bb0bcd2564a88000f435f12feb21c7c902eafc9ea5fe916adfe9d1eae47f3f9a"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low2.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high2.jsonl"
+        ]
+      },
+      "selection": {
+        "anchor": 0.4825698948735577,
+        "arrival_order_sha256": "b9fc12cf3f86bc8a79bee65296e65aa2b8bf2aeca46b2887094c669adcbb9a00",
+        "input_length_order_sha256": "d8d4bd6fc8ba852a45605b673b6b3e4f33b58f459e69f2a032d226ee175b074e",
+        "offered_req_s": 8.5,
+        "offered_req_s_per_gpu": 2.125,
+        "request_id_order_sha256": "56a0616b6b54abafd37875c7cb25f8639afef2706ccc55dfbe568f45859ea382",
+        "selected_count": 2550,
+        "target_count": 2550,
+        "target_req_s_per_gpu": 2.125
+      },
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep2-tp4.json",
+      "study_sha256": "d92a576db031db24bb58f354ea725d7f7567cb76699d387117ac5a6c9317bbb9"
+    },
+    "3": {
+      "merged_trace": {
+        "bytes": 337450256,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep3.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9431,
+        "sha256": "3094084b0bb20cc02eecf465091a5c919b4e5b112f704cdc36a563d1efdcee46",
+        "source_sha256": [
+          "1f7ececb142f9a363d2d1ca25eb7b8488b2cc319a51b55faa384f2a3d51f2142",
+          "6f326234791e1cff4ff866bface0d097d0d6e3844eebb1c97653d8e9c35e9397"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low3.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high3.jsonl"
+        ]
+      },
+      "selection": {
+        "anchor": 0.48664343020532463,
+        "arrival_order_sha256": "efce7339e22d3618cb4d55e6b55bfddb2c563c18faba2a992d5829c13e3f55e9",
+        "input_length_order_sha256": "0792b05fff6729fbd92ab2bb4cb6d31bea7799e232ad42772936bc06efbafb54",
+        "offered_req_s": 8.5,
+        "offered_req_s_per_gpu": 2.125,
+        "request_id_order_sha256": "2a2fabe2c4cf176aeb7e0d32fb8e7dbb1f27429a2e7a0cd18d7d186f23096f19",
+        "selected_count": 2550,
+        "target_count": 2550,
+        "target_req_s_per_gpu": 2.125
+      },
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep3-tp4.json",
+      "study_sha256": "fb8ffe256dace32f4ca8a8d49b662d98c3b69b94ecc8fa826e43068b238884ab"
+    }
+  },
+  "sanity": {
+    "invariants": {
+      "all_repetition_orders_are_permutations": true,
+      "five_unique_configs": true,
+      "same_load_all_repetitions": true,
+      "shared_endpoint_reused_by_both_regimes": true,
+      "three_disjoint_repetitions": true
+    },
+    "red_flags": []
+  },
+  "schema": "action-aware-constraint-pilot-manifest-v1",
+  "source": {
+    "base_manifest": "/home/gahow/phd/aituner/runs/intervention-response-v2/pilot-manifest-v3.json",
+    "base_manifest_sha256": "273db1181dcc9d6b64439650d0642ebe553b12e6aa9adebfbe3758a7977e5611",
+    "source_trace": "/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260312_1000.jsonl",
+    "source_trace_sha256": "875ba869775deb78086477919f03b322da14e2673c7d070e26528c4190912757",
+    "window_id": "chat_w20260312_1000"
+  },
+  "status": "PASS"
+}

runs/action-aware-v0/pilot-manifest-v2.json

@@ -0,0 +1,227 @@
+{
+  "budget": {
+    "expected_h20_hours": [
+      6.0,
+      7.2
+    ],
+    "expected_wall_minutes": [
+      90,
+      110
+    ],
+    "global_hard_cap_h20_hours": 8.0,
+    "hard_cap_h20_hours": 7.295602157380846,
+    "prior_attempt_artifact": "/home/admin/cpfs/wjh/aituner/aituner-action-aware-20260714/runs/action-aware-v0/prior-attempts-v2.json",
+    "prior_attempt_h20_hours": 0.7043978426191542,
+    "safety_h20_hours": 0.25,
+    "session_estimate_h20_hours": 1.35
+  },
+  "burnin": {
+    "anchor": 0.18919793755240089,
+    "arrival_order_sha256": "6c0ac4cb9a30ef501eeeacc8e6cc631c345e976db5ccf530ea5a1ec706d62a24",
+    "input_length_order_sha256": "7939cc20e1a00d1031d27d71508789f38decbbbb6ea59a1df18b2ec342fd2ef8",
+    "offered_req_s": 8.5,
+    "offered_req_s_per_gpu": 2.125,
+    "request_id_order_sha256": "84f4809acbc8acd3b1d14dfa357134a1dc0b9287341624b33f598dafeef54dc7",
+    "selected_count": 510,
+    "study": "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/studies/burnin-tp4.json",
+    "study_sha256": "5d6c2098042909a863efd3112818fbee9bafe96f22898ac98b66846dbe1fef0f"
+  },
+  "configs": [
+    {
+      "id": "b_base",
+      "mbbt": 2048,
+      "mns": 64,
+      "repetition_order": [
+        1,
+        2,
+        3
+      ]
+    },
+    {
+      "id": "a_base",
+      "mbbt": 8192,
+      "mns": 16,
+      "repetition_order": [
+        2,
+        3,
+        1
+      ]
+    },
+    {
+      "id": "shared",
+      "mbbt": 8192,
+      "mns": 64,
+      "repetition_order": [
+        3,
+        1,
+        2
+      ]
+    },
+    {
+      "id": "b_mns",
+      "mbbt": 2048,
+      "mns": 128,
+      "repetition_order": [
+        1,
+        3,
+        2
+      ]
+    },
+    {
+      "id": "a_mbbt",
+      "mbbt": 16384,
+      "mns": 16,
+      "repetition_order": [
+        2,
+        1,
+        3
+      ]
+    }
+  ],
+  "engine": {
+    "burnin_max_elapsed_s": 90.0,
+    "client_timeout_s": 450.0,
+    "disable_slo_early_stop": true,
+    "duration_s": 300.0,
+    "tp": 4
+  },
+  "gates": {
+    "material_kv_usage": 0.9,
+    "minimum_exclusive_fraction": 0.1,
+    "minimum_exclusive_ratio": 5.0,
+    "minimum_relative_winner_margin": 0.1,
+    "phase_fractions": [
+      0.25,
+      0.5,
+      0.75,
+      1.0
+    ]
+  },
+  "regimes": {
+    "A": {
+      "actions": {
+        "mbbt": "a_mbbt",
+        "mns": "shared"
+      },
+      "source": "a_base"
+    },
+    "B": {
+      "actions": {
+        "mbbt": "shared",
+        "mns": "b_mns"
+      },
+      "source": "b_base"
+    }
+  },
+  "repetitions": {
+    "1": {
+      "merged_trace": {
+        "bytes": 337429767,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep1.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9420,
+        "sha256": "68983266aa0e66aa589562f7c08edbd966f9ba4405e20c105adb43777d2dfbf5",
+        "source_sha256": [
+          "b242d1d9086df3accab57b4c92445d5edd581e12f47e12cea227aa63964c6930",
+          "d23b549f7b69af3647308677bbf76f818a3c226a1c98f9a9f93f09ceee46be87"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low1.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high1.jsonl"
+        ]
+      },
+      "selection": {
+        "anchor": 0.48686986110831465,
+        "arrival_order_sha256": "c2ad99986ce558da5901a9c5ec0a00bd69f198c981d8779235f2773a5c87f1c0",
+        "input_length_order_sha256": "9442bfebdc3fab5062dc1f4d688dc28c02afe3fd806c56dd8159f0ac7e6d0b94",
+        "offered_req_s": 8.5,
+        "offered_req_s_per_gpu": 2.125,
+        "request_id_order_sha256": "0bb61dbc9c26875e991d0d4f984134910d37463e5063f86ee960cf4f8aafb771",
+        "selected_count": 2550,
+        "target_count": 2550,
+        "target_req_s_per_gpu": 2.125
+      },
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep1-tp4.json",
+      "study_sha256": "ecfff96e33d458eb1e3b9a6d24386f00cc6f1b19ff926e2ec6320b3f671a7ae3"
+    },
+    "2": {
+      "merged_trace": {
+        "bytes": 337509330,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep2.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9457,
+        "sha256": "f38e8938f6a481fc6725b71b21aa04ff7eaf79783cdfd6e41aa2f074156f00c2",
+        "source_sha256": [
+          "4cbb0baac082bd54af562ce2f39104c5c23b4671672da365a67b1e8c146adf9f",
+          "bb0bcd2564a88000f435f12feb21c7c902eafc9ea5fe916adfe9d1eae47f3f9a"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low2.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high2.jsonl"
+        ]
+      },
+      "selection": {
+        "anchor": 0.4825698948735577,
+        "arrival_order_sha256": "b9fc12cf3f86bc8a79bee65296e65aa2b8bf2aeca46b2887094c669adcbb9a00",
+        "input_length_order_sha256": "d8d4bd6fc8ba852a45605b673b6b3e4f33b58f459e69f2a032d226ee175b074e",
+        "offered_req_s": 8.5,
+        "offered_req_s_per_gpu": 2.125,
+        "request_id_order_sha256": "56a0616b6b54abafd37875c7cb25f8639afef2706ccc55dfbe568f45859ea382",
+        "selected_count": 2550,
+        "target_count": 2550,
+        "target_req_s_per_gpu": 2.125
+      },
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep2-tp4.json",
+      "study_sha256": "d92a576db031db24bb58f354ea725d7f7567cb76699d387117ac5a6c9317bbb9"
+    },
+    "3": {
+      "merged_trace": {
+        "bytes": 337450256,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep3.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9431,
+        "sha256": "3094084b0bb20cc02eecf465091a5c919b4e5b112f704cdc36a563d1efdcee46",
+        "source_sha256": [
+          "1f7ececb142f9a363d2d1ca25eb7b8488b2cc319a51b55faa384f2a3d51f2142",
+          "6f326234791e1cff4ff866bface0d097d0d6e3844eebb1c97653d8e9c35e9397"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low3.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high3.jsonl"
+        ]
+      },
+      "selection": {
+        "anchor": 0.48664343020532463,
+        "arrival_order_sha256": "efce7339e22d3618cb4d55e6b55bfddb2c563c18faba2a992d5829c13e3f55e9",
+        "input_length_order_sha256": "0792b05fff6729fbd92ab2bb4cb6d31bea7799e232ad42772936bc06efbafb54",
+        "offered_req_s": 8.5,
+        "offered_req_s_per_gpu": 2.125,
+        "request_id_order_sha256": "2a2fabe2c4cf176aeb7e0d32fb8e7dbb1f27429a2e7a0cd18d7d186f23096f19",
+        "selected_count": 2550,
+        "target_count": 2550,
+        "target_req_s_per_gpu": 2.125
+      },
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep3-tp4.json",
+      "study_sha256": "fb8ffe256dace32f4ca8a8d49b662d98c3b69b94ecc8fa826e43068b238884ab"
+    }
+  },
+  "sanity": {
+    "invariants": {
+      "all_repetition_orders_are_permutations": true,
+      "five_unique_configs": true,
+      "same_load_all_repetitions": true,
+      "shared_endpoint_reused_by_both_regimes": true,
+      "three_disjoint_repetitions": true
+    },
+    "red_flags": []
+  },
+  "schema": "action-aware-constraint-pilot-manifest-v1",
+  "source": {
+    "base_manifest": "/home/gahow/phd/aituner/runs/intervention-response-v2/pilot-manifest-v3.json",
+    "base_manifest_sha256": "273db1181dcc9d6b64439650d0642ebe553b12e6aa9adebfbe3758a7977e5611",
+    "source_trace": "/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260312_1000.jsonl",
+    "source_trace_sha256": "875ba869775deb78086477919f03b322da14e2673c7d070e26528c4190912757",
+    "window_id": "chat_w20260312_1000"
+  },
+  "status": "PASS"
+}

runs/action-aware-v0/pilot-manifest.json

@@ -0,0 +1,223 @@
+{
+  "budget": {
+    "expected_h20_hours": [
+      6.0,
+      7.2
+    ],
+    "expected_wall_minutes": [
+      90,
+      110
+    ],
+    "hard_cap_h20_hours": 8.0,
+    "safety_h20_hours": 0.25,
+    "session_estimate_h20_hours": 1.35
+  },
+  "burnin": {
+    "anchor": 0.18919793755240089,
+    "arrival_order_sha256": "6c0ac4cb9a30ef501eeeacc8e6cc631c345e976db5ccf530ea5a1ec706d62a24",
+    "input_length_order_sha256": "7939cc20e1a00d1031d27d71508789f38decbbbb6ea59a1df18b2ec342fd2ef8",
+    "offered_req_s": 8.5,
+    "offered_req_s_per_gpu": 2.125,
+    "request_id_order_sha256": "84f4809acbc8acd3b1d14dfa357134a1dc0b9287341624b33f598dafeef54dc7",
+    "selected_count": 510,
+    "study": "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/studies/burnin-tp4.json",
+    "study_sha256": "5d6c2098042909a863efd3112818fbee9bafe96f22898ac98b66846dbe1fef0f"
+  },
+  "configs": [
+    {
+      "id": "b_base",
+      "mbbt": 256,
+      "mns": 64,
+      "repetition_order": [
+        1,
+        2,
+        3
+      ]
+    },
+    {
+      "id": "a_base",
+      "mbbt": 8192,
+      "mns": 16,
+      "repetition_order": [
+        2,
+        3,
+        1
+      ]
+    },
+    {
+      "id": "shared",
+      "mbbt": 8192,
+      "mns": 64,
+      "repetition_order": [
+        3,
+        1,
+        2
+      ]
+    },
+    {
+      "id": "b_mns",
+      "mbbt": 256,
+      "mns": 128,
+      "repetition_order": [
+        1,
+        3,
+        2
+      ]
+    },
+    {
+      "id": "a_mbbt",
+      "mbbt": 16384,
+      "mns": 16,
+      "repetition_order": [
+        2,
+        1,
+        3
+      ]
+    }
+  ],
+  "engine": {
+    "client_timeout_s": 450.0,
+    "disable_slo_early_stop": true,
+    "duration_s": 300.0,
+    "tp": 4
+  },
+  "gates": {
+    "material_kv_usage": 0.9,
+    "minimum_exclusive_fraction": 0.1,
+    "minimum_exclusive_ratio": 5.0,
+    "minimum_relative_winner_margin": 0.1,
+    "phase_fractions": [
+      0.25,
+      0.5,
+      0.75,
+      1.0
+    ]
+  },
+  "regimes": {
+    "A": {
+      "actions": {
+        "mbbt": "a_mbbt",
+        "mns": "shared"
+      },
+      "source": "a_base"
+    },
+    "B": {
+      "actions": {
+        "mbbt": "shared",
+        "mns": "b_mns"
+      },
+      "source": "b_base"
+    }
+  },
+  "repetitions": {
+    "1": {
+      "merged_trace": {
+        "bytes": 337429767,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep1.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9420,
+        "sha256": "68983266aa0e66aa589562f7c08edbd966f9ba4405e20c105adb43777d2dfbf5",
+        "source_sha256": [
+          "b242d1d9086df3accab57b4c92445d5edd581e12f47e12cea227aa63964c6930",
+          "d23b549f7b69af3647308677bbf76f818a3c226a1c98f9a9f93f09ceee46be87"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low1.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high1.jsonl"
+        ]
+      },
+      "selection": {
+        "anchor": 0.48686986110831465,
+        "arrival_order_sha256": "c2ad99986ce558da5901a9c5ec0a00bd69f198c981d8779235f2773a5c87f1c0",
+        "input_length_order_sha256": "9442bfebdc3fab5062dc1f4d688dc28c02afe3fd806c56dd8159f0ac7e6d0b94",
+        "offered_req_s": 8.5,
+        "offered_req_s_per_gpu": 2.125,
+        "request_id_order_sha256": "0bb61dbc9c26875e991d0d4f984134910d37463e5063f86ee960cf4f8aafb771",
+        "selected_count": 2550,
+        "target_count": 2550,
+        "target_req_s_per_gpu": 2.125
+      },
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep1-tp4.json",
+      "study_sha256": "ecfff96e33d458eb1e3b9a6d24386f00cc6f1b19ff926e2ec6320b3f671a7ae3"
+    },
+    "2": {
+      "merged_trace": {
+        "bytes": 337509330,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep2.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9457,
+        "sha256": "f38e8938f6a481fc6725b71b21aa04ff7eaf79783cdfd6e41aa2f074156f00c2",
+        "source_sha256": [
+          "4cbb0baac082bd54af562ce2f39104c5c23b4671672da365a67b1e8c146adf9f",
+          "bb0bcd2564a88000f435f12feb21c7c902eafc9ea5fe916adfe9d1eae47f3f9a"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low2.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high2.jsonl"
+        ]
+      },
+      "selection": {
+        "anchor": 0.4825698948735577,
+        "arrival_order_sha256": "b9fc12cf3f86bc8a79bee65296e65aa2b8bf2aeca46b2887094c669adcbb9a00",
+        "input_length_order_sha256": "d8d4bd6fc8ba852a45605b673b6b3e4f33b58f459e69f2a032d226ee175b074e",
+        "offered_req_s": 8.5,
+        "offered_req_s_per_gpu": 2.125,
+        "request_id_order_sha256": "56a0616b6b54abafd37875c7cb25f8639afef2706ccc55dfbe568f45859ea382",
+        "selected_count": 2550,
+        "target_count": 2550,
+        "target_req_s_per_gpu": 2.125
+      },
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep2-tp4.json",
+      "study_sha256": "d92a576db031db24bb58f354ea725d7f7567cb76699d387117ac5a6c9317bbb9"
+    },
+    "3": {
+      "merged_trace": {
+        "bytes": 337450256,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep3.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9431,
+        "sha256": "3094084b0bb20cc02eecf465091a5c919b4e5b112f704cdc36a563d1efdcee46",
+        "source_sha256": [
+          "1f7ececb142f9a363d2d1ca25eb7b8488b2cc319a51b55faa384f2a3d51f2142",
+          "6f326234791e1cff4ff866bface0d097d0d6e3844eebb1c97653d8e9c35e9397"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low3.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high3.jsonl"
+        ]
+      },
+      "selection": {
+        "anchor": 0.48664343020532463,
+        "arrival_order_sha256": "efce7339e22d3618cb4d55e6b55bfddb2c563c18faba2a992d5829c13e3f55e9",
+        "input_length_order_sha256": "0792b05fff6729fbd92ab2bb4cb6d31bea7799e232ad42772936bc06efbafb54",
+        "offered_req_s": 8.5,
+        "offered_req_s_per_gpu": 2.125,
+        "request_id_order_sha256": "2a2fabe2c4cf176aeb7e0d32fb8e7dbb1f27429a2e7a0cd18d7d186f23096f19",
+        "selected_count": 2550,
+        "target_count": 2550,
+        "target_req_s_per_gpu": 2.125
+      },
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep3-tp4.json",
+      "study_sha256": "fb8ffe256dace32f4ca8a8d49b662d98c3b69b94ecc8fa826e43068b238884ab"
+    }
+  },
+  "sanity": {
+    "invariants": {
+      "all_repetition_orders_are_permutations": true,
+      "five_unique_configs": true,
+      "same_load_all_repetitions": true,
+      "shared_endpoint_reused_by_both_regimes": true,
+      "three_disjoint_repetitions": true
+    },
+    "red_flags": []
+  },
+  "schema": "action-aware-constraint-pilot-manifest-v0",
+  "source": {
+    "base_manifest": "/home/gahow/phd/aituner/runs/intervention-response-v2/pilot-manifest-v3.json",
+    "base_manifest_sha256": "273db1181dcc9d6b64439650d0642ebe553b12e6aa9adebfbe3758a7977e5611",
+    "source_trace": "/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260312_1000.jsonl",
+    "source_trace_sha256": "875ba869775deb78086477919f03b322da14e2673c7d070e26528c4190912757",
+    "window_id": "chat_w20260312_1000"
+  },
+  "status": "PASS"
+}

runs/action-aware-v0/pilot_controller.py

@@ -0,0 +1,638 @@
+#!/usr/bin/env python3
+"""Serialized controller for the crossed-constraint action-aware pilot."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shlex
+import signal
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any, Mapping
+
+
+HERE = Path(__file__).resolve().parent
+PHASE6 = HERE.parent / "opprof-phase6"
+sys.path.insert(0, str(PHASE6))
+
+import opprof_phase6_controller as base  # noqa: E402
+
+
+SCHEMA = "action-aware-constraint-pilot-state-v0"
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    base.atomic_json(path, payload)
+
+
+def wait_all_idle(timeout_s: float = 30.0) -> None:
+    deadline = time.monotonic() + timeout_s
+    last_error: Exception | None = None
+    while time.monotonic() < deadline:
+        try:
+            base.assert_all_idle()
+            return
+        except RuntimeError as error:
+            last_error = error
+            time.sleep(1.0)
+    raise last_error or RuntimeError("GPU idle timeout")
+
+
+def configure(args: argparse.Namespace, manifest: Mapping[str, Any]) -> None:
+    base.WORKDIR = args.run_root.parent
+    base.RUN_ROOT = args.run_root
+    base.STATE = args.run_root / "controller-state.json"
+    base.SOURCE = args.vllm_source
+    base.VENV = args.venv
+    base.AITUNER = args.aituner_root
+    base.MODEL = args.model
+    base.CLIENT = args.client
+    base.GPU_LIMIT = float(manifest["budget"]["hard_cap_h20_hours"])
+    base.MARKER = "action-aware-constraint-pilot-v0"
+
+
+def validate_inputs(args: argparse.Namespace, manifest: Mapping[str, Any]) -> None:
+    if manifest.get("schema") not in {
+        "action-aware-constraint-pilot-manifest-v0",
+        "action-aware-constraint-pilot-manifest-v1",
+    }:
+        raise RuntimeError("unexpected action-aware manifest schema")
+    if manifest.get("status") != "PASS":
+        raise RuntimeError("action-aware manifest did not pass preflight")
+    red_flags = manifest.get("sanity", {}).get("red_flags", [])
+    if red_flags:
+        raise RuntimeError(f"manifest red flags: {red_flags}")
+
+    required = {
+        "manifest": args.manifest,
+        "aituner_root": args.aituner_root,
+        "vllm_source": args.vllm_source,
+        "venv_python": args.venv / "bin/python",
+        "venv_vllm": args.venv / "bin/vllm",
+        "model": args.model,
+        "client": args.client,
+        "burnin_study": Path(manifest["burnin"]["study"]),
+    }
+    for repetition, item in manifest["repetitions"].items():
+        required[f"rep{repetition}_study"] = Path(item["study"])
+        required[f"rep{repetition}_trace"] = Path(item["merged_trace"]["path"])
+    missing = {name: str(path) for name, path in required.items() if not path.exists()}
+    if missing:
+        raise RuntimeError(f"action-aware input paths missing: {missing}")
+
+
+def config_map(manifest: Mapping[str, Any]) -> dict[str, dict[str, Any]]:
+    return {str(item["id"]): dict(item) for item in manifest["configs"]}
+
+
+def server_command(
+    config: Mapping[str, Any], *, gpus: tuple[int, ...], port: int
+) -> list[str]:
+    return [
+        "taskset",
+        "-c",
+        base.cpu_mask(gpus),
+        str(base.VENV / "bin/vllm"),
+        "serve",
+        str(base.MODEL),
+        "--host",
+        "127.0.0.1",
+        "--port",
+        str(port),
+        "--served-model-name",
+        "qwen3-30b-a3b-community",
+        "--max-num-batched-tokens",
+        str(config["mbbt"]),
+        "--max-num-seqs",
+        str(config["mns"]),
+        "--tensor-parallel-size",
+        "4",
+        "--shutdown-timeout",
+        "120",
+    ]
+
+
+def client_command(
+    entry: Mapping[str, Any],
+    config: Mapping[str, Any],
+    *,
+    study: str,
+    anchor: float,
+    output: Path,
+    warmup: bool,
+) -> list[str]:
+    command = [
+        "taskset",
+        "-c",
+        base.cpu_mask(entry["gpus"]),
+        str(base.VENV / "bin/python"),
+        str(base.CLIENT),
+        "warmup" if warmup else "run-anchor",
+        "--study",
+        study,
+        "--cell",
+        str(config["id"]),
+        "--anchor",
+        str(anchor),
+        "--tp",
+        "4",
+        "--mns",
+        str(config["mns"]),
+        "--mbbt",
+        str(config["mbbt"]),
+        "--base-url",
+        f"http://127.0.0.1:{entry['port']}",
+        "--result-dir",
+        str(output),
+        "--disable-slo-early-stop",
+    ]
+    return command
+
+
+def remaining_projection(
+    manifest: Mapping[str, Any], *, completed_sessions: int
+) -> float:
+    remaining = len(manifest["configs"]) - completed_sessions
+    return (
+        remaining * float(manifest["budget"]["session_estimate_h20_hours"])
+        + float(manifest["budget"]["safety_h20_hours"])
+    )
+
+
+def dry_run_plan(
+    args: argparse.Namespace, manifest: Mapping[str, Any]
+) -> dict[str, Any]:
+    sessions = []
+    for index, config in enumerate(manifest["configs"]):
+        entry = {"gpus": (0, 1, 2, 3), "port": 9050 + index}
+        session_root = args.run_root / "sessions" / str(config["id"])
+        first_repetition = str(config["repetition_order"][0])
+        first = manifest["repetitions"][first_repetition]
+        commands = {
+            "server": server_command(config, gpus=entry["gpus"], port=entry["port"]),
+            "warmup": client_command(
+                entry,
+                config,
+                study=first["study"],
+                anchor=float(first["selection"]["anchor"]),
+                output=session_root / "warmup",
+                warmup=True,
+            ),
+            "burnin": client_command(
+                entry,
+                config,
+                study=manifest["burnin"]["study"],
+                anchor=float(manifest["burnin"]["anchor"]),
+                output=session_root / "burnin",
+                warmup=False,
+            ),
+        }
+        for repetition in config["repetition_order"]:
+            item = manifest["repetitions"][str(repetition)]
+            commands[f"rep{repetition}"] = client_command(
+                entry,
+                config,
+                study=item["study"],
+                anchor=float(item["selection"]["anchor"]),
+                output=session_root / f"rep{repetition}",
+                warmup=False,
+            )
+        sessions.append(
+            {
+                "config": config["id"],
+                "mns": config["mns"],
+                "mbbt": config["mbbt"],
+                "port": entry["port"],
+                "repetition_order": config["repetition_order"],
+                "commands": {
+                    role: shlex.join(command) for role, command in commands.items()
+                },
+            }
+        )
+    return {
+        "schema": "action-aware-constraint-pilot-dry-run-v0",
+        "status": "PASS",
+        "manifest": str(args.manifest),
+        "run_root": str(args.run_root),
+        "projected_h20_hours": remaining_projection(
+            manifest, completed_sessions=0
+        ),
+        "hard_cap_h20_hours": manifest["budget"]["hard_cap_h20_hours"],
+        "sessions": sessions,
+    }
+
+
+def load_state(path: Path, hard_cap: float) -> dict[str, Any]:
+    if path.exists():
+        return json.loads(path.read_text(encoding="utf-8"))
+    return {
+        "schema": SCHEMA,
+        "status": "initialized",
+        "hard_cap_h20_hours": hard_cap,
+        "gpu_hours_total": 0.0,
+        "completed_sessions": 0,
+        "sessions": {},
+        "failures": [],
+        "started_at": time.time(),
+    }
+
+
+def append_echo(run_root: Path, line: str) -> None:
+    run_root.mkdir(parents=True, exist_ok=True)
+    with (run_root / "launch-echo.log").open("a", encoding="utf-8") as target:
+        target.write(line + "\n")
+    print(line, flush=True)
+
+
+def start_server(
+    *,
+    args: argparse.Namespace,
+    config: Mapping[str, Any],
+    index: int,
+) -> dict[str, Any]:
+    gpus = (0, 1, 2, 3)
+    session_root = args.run_root / "sessions" / str(config["id"])
+    session_root.mkdir(parents=True, exist_ok=True)
+    port = 9050 + index
+    command = server_command(config, gpus=gpus, port=port)
+    with (session_root / "commands.log").open("a", encoding="utf-8") as log:
+        log.write(f"SERVER {shlex.join(command)}\n")
+    server_log = (session_root / "server.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update(
+        {
+            "CUDA_VISIBLE_DEVICES": "0,1,2,3",
+            "VLLM_OPPROF_DIR": str(session_root / "opprof"),
+            "OPPROF_PHASE6_MARKER": base.MARKER,
+            "AITUNER_ROOT": str(base.AITUNER),
+            "HF_HUB_OFFLINE": "1",
+            "TRANSFORMERS_OFFLINE": "1",
+            "PYTHONUNBUFFERED": "1",
+        }
+    )
+    server = subprocess.Popen(
+        command,
+        cwd=base.SOURCE,
+        env=environment,
+        stdout=server_log,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    base.OWNED_PGIDS.add(server.pid)
+    return {
+        "cell": str(config["id"]),
+        "gpus": gpus,
+        "port": port,
+        "dir": session_root,
+        "server": server,
+        "server_handle": server_log,
+        "spawned_at": time.time(),
+        "results": [],
+    }
+
+
+def validate_result(
+    result: Mapping[str, Any],
+    *,
+    config: Mapping[str, Any],
+    selection: Mapping[str, Any],
+    role: str,
+    warmup: bool,
+) -> None:
+    if result.get("schema") != "action-aware-pilot-result-v0":
+        raise RuntimeError(f"unexpected result schema: {role}")
+    if result.get("config_id") != config["id"]:
+        raise RuntimeError(f"config id mismatch: {role}")
+    if int(result["tp"]) != 4:
+        raise RuntimeError(f"TP mismatch: {role}")
+    if int(result["mns"]) != int(config["mns"]):
+        raise RuntimeError(f"MNS mismatch: {role}")
+    if int(result["mbbt"]) != int(config["mbbt"]):
+        raise RuntimeError(f"MBBT mismatch: {role}")
+    if result.get("slo_early_stop_disabled") is not True:
+        raise RuntimeError(f"SLO early stop was not disabled: {role}")
+    if warmup:
+        if result["kind"] != "warmup" or int(result["selection"]["count"]) != 16:
+            raise RuntimeError(f"invalid warmup: {role}")
+        return
+    if bool(result["early_stopped"]):
+        raise RuntimeError(f"uncensored run early-stopped: {role}")
+    if int(result["selection"]["count"]) != int(selection["selected_count"]):
+        raise RuntimeError(f"selection count mismatch: {role}")
+    if int(result["observed_count"]) != int(selection["selected_count"]):
+        raise RuntimeError(f"request accounting mismatch: {role}")
+    for result_key, selection_key in (
+        ("request_id_order_sha256", "request_id_order_sha256"),
+        ("arrival_order_sha256", "arrival_order_sha256"),
+        ("raw_length_order_sha256", "input_length_order_sha256"),
+    ):
+        if result["selection"][result_key] != selection[selection_key]:
+            raise RuntimeError(f"selection hash mismatch {result_key}: {role}")
+
+
+def burnin_gate(
+    result: Mapping[str, Any],
+    *,
+    expected_count: int,
+    maximum_elapsed_s: float,
+) -> dict[str, Any]:
+    if result.get("kind") != "anchor":
+        raise RuntimeError("burnin gate received a non-anchor result")
+    if int(result["selection"]["count"]) != expected_count:
+        raise RuntimeError("burnin gate received the wrong request set")
+    elapsed_s = float(result["interval"]["elapsed_s"])
+    summary = {
+        "elapsed_s": elapsed_s,
+        "pass_rate": float(result["pass_rate"]),
+        "feasible": bool(result["feasible"]),
+    }
+    if elapsed_s > maximum_elapsed_s:
+        raise RuntimeError(
+            f"burnin throughput gate failed: {elapsed_s:.3f}s > "
+            f"{maximum_elapsed_s:.3f}s"
+        )
+    return summary
+
+
+def run_client(
+    *,
+    entry: dict[str, Any],
+    config: Mapping[str, Any],
+    role: str,
+    study: str,
+    selection: Mapping[str, Any],
+    output: Path,
+    state: Mapping[str, Any],
+    timeout_s: float,
+    warmup: bool = False,
+) -> dict[str, Any]:
+    command = client_command(
+        entry,
+        config,
+        study=study,
+        anchor=float(selection["anchor"]),
+        output=output,
+        warmup=warmup,
+    )
+    with (entry["dir"] / "commands.log").open("a", encoding="utf-8") as log:
+        log.write(f"CLIENT role={role} {shlex.join(command)}\n")
+    handle = (output.parent / f"{output.name}.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update({"AITUNER_ROOT": str(base.AITUNER), "PYTHONUNBUFFERED": "1"})
+    process = subprocess.Popen(
+        command,
+        cwd=base.WORKDIR,
+        env=environment,
+        stdout=handle,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    deadline = time.monotonic() + timeout_s
+    try:
+        while process.poll() is None:
+            if time.monotonic() > deadline:
+                raise TimeoutError(f"client timeout: {config['id']} {role}")
+            if entry["server"].poll() is not None:
+                raise RuntimeError(f"server exited during {config['id']} {role}")
+            base.assert_no_other_compute()
+            if state["gpu_hours_total"] + base.live_gpu_hours([entry]) >= base.GPU_LIMIT:
+                raise RuntimeError("action-aware pilot H20-hour hard cap reached")
+            time.sleep(1.0)
+    except Exception:
+        try:
+            os.killpg(process.pid, signal.SIGTERM)
+        except ProcessLookupError:
+            pass
+        try:
+            process.wait(timeout=10.0)
+        except subprocess.TimeoutExpired:
+            try:
+                os.killpg(process.pid, signal.SIGKILL)
+            except ProcessLookupError:
+                pass
+            process.wait(timeout=10.0)
+        raise
+    finally:
+        handle.close()
+    if process.returncode:
+        raise RuntimeError(
+            f"client failed: config={config['id']} role={role} rc={process.returncode}"
+        )
+    result = json.loads((output / "result.json").read_text(encoding="utf-8"))
+    validate_result(
+        result,
+        config=config,
+        selection=selection,
+        role=role,
+        warmup=warmup,
+    )
+    entry["results"].append(
+        {"anchor": float(selection["anchor"]), "dir": str(output), "kind": result["kind"]}
+    )
+    return result
+
+
+def execute_session(
+    *,
+    args: argparse.Namespace,
+    manifest: Mapping[str, Any],
+    config: Mapping[str, Any],
+    index: int,
+    state: dict[str, Any],
+    state_path: Path,
+) -> None:
+    name = str(config["id"])
+    if state["sessions"].get(name, {}).get("status") == "complete":
+        return
+    projection = remaining_projection(
+        manifest, completed_sessions=int(state["completed_sessions"])
+    )
+    if float(state["gpu_hours_total"]) + projection > base.GPU_LIMIT:
+        raise RuntimeError(f"projected cost exceeds cap before {name}")
+    load_values = {
+        float(item["selection"]["offered_req_s_per_gpu"])
+        for item in manifest["repetitions"].values()
+    }
+    load_text = (
+        f"{next(iter(load_values)):.6g}"
+        if len(load_values) == 1
+        else ",".join(f"{value:.6g}" for value in sorted(load_values))
+    )
+    echo = (
+        f"ACTION_AWARE_SESSION_ECHO host=dash0 config={name} tp=4 "
+        f"mns={config['mns']} mbbt={config['mbbt']} gpus=0-3 "
+        f"workload={manifest['source']['window_id']} load_per_gpu={load_text} "
+        f"duration_s={manifest['engine']['duration_s']} "
+        f"repetitions={','.join(map(str, config['repetition_order']))} "
+        f"source={args.manifest} output={args.run_root / 'sessions' / name} "
+        f"spent_h20h={state['gpu_hours_total']:.6f} "
+        f"remaining_projection_h20h={projection:.3f} cap_h20h={base.GPU_LIMIT:.1f}"
+    )
+    append_echo(args.run_root, echo)
+    wait_all_idle()
+    session_state = {
+        "status": "starting",
+        "mns": int(config["mns"]),
+        "mbbt": int(config["mbbt"]),
+        "repetition_order": list(config["repetition_order"]),
+        "started_at": time.time(),
+        "runs": [],
+    }
+    state["status"] = "running"
+    state["sessions"][name] = session_state
+    atomic_json(state_path, state)
+    entry = start_server(args=args, config=config, index=index)
+    failure: Exception | None = None
+    try:
+        base.wait_ready(entry)
+        first = manifest["repetitions"][str(config["repetition_order"][0])]
+        session_state["status"] = "warmup"
+        atomic_json(state_path, state)
+        run_client(
+            entry=entry,
+            config=config,
+            role="warmup",
+            study=first["study"],
+            selection=first["selection"],
+            output=entry["dir"] / "warmup",
+            state=state,
+            timeout_s=180.0,
+            warmup=True,
+        )
+        session_state["status"] = "burnin"
+        atomic_json(state_path, state)
+        burnin = manifest["burnin"]
+        burnin_result = run_client(
+            entry=entry,
+            config=config,
+            role="burnin",
+            study=burnin["study"],
+            selection=burnin,
+            output=entry["dir"] / "burnin",
+            state=state,
+            timeout_s=float(manifest["engine"]["client_timeout_s"]),
+        )
+        session_state["burnin"] = burnin_gate(
+            burnin_result,
+            expected_count=int(burnin["selected_count"]),
+            maximum_elapsed_s=float(manifest["engine"]["burnin_max_elapsed_s"]),
+        )
+        atomic_json(state_path, state)
+        session_state["status"] = "measured"
+        atomic_json(state_path, state)
+        for repetition in config["repetition_order"]:
+            item = manifest["repetitions"][str(repetition)]
+            role = f"rep{repetition}"
+            result = run_client(
+                entry=entry,
+                config=config,
+                role=role,
+                study=item["study"],
+                selection=item["selection"],
+                output=entry["dir"] / role,
+                state=state,
+                timeout_s=float(manifest["engine"]["client_timeout_s"]),
+            )
+            session_state["runs"].append(
+                {
+                    "repetition": int(repetition),
+                    "pass_rate": result["pass_rate"],
+                    "feasible": result["feasible"],
+                    "slo_pass_count": result["slo_pass_count"],
+                    "elapsed_s": result["interval"]["elapsed_s"],
+                }
+            )
+            atomic_json(state_path, state)
+        session_state["status"] = "stopping"
+        atomic_json(state_path, state)
+    except Exception as error:  # noqa: BLE001
+        failure = error
+    finally:
+        try:
+            base.stop_entry(entry)
+        except Exception as error:  # noqa: BLE001
+            failure = failure or error
+        time.sleep(2.0)
+        try:
+            wait_all_idle()
+        except Exception as error:  # noqa: BLE001
+            failure = failure or error
+
+    session_hours = base.live_gpu_hours([entry])
+    state["gpu_hours_total"] += session_hours
+    session_state["gpu_hours"] = session_hours
+    if failure is not None:
+        session_state["status"] = "failed"
+        session_state["failure"] = repr(failure)
+        state["status"] = "failed"
+        state["failures"].append({"session": name, "failure": repr(failure)})
+        atomic_json(state_path, state)
+        raise failure
+    validation = base.validate_cell(entry)
+    session_state["validation"] = validation
+    session_state["status"] = "complete"
+    session_state["completed_at"] = time.time()
+    state["completed_sessions"] += 1
+    atomic_json(state_path, state)
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("--manifest", type=Path, required=True)
+    result.add_argument("--run-root", type=Path, required=True)
+    result.add_argument("--aituner-root", type=Path, required=True)
+    result.add_argument("--vllm-source", type=Path, required=True)
+    result.add_argument("--venv", type=Path, required=True)
+    result.add_argument("--model", type=Path, required=True)
+    result.add_argument("--client", type=Path, required=True)
+    result.add_argument("--dry-run", action="store_true")
+    return result
+
+
+def main() -> None:
+    args = parser().parse_args()
+    manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
+    validate_inputs(args, manifest)
+    configure(args, manifest)
+    if args.dry_run:
+        print(json.dumps(dry_run_plan(args, manifest), indent=2, sort_keys=True))
+        return
+    args.run_root.mkdir(parents=True, exist_ok=True)
+    copied_manifest = args.run_root / "pilot-manifest.json"
+    if not copied_manifest.exists():
+        atomic_json(copied_manifest, manifest)
+    state_path = args.run_root / "controller-state.json"
+    state = load_state(state_path, base.GPU_LIMIT)
+    state["status"] = "running"
+    atomic_json(state_path, state)
+    for index, config in enumerate(manifest["configs"]):
+        execute_session(
+            args=args,
+            manifest=manifest,
+            config=config,
+            index=index,
+            state=state,
+            state_path=state_path,
+        )
+    state["status"] = "complete"
+    state["completed_at"] = time.time()
+    atomic_json(state_path, state)
+    wait_all_idle()
+    print(
+        json.dumps(
+            {
+                "status": state["status"],
+                "completed_sessions": state["completed_sessions"],
+                "gpu_hours_total": state["gpu_hours_total"],
+            },
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/action-aware-v0/prepare_pilot.py

@@ -0,0 +1,195 @@
+#!/usr/bin/env python3
+"""Freeze the crossed-constraint action-aware development pilot."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+from pathlib import Path
+from typing import Any
+
+
+SCHEMA_V0 = "action-aware-constraint-pilot-manifest-v0"
+SCHEMA_V1 = "action-aware-constraint-pilot-manifest-v1"
+
+
+def configs(token_source_mbbt: int) -> tuple[dict[str, Any], ...]:
+    return (
+        {
+            "id": "b_base",
+            "mns": 64,
+            "mbbt": token_source_mbbt,
+            "repetition_order": [1, 2, 3],
+        },
+        {"id": "a_base", "mns": 16, "mbbt": 8192, "repetition_order": [2, 3, 1]},
+        {"id": "shared", "mns": 64, "mbbt": 8192, "repetition_order": [3, 1, 2]},
+        {
+            "id": "b_mns",
+            "mns": 128,
+            "mbbt": token_source_mbbt,
+            "repetition_order": [1, 3, 2],
+        },
+        {"id": "a_mbbt", "mns": 16, "mbbt": 16384, "repetition_order": [2, 1, 3]},
+    )
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix(path.suffix + ".tmp")
+    temporary.write_text(
+        json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+    )
+    os.replace(temporary, path)
+
+
+def build(
+    base_path: Path,
+    *,
+    token_source_mbbt: int = 256,
+    prior_attempt_h20_hours: float = 0.0,
+    prior_attempt_artifact: str | None = None,
+) -> dict[str, Any]:
+    if token_source_mbbt <= 0:
+        raise ValueError("token source MBBT must be positive")
+    if prior_attempt_h20_hours < 0.0 or prior_attempt_h20_hours >= 8.0:
+        raise ValueError("prior attempt cost must be in [0, 8)")
+    base = json.loads(base_path.read_text(encoding="utf-8"))
+    if base.get("schema") != "intervention-response-phase-aware-pilot-manifest-v3":
+        raise ValueError("unexpected base manifest schema")
+    if base.get("status") != "PASS":
+        raise ValueError("base manifest did not pass its preflight")
+    if sorted(int(key) for key in base["repetitions"]) != [1, 2, 3]:
+        raise ValueError("base manifest must contain exactly three repetitions")
+
+    repetitions = {}
+    selection_hashes = []
+    for repetition in (1, 2, 3):
+        source = base["repetitions"][str(repetition)]
+        selection = dict(source["selections"]["mid"])
+        selection_hashes.append(selection["request_id_order_sha256"])
+        repetitions[str(repetition)] = {
+            "study": source["study"],
+            "study_sha256": source["study_sha256"],
+            "selection": selection,
+            "merged_trace": source["merged_trace"],
+        }
+
+    frozen_configs = configs(token_source_mbbt)
+    config_ids = [str(config["id"]) for config in frozen_configs]
+    schema = (
+        SCHEMA_V0
+        if token_source_mbbt == 256 and prior_attempt_h20_hours == 0.0
+        else SCHEMA_V1
+    )
+    payload = {
+        "schema": schema,
+        "status": "PASS",
+        "source": {
+            "base_manifest": str(base_path.resolve()),
+            "base_manifest_sha256": sha256_file(base_path),
+            "window_id": base["source"]["window_id"],
+            "source_trace": base["source"]["source_trace"],
+            "source_trace_sha256": base["source"]["source_trace_sha256"],
+        },
+        "engine": {
+            "tp": 4,
+            "duration_s": 300.0,
+            "disable_slo_early_stop": True,
+            "client_timeout_s": 450.0,
+            "burnin_max_elapsed_s": 90.0,
+        },
+        "burnin": base["burnin"],
+        "repetitions": repetitions,
+        "configs": [dict(config) for config in frozen_configs],
+        "regimes": {
+            "A": {
+                "source": "a_base",
+                "actions": {"mns": "shared", "mbbt": "a_mbbt"},
+            },
+            "B": {
+                "source": "b_base",
+                "actions": {"mns": "b_mns", "mbbt": "shared"},
+            },
+        },
+        "budget": {
+            "global_hard_cap_h20_hours": 8.0,
+            "hard_cap_h20_hours": 8.0 - prior_attempt_h20_hours,
+            "prior_attempt_h20_hours": prior_attempt_h20_hours,
+            "prior_attempt_artifact": prior_attempt_artifact,
+            "session_estimate_h20_hours": 1.35,
+            "safety_h20_hours": 0.25,
+            "expected_h20_hours": [6.0, 7.2],
+            "expected_wall_minutes": [90, 110],
+        },
+        "gates": {
+            "minimum_relative_winner_margin": 0.10,
+            "minimum_exclusive_fraction": 0.10,
+            "minimum_exclusive_ratio": 5.0,
+            "phase_fractions": [0.25, 0.50, 0.75, 1.0],
+            "material_kv_usage": 0.90,
+        },
+        "sanity": {
+            "invariants": {
+                "five_unique_configs": len(config_ids) == len(set(config_ids)) == 5,
+                "three_disjoint_repetitions": len(set(selection_hashes)) == 3,
+                "same_load_all_repetitions": len(
+                    {
+                        float(item["selection"]["offered_req_s_per_gpu"])
+                        for item in repetitions.values()
+                    }
+                )
+                == 1,
+                "all_repetition_orders_are_permutations": all(
+                    sorted(config["repetition_order"]) == [1, 2, 3]
+                    for config in frozen_configs
+                ),
+            }
+        },
+    }
+    payload["sanity"]["invariants"]["shared_endpoint_reused_by_both_regimes"] = (
+        payload["regimes"]["A"]["actions"]["mns"]
+        == payload["regimes"]["B"]["actions"]["mbbt"]
+        == "shared"
+    )
+    payload["sanity"]["red_flags"] = [
+        name
+        for name, passed in payload["sanity"]["invariants"].items()
+        if not passed
+    ]
+    if payload["sanity"]["red_flags"]:
+        payload["status"] = "FAIL"
+    return payload
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base-manifest", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--token-source-mbbt", type=int, default=256)
+    parser.add_argument("--prior-attempt-h20-hours", type=float, default=0.0)
+    parser.add_argument("--prior-attempt-artifact")
+    args = parser.parse_args()
+    payload = build(
+        args.base_manifest,
+        token_source_mbbt=args.token_source_mbbt,
+        prior_attempt_h20_hours=args.prior_attempt_h20_hours,
+        prior_attempt_artifact=args.prior_attempt_artifact,
+    )
+    atomic_json(args.output, payload)
+    print(json.dumps(payload["sanity"], sort_keys=True))
+    if payload["status"] != "PASS":
+        raise SystemExit("manifest preflight failed")
+
+
+if __name__ == "__main__":
+    main()

runs/action-aware-v0/prior-attempts-v2.json

@@ -0,0 +1,24 @@
+{
+  "global_hard_cap_h20_hours": 8.0,
+  "invariants": {
+    "all_gpus_idle_after_each_stop": true,
+    "no_completed_measured_runs": true,
+    "no_prior_runtime_data_reused": true
+  },
+  "prior_attempt_h20_hours": 0.7043978426191542,
+  "schema": "action-aware-prior-attempts-v2",
+  "stops": [
+    {
+      "artifact": "/home/admin/cpfs/wjh/action-aware-constraint-v0-20260714/operational-stop-v0.json",
+      "h20_hours": 0.38598689953486126,
+      "reason": "MBBT256 burn-in remained throughput-backlogged",
+      "stage": "burnin"
+    },
+    {
+      "artifact": "/home/admin/cpfs/wjh/action-aware-constraint-v1-20260714/operational-stop-v1.json",
+      "h20_hours": 0.31841094308429296,
+      "reason": "controller passed the warmup result to the burn-in gate",
+      "stage": "first measured run in flight; zero measured results completed"
+    }
+  ]
+}

runs/action-aware-v0/test_pilot.py

@@ -0,0 +1,292 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import copy
+import importlib.util
+from pathlib import Path
+from types import SimpleNamespace
+
+
+HERE = Path(__file__).resolve().parent
+ROOT = HERE.parents[1]
+
+
+def load(name: str, filename: str):
+    spec = importlib.util.spec_from_file_location(name, HERE / filename)
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+def record(*, waiting: int, running: int, tokens: int) -> dict:
+    return {
+        "queues": {"waiting": waiting, "deferred": 0, "running": running},
+        "prefill_tokens": tokens,
+        "decode_tokens": 0,
+        "kv": {"usage": 0.5},
+        "preemptions": 0,
+    }
+
+
+def fake_run(
+    config: str,
+    repetition: int,
+    *,
+    goodput: float,
+    mns_score: float = 0.0,
+    mbbt_score: float = 0.0,
+    ambiguous: float = 0.0,
+) -> dict:
+    binding = {
+        "mns_exclusive_fraction": mns_score,
+        "mbbt_exclusive_fraction": mbbt_score,
+        "both_fraction": ambiguous,
+        "waiting_unresolved_fraction": 0.0,
+        "kv_usage_max": 0.5,
+        "preemptions": 0,
+    }
+    phases = {
+        phase: {
+            "mns_exclusive_fraction": mns_score,
+            "mbbt_exclusive_fraction": mbbt_score,
+        }
+        for phase in ("0.25", "0.50", "0.75", "1.00")
+    }
+    return {
+        "config_id": config,
+        "repetition": repetition,
+        "outcome": {"slo_goodput_req_s": goodput},
+        "binding": binding,
+        "phases": phases,
+    }
+
+
+def main() -> None:
+    analysis = load("action_aware_analysis", "analyze_pilot.py")
+    summary = analysis.binding_summary(
+        [
+            record(waiting=1, running=16, tokens=8),
+            record(waiting=1, running=8, tokens=32),
+            record(waiting=1, running=16, tokens=32),
+            record(waiting=1, running=8, tokens=8),
+            record(waiting=0, running=8, tokens=8),
+        ],
+        mns=16,
+        mbbt=32,
+    )
+    assert summary["mns_exclusive_count"] == 1
+    assert summary["mbbt_exclusive_count"] == 1
+    assert summary["both_count"] == 1
+    assert summary["waiting_unresolved_count"] == 1
+    assert summary["waiting_count"] == 4
+
+    # A per-step stream may have a submit gap above one second when the
+    # preceding model execution itself spans that interval.  Such a gap is
+    # covered telemetry, not a dropped-record interval.
+    asynchronous = [
+        {"submit_mono_ns": 0, "complete_mono_ns": 1_200_000_000},
+        {"submit_mono_ns": 1_100_000_000, "complete_mono_ns": 1_300_000_000},
+    ]
+    coverage, covered = analysis.telemetry_coverage(
+        asynchronous, start_ns=0, end_ns=1_100_000_000
+    )
+    assert coverage["max_internal_submit_gap_s"] == 1.1
+    assert coverage["max_uncovered_gap_s"] == 0.0
+    assert covered
+    missing = copy.deepcopy(asynchronous)
+    missing[0]["complete_mono_ns"] = 0
+    assert not analysis.telemetry_coverage(
+        missing, start_ns=0, end_ns=1_100_000_000
+    )[1]
+
+    mechanism = analysis.mechanism_summary(
+        [
+            {
+                "model_executed": True,
+                "submit_mono_ns": 0,
+                "complete_mono_ns": 2_000_000,
+                "prefill_tokens": 8,
+                "prefill_requests": 2,
+                "chunked_prefill": {
+                    "first": 1,
+                    "middle": 0,
+                    "final": 0,
+                    "unsplit": 1,
+                    "tokens": 8,
+                },
+                "prefix": {"local": {"queries": 10, "hits": 2}},
+            },
+            {
+                "model_executed": True,
+                "submit_mono_ns": 2_000_000,
+                "complete_mono_ns": 3_000_000,
+                "prefill_tokens": 0,
+                "prefill_requests": 0,
+                "chunked_prefill": {
+                    "first": 0,
+                    "middle": 0,
+                    "final": 0,
+                    "unsplit": 0,
+                    "tokens": 0,
+                },
+                "prefix": {"local": {"queries": 0, "hits": 0}},
+            },
+        ]
+    )
+    assert mechanism["prefill"]["requests_per_step"] == 2.0
+    assert mechanism["prefill"]["chunks"]["first"] == 1
+    assert mechanism["prefix"]["hit_rate"] == 0.2
+    assert all(mechanism["sanity"]["invariants"].values())
+
+    manifest = {
+        "repetitions": {str(index): {} for index in (1, 2, 3)},
+        "regimes": {
+            "A": {
+                "source": "a_base",
+                "actions": {"mns": "shared", "mbbt": "a_mbbt"},
+            },
+            "B": {
+                "source": "b_base",
+                "actions": {"mns": "b_mns", "mbbt": "shared"},
+            },
+        },
+        "gates": {
+            "minimum_relative_winner_margin": 0.10,
+            "minimum_exclusive_fraction": 0.10,
+            "minimum_exclusive_ratio": 5.0,
+            "material_kv_usage": 0.90,
+        },
+    }
+    runs = []
+    for repetition in (1, 2, 3):
+        runs.extend(
+            [
+                fake_run(
+                    "a_base",
+                    repetition,
+                    goodput=1.0,
+                    mns_score=0.8,
+                    mbbt_score=0.01,
+                ),
+                fake_run(
+                    "b_base",
+                    repetition,
+                    goodput=1.0,
+                    mns_score=0.01,
+                    mbbt_score=0.7,
+                ),
+                fake_run("shared", repetition, goodput=3.0),
+                fake_run("a_mbbt", repetition, goodput=1.5),
+                fake_run("b_mns", repetition, goodput=1.2),
+            ]
+        )
+    result = analysis.evaluate_decisions(runs, manifest)
+    assert result["decision"] == "STOP_NO_NEW_INSTRUMENTATION_NEEDED"
+    assert result["baselines"] == {
+        "always_mns_correct": 3,
+        "always_mbbt_correct": 3,
+        "binding_correct": 6,
+        "decision_count": 6,
+    }
+
+    ambiguous = copy.deepcopy(runs)
+    for run in ambiguous:
+        if run["config_id"] == "b_base":
+            run["binding"]["both_fraction"] = 0.8
+    assert (
+        analysis.evaluate_decisions(ambiguous, manifest)["decision"]
+        == "OPEN_EXACT_ATTRIBUTION_ABLATION"
+    )
+
+    wrong = copy.deepcopy(runs)
+    for run in wrong:
+        if run["config_id"] == "b_base":
+            run["binding"]["mns_exclusive_fraction"] = 0.8
+            run["binding"]["mbbt_exclusive_fraction"] = 0.01
+            for phase in run["phases"].values():
+                phase["mns_exclusive_fraction"] = 0.8
+                phase["mbbt_exclusive_fraction"] = 0.01
+    assert (
+        analysis.evaluate_decisions(wrong, manifest)["decision"]
+        == "STOP_BINDING_NOT_PREDICTIVE"
+    )
+
+    prepare = load("action_aware_prepare", "prepare_pilot.py")
+    frozen = prepare.build(
+        ROOT / "runs/intervention-response-v2/pilot-manifest-v3.json"
+    )
+    assert frozen["status"] == "PASS"
+    assert frozen["sanity"]["red_flags"] == []
+    assert [config["id"] for config in frozen["configs"]] == [
+        "b_base",
+        "a_base",
+        "shared",
+        "b_mns",
+        "a_mbbt",
+    ]
+
+    controller = load("action_aware_controller", "pilot_controller.py")
+    args = SimpleNamespace(
+        manifest=Path("/tmp/manifest.json"),
+        run_root=Path("/tmp/action-aware"),
+        aituner_root=Path("/tmp/aituner"),
+        vllm_source=Path("/tmp/vllm"),
+        venv=Path("/tmp/venv"),
+        model=Path("/tmp/model"),
+        client=Path("/tmp/client.py"),
+    )
+    controller.configure(args, frozen)
+    plan = controller.dry_run_plan(args, frozen)
+    assert plan["status"] == "PASS"
+    assert len(plan["sessions"]) == 5
+    assert plan["projected_h20_hours"] == 7.0
+    assert "--max-num-batched-tokens 256" in plan["sessions"][0]["commands"]["server"]
+    revised = prepare.build(
+        ROOT / "runs/intervention-response-v2/pilot-manifest-v3.json",
+        token_source_mbbt=2048,
+        prior_attempt_h20_hours=0.38598689953486126,
+        prior_attempt_artifact="/tmp/operational-stop-v0.json",
+    )
+    assert revised["schema"] == "action-aware-constraint-pilot-manifest-v1"
+    assert revised["configs"][0]["mbbt"] == 2048
+    assert revised["configs"][3]["mbbt"] == 2048
+    assert revised["budget"]["hard_cap_h20_hours"] < 8.0
+    controller.configure(args, revised)
+    revised_plan = controller.dry_run_plan(args, revised)
+    assert revised_plan["projected_h20_hours"] < revised_plan["hard_cap_h20_hours"]
+    assert (
+        "--max-num-batched-tokens 2048"
+        in revised_plan["sessions"][0]["commands"]["server"]
+    )
+    accepted_burnin = {
+        "kind": "anchor",
+        "selection": {"count": 510},
+        "interval": {"elapsed_s": 61.25},
+        "pass_rate": 0.5,
+        "feasible": False,
+    }
+    assert controller.burnin_gate(
+        accepted_burnin, expected_count=510, maximum_elapsed_s=90.0
+    )["elapsed_s"] == 61.25
+    warmup = copy.deepcopy(accepted_burnin)
+    warmup["kind"] = "warmup"
+    try:
+        controller.burnin_gate(warmup, expected_count=510, maximum_elapsed_s=90.0)
+    except RuntimeError as error:
+        assert "non-anchor" in str(error)
+    else:
+        raise AssertionError("warmup incorrectly passed the burnin gate")
+    slow = copy.deepcopy(accepted_burnin)
+    slow["interval"]["elapsed_s"] = 91.0
+    try:
+        controller.burnin_gate(slow, expected_count=510, maximum_elapsed_s=90.0)
+    except RuntimeError as error:
+        assert "throughput gate failed" in str(error)
+    else:
+        raise AssertionError("slow burnin incorrectly passed the throughput gate")
+    print("action-aware constraint pilot: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/active-intervention-v0/analyze_prospective.py

@@ -0,0 +1,325 @@
+#!/usr/bin/env python3
+"""Audit held-out action/measurement choices against the exact 2x2 surface."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+import os
+import statistics
+from pathlib import Path
+from typing import Any, Mapping
+
+
+SCHEMA = "active-intervention-prospective-audit-v0"
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix(path.suffix + ".tmp")
+    temporary.write_text(
+        json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+    )
+    os.replace(temporary, path)
+
+
+def numeric(values: list[float]) -> dict[str, Any]:
+    finite = [float(value) for value in values]
+    if not finite or any(not math.isfinite(value) for value in finite):
+        raise ValueError("numeric summary requires finite values")
+    return {
+        "n": len(finite),
+        "min": min(finite),
+        "max": max(finite),
+        "distinct_n": len(set(finite)),
+    }
+
+
+def load_surface(
+    manifest: Mapping[str, Any], run_root: Path
+) -> tuple[dict[str, Any], list[dict[str, Any]]]:
+    rows = []
+    aggregate = {}
+    duration_s = float(manifest["engine"]["duration_s"])
+    tp = int(manifest["engine"]["tp"])
+    for config in manifest["configs"]:
+        config_id = str(config["id"])
+        values = []
+        for repetition in sorted(int(key) for key in manifest["repetitions"]):
+            expected = manifest["repetitions"][str(repetition)]["selection"]
+            result_path = (
+                run_root / "sessions" / config_id / f"rep{repetition}" / "result.json"
+            )
+            result = json.loads(result_path.read_text(encoding="utf-8"))
+            if result["selection"]["request_id_order_sha256"] != expected[
+                "request_id_order_sha256"
+            ]:
+                raise ValueError(f"request hash mismatch: {config_id} rep{repetition}")
+            offered_total = float(expected["offered_req_s_per_gpu"]) * tp
+            normalized = float(result["slo_pass_count"]) / duration_s / offered_total
+            values.append(normalized)
+            rows.append(
+                {
+                    "config_id": config_id,
+                    "mns": int(config["mns"]),
+                    "mbbt": int(config["mbbt"]),
+                    "repetition": repetition,
+                    "normalized_slo_goodput": normalized,
+                    "slo_goodput_req_s": float(result["slo_pass_count"]) / duration_s,
+                    "pass_rate": float(result["pass_rate"]),
+                    "elapsed_s": float(result["interval"]["elapsed_s"]),
+                    "result": str(result_path),
+                    "result_sha256": sha256_file(result_path),
+                }
+            )
+        aggregate[config_id] = {
+            "normalized_slo_goodput_values": values,
+            "median_normalized_slo_goodput": float(statistics.median(values)),
+            "sanity": numeric(values),
+        }
+    return aggregate, rows
+
+
+def source_cost_estimate(
+    *,
+    source_session: Mapping[str, Any],
+    source_rows: list[Mapping[str, Any]],
+    cutoff_s: float,
+    tp: int,
+) -> dict[str, float]:
+    actual_h20_hours = float(source_session["gpu_hours"])
+    measured_replay_h20_hours = (
+        tp * sum(float(row["elapsed_s"]) for row in source_rows) / 3600.0
+    )
+    fixed_h20_hours = max(0.0, actual_h20_hours - measured_replay_h20_hours)
+    prefix_replay_h20_hours = tp * len(source_rows) * cutoff_s / 3600.0
+    return {
+        "actual_full_session_h20_hours": actual_h20_hours,
+        "fixed_startup_warmup_burnin_cleanup_h20_hours": fixed_h20_hours,
+        "prefix_replay_h20_hours_lower_bound": prefix_replay_h20_hours,
+        "counterfactual_all_in_h20_hours_lower_bound": fixed_h20_hours
+        + prefix_replay_h20_hours,
+    }
+
+
+def replay_policy(
+    *,
+    mode: str,
+    manifest: Mapping[str, Any],
+    decision: Mapping[str, Any],
+    surface: Mapping[str, Any],
+    session_costs: Mapping[str, float],
+    source_cost: Mapping[str, float],
+) -> dict[str, Any]:
+    acceptable_regret = float(manifest["gates"]["acceptable_regret"])
+    source_id = str(manifest["source_config_id"])
+    oracle = max(
+        float(item["median_normalized_slo_goodput"]) for item in surface.values()
+    )
+    cumulative = float(source_cost["counterfactual_all_in_h20_hours_lower_bound"])
+    source_score = float(surface[source_id]["median_normalized_slo_goodput"])
+    source_regret = 1.0 - source_score / oracle if oracle > 0 else 0.0
+    points = [
+        {
+            "action_id": "noop",
+            "config_id": source_id,
+            "score": source_score,
+            "regret": source_regret,
+            "cumulative_h20_hours_lower_bound": cumulative,
+        }
+    ]
+    hit = points[0] if source_regret <= acceptable_regret + 1e-12 else None
+    seen = {source_id}
+    for action_id in decision["decisions"][mode]["intervention_order"]:
+        config_id = str(manifest["actions"][action_id])
+        if config_id in seen:
+            continue
+        seen.add(config_id)
+        cumulative += float(session_costs[config_id])
+        score = float(surface[config_id]["median_normalized_slo_goodput"])
+        regret = 1.0 - score / oracle if oracle > 0 else 0.0
+        point = {
+            "action_id": action_id,
+            "config_id": config_id,
+            "score": score,
+            "regret": regret,
+            "cumulative_h20_hours_lower_bound": cumulative,
+        }
+        points.append(point)
+        if hit is None and regret <= acceptable_regret + 1e-12:
+            hit = point
+    return {
+        "mode": mode,
+        "measurement_cutoff_s": float(
+            decision["decisions"][mode]["selected_cutoff_s"]
+        ),
+        "selected_action": decision["decisions"][mode]["selected_action"],
+        "decision_kind": decision["decisions"][mode]["decision_kind"],
+        "intervention_order": decision["decisions"][mode]["intervention_order"],
+        "source_cost": dict(source_cost),
+        "oracle_normalized_slo_goodput": oracle,
+        "cost_to_acceptable": hit,
+        "reached_acceptable": hit is not None,
+        "points": points,
+    }
+
+
+def build_audit(
+    *, manifest_path: Path, decision_path: Path, run_root: Path
+) -> dict[str, Any]:
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    decision = json.loads(decision_path.read_text(encoding="utf-8"))
+    state_path = run_root / "controller-state.json"
+    state = json.loads(state_path.read_text(encoding="utf-8"))
+    if manifest.get("schema") != "active-intervention-prospective-manifest-v0":
+        raise ValueError("unexpected prospective manifest schema")
+    if decision.get("schema") != "active-intervention-prospective-decision-v0":
+        raise ValueError("unexpected prospective decision schema")
+    if decision["manifest_sha256"] != sha256_file(manifest_path):
+        raise ValueError("decision does not match prospective manifest")
+    surface, rows = load_surface(manifest, run_root)
+    source_id = str(manifest["source_config_id"])
+    sessions = state["sessions"]
+    session_costs = {
+        config_id: float(sessions[config_id]["gpu_hours"])
+        for config_id in surface
+    }
+    source_rows = [row for row in rows if row["config_id"] == source_id]
+    policies = {}
+    for mode in ("outcome_only", "telemetry"):
+        cost = source_cost_estimate(
+            source_session=sessions[source_id],
+            source_rows=source_rows,
+            cutoff_s=float(decision["decisions"][mode]["selected_cutoff_s"]),
+            tp=int(manifest["engine"]["tp"]),
+        )
+        policies[mode] = replay_policy(
+            mode=mode,
+            manifest=manifest,
+            decision=decision,
+            surface=surface,
+            session_costs=session_costs,
+            source_cost=cost,
+        )
+    outcome_hit = policies["outcome_only"]["cost_to_acceptable"]
+    telemetry_hit = policies["telemetry"]["cost_to_acceptable"]
+    if outcome_hit is None or telemetry_hit is None:
+        reduction = None
+    else:
+        outcome_cost = float(outcome_hit["cumulative_h20_hours_lower_bound"])
+        telemetry_cost = float(telemetry_hit["cumulative_h20_hours_lower_bound"])
+        reduction = 1.0 - telemetry_cost / outcome_cost if outcome_cost > 0 else 0.0
+    confirmation_trigger = bool(
+        reduction is not None
+        and reduction
+        >= float(manifest["gates"]["confirmation_trigger_gpu_cost_reduction"])
+        and policies["telemetry"]["reached_acceptable"]
+    )
+    contribution_gate = bool(
+        reduction is not None
+        and reduction >= float(manifest["gates"]["contribution_gpu_cost_reduction"])
+        and policies["telemetry"]["reached_acceptable"]
+    )
+    status = (
+        "TRIGGER_ACTUAL_EARLY_STOP_CONFIRMATION"
+        if confirmation_trigger
+        else "STOP_NO_PROSPECTIVE_GPU_COST_SIGNAL"
+    )
+    normalized_values = [float(row["normalized_slo_goodput"]) for row in rows]
+    costs = list(session_costs.values())
+    invariants = {
+        "controller_complete": state.get("status") == "complete",
+        "four_sessions_complete": len(sessions) == 4
+        and all(item.get("status") == "complete" for item in sessions.values()),
+        "twelve_surface_outcomes": len(rows) == 12,
+        "nonnegative_goodput": all(value >= 0.0 for value in normalized_values),
+        "normalized_goodput_bounded": all(value <= 1.0 + 1e-12 for value in normalized_values),
+        "surface_not_all_identical": len(set(normalized_values)) > 1,
+        "nonnegative_session_costs": all(value >= 0.0 for value in costs),
+        "policy_replay_reaches_oracle_surface": all(
+            policy["reached_acceptable"] for policy in policies.values()
+        ),
+    }
+    red_flags = [name for name, passed in invariants.items() if not passed]
+    if red_flags:
+        status = "STOP_SANITY"
+    return {
+        "schema": SCHEMA,
+        "status": status,
+        "claim_boundary": (
+            "Prospective exact-surface replay. Prefix source costs reconstruct the "
+            "measured fixed overhead plus selected replay seconds; actual early-stop "
+            "confirmation is required before claiming GPU-cost reduction."
+        ),
+        "manifest": str(manifest_path),
+        "manifest_sha256": sha256_file(manifest_path),
+        "decision": str(decision_path),
+        "decision_sha256": sha256_file(decision_path),
+        "controller_state": str(state_path),
+        "controller_state_sha256": sha256_file(state_path),
+        "surface": surface,
+        "rows": rows,
+        "session_costs_h20_hours": session_costs,
+        "annotation_campaign_h20_hours": float(state["gpu_hours_total"]),
+        "policies": policies,
+        "comparison": {
+            "telemetry_gpu_cost_reduction_fraction": reduction,
+            "confirmation_trigger": confirmation_trigger,
+            "contribution_gate": contribution_gate,
+            "confirmation_trigger_threshold": manifest["gates"][
+                "confirmation_trigger_gpu_cost_reduction"
+            ],
+            "contribution_threshold": manifest["gates"][
+                "contribution_gpu_cost_reduction"
+            ],
+            "action_changed": policies["outcome_only"]["selected_action"]
+            != policies["telemetry"]["selected_action"],
+            "measurement_changed": policies["outcome_only"]["measurement_cutoff_s"]
+            != policies["telemetry"]["measurement_cutoff_s"],
+        },
+        "sanity": {
+            "invariants": invariants,
+            "red_flags": red_flags,
+            "normalized_slo_goodput": numeric(normalized_values),
+            "session_h20_hours": numeric(costs),
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--decision", type=Path, required=True)
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    audit = build_audit(
+        manifest_path=args.manifest,
+        decision_path=args.decision,
+        run_root=args.run_root,
+    )
+    atomic_json(args.output, audit)
+    print(
+        json.dumps(
+            {
+                "status": audit["status"],
+                "comparison": audit["comparison"],
+                "sanity": audit["sanity"],
+            },
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/active-intervention-v0/extract_training.py

@@ -0,0 +1,324 @@
+#!/usr/bin/env python3
+"""Extract paired source/action examples from the accepted action-aware run."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+import os
+import sys
+from pathlib import Path
+from statistics import fmean
+from typing import Any, Mapping
+
+
+PHASES = ("0.25", "0.50", "0.75", "1.00")
+HERE = Path(__file__).resolve().parent
+COMMON_STATE = HERE.parent / "telemetry-residual"
+sys.path.insert(0, str(COMMON_STATE))
+
+from common_state import summarize_engine  # noqa: E402
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix(path.suffix + ".tmp")
+    temporary.write_text(
+        json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+    )
+    os.replace(temporary, path)
+
+
+def load_jsonl(path: Path) -> list[dict[str, Any]]:
+    records = []
+    with path.open(encoding="utf-8") as source:
+        for line_number, line in enumerate(source, 1):
+            if not line.strip():
+                continue
+            try:
+                records.append(json.loads(line))
+            except json.JSONDecodeError as error:
+                raise ValueError(f"{path}:{line_number}: invalid JSON") from error
+    if not records:
+        raise ValueError(f"{path}: no request records")
+    return records
+
+
+def prefix_outcome(
+    requests: list[Mapping[str, Any]], *, cutoff_s: float, offered_total: float
+) -> dict[str, float]:
+    admitted = [request for request in requests if float(request["arrival_s"]) <= cutoff_s]
+    completed = [
+        request
+        for request in requests
+        if request.get("completed_elapsed_s") is not None
+        and float(request["completed_elapsed_s"]) <= cutoff_s
+    ]
+    if not admitted:
+        raise ValueError("prefix has no admitted requests")
+    admitted_ids = {str(request["request_id"]) for request in admitted}
+    if any(str(request["request_id"]) not in admitted_ids for request in completed):
+        raise ValueError("prefix completion precedes admission")
+    passed = sum(bool(request["slo_pass"]) for request in completed)
+    ttft = [float(request["ttft_ms"]) for request in completed]
+    tpot = [float(request["tpot_ms"]) for request in completed]
+    total = len(requests)
+    return {
+        "normalized_slo_goodput": passed / cutoff_s / offered_total,
+        "admitted_fraction": len(admitted) / total,
+        "completed_over_admitted": len(completed) / len(admitted),
+        "completed_pass_rate": passed / max(1, len(completed)),
+        "completed_fail_fraction_of_total": (len(completed) - passed) / total,
+        "outstanding_over_admitted": (len(admitted) - len(completed)) / len(admitted),
+        "ttft_max_over_slo_max": max(ttft, default=0.0) / 6000.0,
+        "ttft_mean_over_slo_max": fmean(ttft) / 6000.0 if ttft else 0.0,
+        "tpot_max_over_slo": max(tpot, default=0.0) / 50.0,
+        "tpot_mean_over_slo": fmean(tpot) / 50.0 if tpot else 0.0,
+        "admitted_input_tokens_mean_over_limit": fmean(
+            float(request["raw_input_tokens"]) for request in admitted
+        )
+        / 8192.0,
+    }
+
+
+def telemetry_record(state: Mapping[str, Any]) -> dict[str, float]:
+    common = state["common"]
+    engine = state["engine_only"]
+    executed_steps = int(state["sanity"]["executed_steps"])
+    if executed_steps <= 0:
+        raise ValueError("telemetry phase contains no executed engine steps")
+    return {
+        "scheduler_steps_per_s": float(common["scheduler_steps_per_s"]),
+        "batch_size_mean": float(common["batch_size"]["mean"]),
+        "batch_size_cv": float(common["batch_size"]["cv"]),
+        "batch_tokens_mean": float(common["batch_tokens"]["mean"]),
+        "batch_tokens_cv": float(common["batch_tokens"]["cv"]),
+        "decode_batch_size_mean": float(common["decode_batch_size"]["mean"]),
+        "decode_batch_size_cv": float(common["decode_batch_size"]["cv"]),
+        "prefill_token_fraction": float(common["prefill_token_fraction"]),
+        "queue_waiting_mean": float(common["queue_waiting_mean"]),
+        "queue_running_mean": float(common["queue_running_mean"]),
+        "preemptions_per_step": float(common["preemptions"]) / executed_steps,
+        "kv_usage_mean": float(engine["kv_usage_mean"]),
+        "kv_usage_max": float(engine["kv_usage_max"]),
+        "kv_usage_end_minus_start": float(engine["kv_usage_end_minus_start"]),
+        "graph_none_share": float(engine["graph_none_share"]),
+        "graph_full_share": float(engine["graph_full_share"]),
+        "graph_padding_fraction": float(engine["graph_padding_fraction"]),
+    }
+
+
+def load_stream(path: Path, *, expected_sha256: str) -> list[dict[str, Any]]:
+    if sha256_file(path) != expected_sha256:
+        raise ValueError(f"engine stream hash mismatch: {path}")
+    decoded = load_jsonl(path)
+    records = [row for row in decoded if "step_index" in row]
+    if not records:
+        raise ValueError(f"engine stream has no Layer-1 records: {path}")
+    return records
+
+
+def build_dataset(
+    *, audit_path: Path, manifest_path: Path, run_root: Path
+) -> dict[str, Any]:
+    audit = json.loads(audit_path.read_text(encoding="utf-8"))
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    if audit.get("schema") != "action-aware-constraint-pilot-audit-v0":
+        raise ValueError("unexpected action-aware audit schema")
+    if audit["sanity"]["red_flags"]:
+        raise ValueError(f"action-aware audit red flags: {audit['sanity']['red_flags']}")
+    configs = {str(item["id"]): item for item in manifest["configs"]}
+    runs = {
+        (str(run["config_id"]), int(run["repetition"])): run
+        for run in audit["runs"]
+    }
+    source_ids = {str(regime["source"]) for regime in manifest["regimes"].values()}
+    stream_entries = {
+        str(item["config_id"]): item
+        for item in audit["streams"]
+        if str(item["config_id"]) in source_ids
+    }
+    if set(stream_entries) != source_ids:
+        raise ValueError("audit is missing a source config engine stream")
+    streams = {
+        config_id: load_stream(
+            Path(item["stream"]), expected_sha256=str(item["stream_sha256"])
+        )
+        for config_id, item in stream_entries.items()
+    }
+    examples = []
+    request_hashes = []
+    for regime_name, regime in sorted(manifest["regimes"].items()):
+        source_id = str(regime["source"])
+        for repetition in sorted(int(value) for value in manifest["repetitions"]):
+            source_run = runs[(source_id, repetition)]
+            source_config = configs[source_id]
+            request_path = run_root / "sessions" / source_id / f"rep{repetition}" / "requests.jsonl"
+            requests = load_jsonl(request_path)
+            request_hashes.append(sha256_file(request_path))
+            offered_rate_per_gpu = float(
+                manifest["repetitions"][str(repetition)]["selection"][
+                    "offered_req_s_per_gpu"
+                ]
+            )
+            offered_total = offered_rate_per_gpu * int(manifest["engine"]["tp"])
+            source_goodput = float(source_run["outcome"]["slo_goodput_req_s"])
+            source_normalized = min(1.0, source_goodput / offered_total)
+            decision_id = f"{regime_name}-rep{repetition}"
+            for phase in PHASES:
+                cutoff_s = float(manifest["engine"]["duration_s"]) * float(phase)
+                outcome = prefix_outcome(
+                    requests, cutoff_s=cutoff_s, offered_total=offered_total
+                )
+                admitted_count = sum(
+                    float(request["arrival_s"]) <= cutoff_s for request in requests
+                )
+                start_ns = int(source_run["state"]["interval"]["start_ns"])
+                phase_state = summarize_engine(
+                    streams[source_id],
+                    start_ns=start_ns,
+                    end_ns=start_ns + round(cutoff_s * 1e9),
+                    request_count=admitted_count,
+                )
+                if not all(phase_state["sanity"]["invariants"].values()):
+                    raise ValueError(
+                        f"engine state invariant failed: {decision_id} phase {phase}"
+                    )
+                telemetry = telemetry_record(phase_state)
+                actions = {"noop": source_id, **regime["actions"]}
+                for action_name, target_id in sorted(actions.items()):
+                    target_run = runs[(str(target_id), repetition)]
+                    target_config = configs[str(target_id)]
+                    target_goodput = float(target_run["outcome"]["slo_goodput_req_s"])
+                    normalized = target_goodput / offered_total
+                    if not 0.0 <= normalized <= 1.0 + 1e-12:
+                        raise ValueError("target normalized goodput is outside [0, 1]")
+                    examples.append(
+                        {
+                            "phase": phase,
+                            "cutoff_s": cutoff_s,
+                            "decision_id": decision_id,
+                            "regime": regime_name,
+                            "repetition": repetition,
+                            "source": {
+                                "config_id": source_id,
+                                "mns": int(source_config["mns"]),
+                                "mbbt": int(source_config["mbbt"]),
+                                "offered_rate_per_gpu": offered_rate_per_gpu,
+                                "outcome": outcome,
+                                "telemetry": telemetry,
+                            },
+                            "action": {
+                                "id": action_name,
+                                "target_config_id": str(target_id),
+                                "target_mns": int(target_config["mns"]),
+                                "target_mbbt": int(target_config["mbbt"]),
+                            },
+                            "target_slo_goodput_req_s": target_goodput,
+                            "target_normalized_goodput": min(1.0, normalized),
+                            "source_normalized_goodput": source_normalized,
+                            "target_delta_normalized_goodput": min(1.0, normalized)
+                            - source_normalized,
+                        }
+                    )
+    invariants = {
+        "expected_examples": len(examples) == len(PHASES) * 2 * 3 * 3,
+        "four_phases": sorted({example["phase"] for example in examples})
+        == sorted(PHASES),
+        "six_decisions": len({example["decision_id"] for example in examples}) == 6,
+        "three_actions_per_decision_phase": all(
+            sum(
+                item["decision_id"] == decision
+                and item["phase"] == phase
+                for item in examples
+            )
+            == 3
+            for decision in {item["decision_id"] for item in examples}
+            for phase in PHASES
+        ),
+        "targets_not_all_identical": len(
+            {example["target_normalized_goodput"] for example in examples}
+        )
+        > 1,
+        "bounded_prefix_ratios": all(
+            0.0 <= float(value) <= 1.0
+            for example in examples
+            for key, value in example["source"]["outcome"].items()
+            if key
+            in {
+                "admitted_fraction",
+                "completed_over_admitted",
+                "completed_pass_rate",
+                "completed_fail_fraction_of_total",
+                "outstanding_over_admitted",
+            }
+        ),
+        "direct_telemetry_without_binding_labels": all(
+            not any(token in key for token in ("exclusive", "unresolved", "both"))
+            for example in examples
+            for key in example["source"]["telemetry"]
+        ),
+        "treatment_effects_bounded": all(
+            -1.0 <= float(example["target_delta_normalized_goodput"]) <= 1.0
+            for example in examples
+        ),
+    }
+    red_flags = [name for name, passed in invariants.items() if not passed]
+    if red_flags:
+        raise RuntimeError(f"training dataset sanity failed: {red_flags}")
+    return {
+        "schema": "active-intervention-training-v0",
+        "status": "VALID",
+        "provenance": {
+            "audit": str(audit_path),
+            "audit_sha256": sha256_file(audit_path),
+            "manifest": str(manifest_path),
+            "manifest_sha256": sha256_file(manifest_path),
+            "run_root": str(run_root),
+            "source_request_sha256": sorted(set(request_hashes)),
+            "source_stream_sha256": sorted(
+                str(item["stream_sha256"]) for item in stream_entries.values()
+            ),
+        },
+        "examples": examples,
+        "sanity": {"invariants": invariants, "red_flags": red_flags},
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--audit", type=Path, required=True)
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    dataset = build_dataset(
+        audit_path=args.audit,
+        manifest_path=args.manifest,
+        run_root=args.run_root,
+    )
+    atomic_json(args.output, dataset)
+    print(
+        json.dumps(
+            {
+                "status": dataset["status"],
+                "examples": len(dataset["examples"]),
+                "sanity": dataset["sanity"],
+            },
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/active-intervention-v0/model.py

@@ -0,0 +1,287 @@
+#!/usr/bin/env python3
+"""Small-data action-response model for the active intervention pilot.
+
+The model predicts the paired normalized SLO-goodput treatment effect from a
+source measurement and a full MNS/MBBT action.  Telemetry features are direct,
+continuous engine measurements; there is no diagnosis-to-action rule here.
+"""
+
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass
+from typing import Any, Iterable, Mapping, Sequence
+
+import numpy as np
+
+
+PREFIX_FEATURES = (
+    "normalized_slo_goodput",
+    "admitted_fraction",
+    "completed_over_admitted",
+    "completed_pass_rate",
+    "completed_fail_fraction_of_total",
+    "outstanding_over_admitted",
+    "ttft_max_over_slo_max",
+    "ttft_mean_over_slo_max",
+    "tpot_max_over_slo",
+    "tpot_mean_over_slo",
+    "admitted_input_tokens_mean_over_limit",
+)
+
+TELEMETRY_FEATURES = (
+    "scheduler_steps_per_s",
+    "batch_size_mean",
+    "batch_size_cv",
+    "batch_tokens_mean",
+    "batch_tokens_cv",
+    "decode_batch_size_mean",
+    "decode_batch_size_cv",
+    "prefill_token_fraction",
+    "queue_waiting_mean",
+    "queue_running_mean",
+    "preemptions_per_step",
+    "kv_usage_mean",
+    "kv_usage_max",
+    "kv_usage_end_minus_start",
+    "graph_none_share",
+    "graph_full_share",
+    "graph_padding_fraction",
+)
+
+
+def _finite(value: Any, name: str) -> float:
+    result = float(value)
+    if not math.isfinite(result):
+        raise ValueError(f"{name} must be finite")
+    return result
+
+
+def feature_vector(
+    example: Mapping[str, Any], *, include_telemetry: bool
+) -> tuple[list[str], np.ndarray]:
+    source = example["source"]
+    action = example["action"]
+    source_log_mns = math.log2(_finite(source["mns"], "source MNS"))
+    source_log_mbbt = math.log2(_finite(source["mbbt"], "source MBBT"))
+    target_log_mns = math.log2(_finite(action["target_mns"], "target MNS"))
+    target_log_mbbt = math.log2(_finite(action["target_mbbt"], "target MBBT"))
+    delta_mns = target_log_mns - source_log_mns
+    delta_mbbt = target_log_mbbt - source_log_mbbt
+    names = [
+        "source_log2_mns",
+        "source_log2_mbbt",
+        "target_log2_mns",
+        "target_log2_mbbt",
+        "delta_log2_mns",
+        "delta_log2_mbbt",
+        "delta_product",
+        "offered_rate_per_gpu",
+    ]
+    values = [
+        source_log_mns,
+        source_log_mbbt,
+        target_log_mns,
+        target_log_mbbt,
+        delta_mns,
+        delta_mbbt,
+        delta_mns * delta_mbbt,
+        _finite(source["offered_rate_per_gpu"], "offered rate"),
+    ]
+    for name in PREFIX_FEATURES:
+        names.append(f"outcome.{name}")
+        values.append(_finite(source["outcome"][name], name))
+    if include_telemetry:
+        for name in TELEMETRY_FEATURES:
+            value = _finite(source["telemetry"][name], name)
+            names.extend(
+                (
+                    f"telemetry.{name}",
+                    f"telemetry.{name}*delta_mns",
+                    f"telemetry.{name}*delta_mbbt",
+                )
+            )
+            values.extend((value, value * delta_mns, value * delta_mbbt))
+    vector = np.asarray(values, dtype=np.float64)
+    if not np.all(np.isfinite(vector)):
+        raise ValueError("feature vector contains a non-finite value")
+    return names, vector
+
+
+@dataclass(frozen=True)
+class RidgeModel:
+    feature_names: tuple[str, ...]
+    mean: np.ndarray
+    scale: np.ndarray
+    weights: np.ndarray
+    intercept: float
+    regularization: float
+
+    def predict(self, values: np.ndarray) -> float:
+        if values.shape != self.mean.shape:
+            raise ValueError("ridge prediction feature shape mismatch")
+        normalized = (values - self.mean) / self.scale
+        return float(self.intercept + normalized @ self.weights)
+
+    def to_json(self) -> dict[str, Any]:
+        return {
+            "feature_names": list(self.feature_names),
+            "mean": self.mean.tolist(),
+            "scale": self.scale.tolist(),
+            "weights": self.weights.tolist(),
+            "intercept": self.intercept,
+            "regularization": self.regularization,
+        }
+
+    @classmethod
+    def from_json(cls, payload: Mapping[str, Any]) -> "RidgeModel":
+        return cls(
+            feature_names=tuple(str(value) for value in payload["feature_names"]),
+            mean=np.asarray(payload["mean"], dtype=np.float64),
+            scale=np.asarray(payload["scale"], dtype=np.float64),
+            weights=np.asarray(payload["weights"], dtype=np.float64),
+            intercept=float(payload["intercept"]),
+            regularization=float(payload["regularization"]),
+        )
+
+
+def fit_ridge(
+    examples: Sequence[Mapping[str, Any]],
+    *,
+    include_telemetry: bool,
+    regularization: float,
+) -> RidgeModel:
+    if not examples:
+        raise ValueError("ridge fit requires examples")
+    if regularization <= 0:
+        raise ValueError("ridge regularization must be positive")
+    encoded = [
+        feature_vector(example, include_telemetry=include_telemetry)
+        for example in examples
+    ]
+    names = encoded[0][0]
+    if any(item[0] != names for item in encoded):
+        raise ValueError("feature names changed across examples")
+    x = np.stack([item[1] for item in encoded])
+    y = np.asarray(
+        [
+            _finite(example["target_delta_normalized_goodput"], "target effect")
+            for example in examples
+        ],
+        dtype=np.float64,
+    )
+    mean = x.mean(axis=0)
+    scale = x.std(axis=0)
+    scale[scale < 1e-12] = 1.0
+    normalized = (x - mean) / scale
+    intercept = float(y.mean())
+    centered = y - intercept
+    system = normalized.T @ normalized + regularization * np.eye(x.shape[1])
+    weights = np.linalg.solve(system, normalized.T @ centered)
+    return RidgeModel(
+        feature_names=tuple(names),
+        mean=mean,
+        scale=scale,
+        weights=weights,
+        intercept=intercept,
+        regularization=regularization,
+    )
+
+
+def fit_jackknife_ensemble(
+    examples: Sequence[Mapping[str, Any]],
+    *,
+    include_telemetry: bool,
+    regularization: float,
+    group_key: str = "decision_id",
+) -> list[RidgeModel]:
+    groups = sorted({str(example[group_key]) for example in examples})
+    if len(groups) < 3:
+        raise ValueError("jackknife ensemble requires at least three groups")
+    models = []
+    for held_out in groups:
+        training = [
+            example for example in examples if str(example[group_key]) != held_out
+        ]
+        models.append(
+            fit_ridge(
+                training,
+                include_telemetry=include_telemetry,
+                regularization=regularization,
+            )
+        )
+    return models
+
+
+def ensemble_predict(
+    models: Sequence[RidgeModel],
+    example: Mapping[str, Any],
+    *,
+    include_telemetry: bool,
+) -> dict[str, float]:
+    if not models:
+        raise ValueError("ensemble prediction requires models")
+    source = example["source"]
+    action = example["action"]
+    if (
+        int(action["target_mns"]) == int(source["mns"])
+        and int(action["target_mbbt"]) == int(source["mbbt"])
+    ):
+        return {"mean": 0.0, "std": 0.0, "min": 0.0, "max": 0.0, "distinct_n": 1}
+    names, values = feature_vector(example, include_telemetry=include_telemetry)
+    if any(model.feature_names != tuple(names) for model in models):
+        raise ValueError("ensemble feature schema mismatch")
+    raw = np.asarray([model.predict(values) for model in models], dtype=np.float64)
+    clipped = np.clip(raw, -1.0, 1.0)
+    return {
+        "mean": float(clipped.mean()),
+        "std": float(clipped.std(ddof=0)),
+        "min": float(clipped.min()),
+        "max": float(clipped.max()),
+        "distinct_n": len(set(float(value) for value in clipped)),
+    }
+
+
+def select_action(
+    models: Sequence[RidgeModel],
+    candidates: Sequence[Mapping[str, Any]],
+    *,
+    include_telemetry: bool,
+    confidence_z: float = 1.0,
+    minimum_margin: float = 0.02,
+) -> dict[str, Any]:
+    if len(candidates) < 2:
+        raise ValueError("action selection requires at least two candidates")
+    rows = []
+    for example in candidates:
+        prediction = ensemble_predict(
+            models, example, include_telemetry=include_telemetry
+        )
+        rows.append(
+            {
+                "action_id": str(example["action"]["id"]),
+                "prediction": prediction,
+                "lower": prediction["mean"] - confidence_z * prediction["std"],
+                "upper": prediction["mean"] + confidence_z * prediction["std"],
+            }
+        )
+    rows.sort(key=lambda row: (-row["prediction"]["mean"], row["action_id"]))
+    best, second = rows[:2]
+    margin = float(best["prediction"]["mean"] - second["prediction"]["mean"])
+    confident = bool(
+        margin >= minimum_margin and best["lower"] > second["upper"]
+    )
+    return {
+        "selected_action": best["action_id"],
+        "confident": confident,
+        "predicted_margin": margin,
+        "candidates": rows,
+    }
+
+
+def models_to_json(models: Iterable[RidgeModel]) -> list[dict[str, Any]]:
+    return [model.to_json() for model in models]
+
+
+def models_from_json(payload: Iterable[Mapping[str, Any]]) -> list[RidgeModel]:
+    return [RidgeModel.from_json(item) for item in payload]

runs/active-intervention-v0/prepare_prospective.py

@@ -0,0 +1,363 @@
+#!/usr/bin/env python3
+"""Freeze the unseen-trace 2x2 active intervention development surface."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+
+AITUNER_ROOT = Path(os.environ.get("AITUNER_ROOT", Path(__file__).resolve().parents[2]))
+sys.path.insert(0, str(AITUNER_ROOT / "src"))
+
+from aituner.spec import load_study_spec  # noqa: E402
+from aituner.trace import load_trace_requests, select_requests_for_threshold  # noqa: E402
+
+
+SCHEMA = "active-intervention-prospective-manifest-v0"
+TP = 4
+REPETITIONS = (1, 2, 3)
+DURATION_S = 300.0
+REPLAY_TIME_SCALE = 0.5
+OFFERED_RATE_PER_GPU = 2.75
+TARGET_COUNT = round(OFFERED_RATE_PER_GPU * DURATION_S * TP)
+WINDOW_ID = "chat_w20260313_1000"
+ENGINE_VERSION = "0.24.1.dev3+g668cfb7e2"
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix(path.suffix + ".tmp")
+    temporary.write_text(
+        json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+    )
+    os.replace(temporary, path)
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def order_hash(values: list[str]) -> str:
+    return hashlib.sha256("\n".join(values).encode()).hexdigest()
+
+
+def configs() -> list[dict[str, Any]]:
+    return [
+        {
+            "id": "source_mns32_mbbt4096",
+            "mns": 32,
+            "mbbt": 4096,
+            "repetition_order": [1, 2, 3],
+        },
+        {
+            "id": "mns64_mbbt4096",
+            "mns": 64,
+            "mbbt": 4096,
+            "repetition_order": [2, 3, 1],
+        },
+        {
+            "id": "mns32_mbbt8192",
+            "mns": 32,
+            "mbbt": 8192,
+            "repetition_order": [3, 1, 2],
+        },
+        {
+            "id": "joint_mns64_mbbt8192",
+            "mns": 64,
+            "mbbt": 8192,
+            "repetition_order": [1, 3, 2],
+        },
+    ]
+
+
+def partition_trace(source: Path, output_root: Path) -> dict[str, Any]:
+    source_sha = sha256_file(source)
+    output_root.mkdir(parents=True, exist_ok=True)
+    paths = {rep: output_root / f"rep{rep}.jsonl" for rep in REPETITIONS}
+    temporary = {rep: path.with_suffix(".jsonl.tmp") for rep, path in paths.items()}
+    handles = {rep: temporary[rep].open("w", encoding="utf-8") for rep in REPETITIONS}
+    counts = {rep: 0 for rep in REPETITIONS}
+    id_digests = {rep: hashlib.sha256() for rep in REPETITIONS}
+    total = 0
+    try:
+        with source.open(encoding="utf-8") as input_file:
+            for line_number, line in enumerate(input_file, start=1):
+                if not line.strip():
+                    continue
+                row = json.loads(line)
+                original_id = str(row.get("request_id") or row.get("id") or line_number)
+                digest = hashlib.sha256(
+                    f"{source_sha}:{line_number}:{original_id}".encode()
+                ).hexdigest()
+                repetition = int(digest[:16], 16) % len(REPETITIONS) + 1
+                row["request_id"] = f"active-r{repetition}-{digest}"
+                handles[repetition].write(json.dumps(row, ensure_ascii=False) + "\n")
+                counts[repetition] += 1
+                total += 1
+                id_digests[repetition].update(row["request_id"].encode() + b"\n")
+    finally:
+        for handle in handles.values():
+            handle.close()
+    for repetition in REPETITIONS:
+        os.replace(temporary[repetition], paths[repetition])
+    partitions = {
+        str(rep): {
+            "path": str(paths[rep]),
+            "rows": counts[rep],
+            "bytes": paths[rep].stat().st_size,
+            "sha256": sha256_file(paths[rep]),
+            "request_id_order_sha256": id_digests[rep].hexdigest(),
+        }
+        for rep in REPETITIONS
+    }
+    return {
+        "source": str(source),
+        "source_sha256": source_sha,
+        "source_rows": total,
+        "partition_rule": "sha256(source_sha:line_number:original_id) modulo 3",
+        "partitions": partitions,
+    }
+
+
+def materialize_study(
+    base_study: Path,
+    target: Path,
+    *,
+    repetition: int,
+    trace_path: Path,
+    windows_path: Path,
+) -> None:
+    payload = json.loads(base_study.read_text(encoding="utf-8"))
+    payload["study_id"] = f"active-intervention-trace13-rep{repetition}"
+    payload["hardware"]["host_candidates"] = ["dash0"]
+    payload["engine"]["engine_version"] = ENGINE_VERSION
+    trace = payload["trace"]
+    trace.update(
+        {
+            "windows_path": str(windows_path),
+            "window_id": WINDOW_ID,
+            "trace_file_override": str(trace_path),
+            "completion_tokens_override": 128,
+            "replay_time_scale": REPLAY_TIME_SCALE,
+            "early_stop_max_lag_s": None,
+            "early_stop_max_elapsed_s": 360.0,
+            "restart_engine_after_early_stop": False,
+            "adaptive_stop": {"enabled": False},
+        }
+    )
+    atomic_json(target, payload)
+
+
+def attainable_anchor(requests: list[Any], target_count: int) -> tuple[float, list[Any]]:
+    ordered = sorted(float(request.sampling_u) for request in requests)
+    if target_count <= 0 or target_count > len(ordered):
+        raise ValueError(
+            f"target count {target_count} is outside available range 1..{len(ordered)}"
+        )
+    candidates = []
+    for index in sorted({target_count - 1, min(target_count, len(ordered) - 1)}):
+        anchor = ordered[index]
+        selected = select_requests_for_threshold(requests, threshold=anchor)
+        candidates.append((abs(len(selected) - target_count), len(selected), anchor, selected))
+    _error, _count, anchor, selected = min(
+        candidates, key=lambda item: (item[0], item[1], item[2])
+    )
+    return anchor, selected
+
+
+def selection_record(selected: list[Any]) -> dict[str, Any]:
+    return {
+        "anchor": max(float(request.sampling_u) for request in selected),
+        "selected_count": len(selected),
+        "target_count": TARGET_COUNT,
+        "offered_req_s": len(selected) / DURATION_S,
+        "offered_req_s_per_gpu": len(selected) / DURATION_S / TP,
+        "request_id_order_sha256": order_hash([request.row_id for request in selected]),
+        "arrival_order_sha256": order_hash(
+            [f"{request.arrival_s:.12f}" for request in selected]
+        ),
+        "input_length_order_sha256": order_hash(
+            [str(request.prompt_tokens_hint) for request in selected]
+        ),
+    }
+
+
+def build(
+    *,
+    base_study: Path,
+    base_action_manifest: Path,
+    source_trace: Path,
+    windows_path: Path,
+    private_root: Path,
+    policy_path: Path,
+) -> dict[str, Any]:
+    base_manifest = json.loads(base_action_manifest.read_text(encoding="utf-8"))
+    if base_manifest.get("status") != "PASS":
+        raise ValueError("base action-aware manifest did not pass")
+    policy = json.loads(policy_path.read_text(encoding="utf-8"))
+    if policy.get("schema") != "active-intervention-policy-v0":
+        raise ValueError("unexpected frozen policy schema")
+    if policy.get("sanity", {}).get("red_flags"):
+        raise ValueError("frozen policy contains red flags")
+
+    partition = partition_trace(source_trace, private_root / "traces")
+    repetitions = {}
+    selected_sets: list[set[str]] = []
+    for repetition in REPETITIONS:
+        trace_path = Path(partition["partitions"][str(repetition)]["path"])
+        study_path = private_root / "studies" / f"rep{repetition}-tp4.json"
+        materialize_study(
+            base_study,
+            study_path,
+            repetition=repetition,
+            trace_path=trace_path,
+            windows_path=windows_path,
+        )
+        study = load_study_spec(study_path)
+        window, requests = load_trace_requests(study, study_spec_path=study_path)
+        duration_s = float(window.window_end - window.window_start)
+        if not math.isclose(duration_s, DURATION_S, abs_tol=1e-9):
+            raise ValueError(f"rep{repetition}: duration {duration_s} != {DURATION_S}")
+        _anchor, selected = attainable_anchor(requests, TARGET_COUNT)
+        record = selection_record(selected)
+        selected_sets.append({request.row_id for request in selected})
+        repetitions[str(repetition)] = {
+            "study": str(study_path),
+            "study_sha256": sha256_file(study_path),
+            "trace": partition["partitions"][str(repetition)],
+            "available_filtered_requests": len(requests),
+            "selection": record,
+        }
+
+    frozen_configs = configs()
+    config_ids = {str(config["id"]) for config in frozen_configs}
+    invariants = {
+        "three_nonempty_trace_partitions": all(
+            int(item["rows"]) > 0 for item in partition["partitions"].values()
+        ),
+        "partition_rows_conserved": sum(
+            int(item["rows"]) for item in partition["partitions"].values()
+        )
+        == int(partition["source_rows"]),
+        "selected_sets_disjoint": all(
+            not selected_sets[left] & selected_sets[right]
+            for left in range(len(selected_sets))
+            for right in range(left + 1, len(selected_sets))
+        ),
+        "target_count_attained": all(
+            abs(int(item["selection"]["selected_count"]) - TARGET_COUNT) <= 1
+            for item in repetitions.values()
+        ),
+        "four_unique_configs": len(config_ids) == 4,
+        "two_by_two_surface": {
+            (int(config["mns"]), int(config["mbbt"]))
+            for config in frozen_configs
+        }
+        == {(32, 4096), (64, 4096), (32, 8192), (64, 8192)},
+        "repetition_orders_are_permutations": all(
+            sorted(config["repetition_order"]) == list(REPETITIONS)
+            for config in frozen_configs
+        ),
+    }
+    red_flags = [name for name, passed in invariants.items() if not passed]
+    return {
+        "schema": SCHEMA,
+        "status": "PASS" if not red_flags else "STOP",
+        "source": {
+            "window_id": WINDOW_ID,
+            "source_trace": str(source_trace),
+            "source_trace_sha256": partition["source_sha256"],
+            "windows_path": str(windows_path),
+            "base_study": str(base_study),
+            "base_study_sha256": sha256_file(base_study),
+            "base_action_manifest": str(base_action_manifest),
+            "base_action_manifest_sha256": sha256_file(base_action_manifest),
+        },
+        "policy": {
+            "path": str(policy_path),
+            "sha256": sha256_file(policy_path),
+            "status": policy["status"],
+            "training": policy["training"],
+            "measurement_policy": policy["measurement_policy"],
+            "launch_reason": (
+                "bounded unseen-trace joint-action test after a negative narrow "
+                "retrospective replay"
+            ),
+        },
+        "engine": {
+            "tp": TP,
+            "duration_s": DURATION_S,
+            "client_timeout_s": 450.0,
+            "burnin_max_elapsed_s": 90.0,
+            "disable_slo_early_stop": True,
+        },
+        "burnin": base_manifest["burnin"],
+        "private": {"trace_partition": partition},
+        "repetitions": repetitions,
+        "configs": frozen_configs,
+        "source_config_id": "source_mns32_mbbt4096",
+        "actions": {
+            "noop": "source_mns32_mbbt4096",
+            "mns": "mns64_mbbt4096",
+            "mbbt": "mns32_mbbt8192",
+            "joint": "joint_mns64_mbbt8192",
+        },
+        "checkpoints": {
+            "fractions": [0.25, 0.50, 0.75, 1.0],
+            "seconds": [75.0, 150.0, 225.0, 300.0],
+        },
+        "gates": {
+            "acceptable_regret": 0.02,
+            "source_ceiling_normalized_goodput": 0.98,
+            "confirmation_trigger_gpu_cost_reduction": 0.10,
+            "contribution_gpu_cost_reduction": 0.20,
+            "maximum_task_regret": 0.05,
+        },
+        "budget": {
+            "hard_cap_h20_hours": 6.0,
+            "session_estimate_h20_hours": 1.3,
+            "safety_h20_hours": 0.3,
+            "expected_h20_hours": [4.6, 5.5],
+            "expected_wall_minutes": [75, 100],
+        },
+        "sanity": {"invariants": invariants, "red_flags": red_flags},
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base-study", type=Path, required=True)
+    parser.add_argument("--base-action-manifest", type=Path, required=True)
+    parser.add_argument("--source-trace", type=Path, required=True)
+    parser.add_argument("--windows-path", type=Path, required=True)
+    parser.add_argument("--private-root", type=Path, required=True)
+    parser.add_argument("--policy", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    payload = build(
+        base_study=args.base_study,
+        base_action_manifest=args.base_action_manifest,
+        source_trace=args.source_trace,
+        windows_path=args.windows_path,
+        private_root=args.private_root,
+        policy_path=args.policy,
+    )
+    atomic_json(args.output, payload)
+    print(json.dumps({"status": payload["status"], "sanity": payload["sanity"]}))
+    if payload["status"] != "PASS":
+        raise SystemExit("prospective manifest preflight failed")
+
+
+if __name__ == "__main__":
+    main()

runs/active-intervention-v0/prospective_controller.py

@@ -0,0 +1,198 @@
+#!/usr/bin/env python3
+"""Run source first, select the next intervention, then annotate the 2x2 surface."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Any, Mapping
+
+
+HERE = Path(__file__).resolve().parent
+ACTION_DIR = HERE.parent / "action-aware-v0"
+sys.path.insert(0, str(ACTION_DIR))
+sys.path.insert(0, str(HERE))
+
+import pilot_controller as action_controller  # noqa: E402
+import prospective_decision  # noqa: E402
+
+
+SCHEMA = "active-intervention-prospective-state-v0"
+
+
+def validate_inputs(args: argparse.Namespace, manifest: Mapping[str, Any]) -> None:
+    if manifest.get("schema") != "active-intervention-prospective-manifest-v0":
+        raise RuntimeError("unexpected active intervention manifest schema")
+    if manifest.get("status") != "PASS" or manifest["sanity"]["red_flags"]:
+        raise RuntimeError("active intervention manifest did not pass preflight")
+    required = {
+        "manifest": args.manifest,
+        "policy": args.policy,
+        "aituner_root": args.aituner_root,
+        "vllm_source": args.vllm_source,
+        "venv_python": args.venv / "bin/python",
+        "venv_vllm": args.venv / "bin/vllm",
+        "model": args.model,
+        "client": args.client,
+        "burnin_study": Path(manifest["burnin"]["study"]),
+    }
+    for repetition, item in manifest["repetitions"].items():
+        required[f"rep{repetition}_study"] = Path(item["study"])
+        required[f"rep{repetition}_trace"] = Path(item["trace"]["path"])
+    missing = {name: str(path) for name, path in required.items() if not path.exists()}
+    if missing:
+        raise RuntimeError(f"active intervention input paths missing: {missing}")
+    if prospective_decision.sha256_file(args.policy) != manifest["policy"]["sha256"]:
+        raise RuntimeError("active intervention policy hash mismatch")
+
+
+def dry_run(args: argparse.Namespace, manifest: Mapping[str, Any]) -> dict[str, Any]:
+    plan = action_controller.dry_run_plan(args, manifest)
+    return {
+        "schema": "active-intervention-prospective-dry-run-v0",
+        "status": "PASS",
+        "manifest": str(args.manifest),
+        "policy": str(args.policy),
+        "source_first": manifest["source_config_id"],
+        "post_source_order": "selected by telemetry policy; all remaining cells then annotated",
+        "candidate_actions": manifest["actions"],
+        "projected_h20_hours": plan["projected_h20_hours"],
+        "hard_cap_h20_hours": plan["hard_cap_h20_hours"],
+        "sessions": plan["sessions"],
+    }
+
+
+def load_or_build_decision(
+    *, args: argparse.Namespace, run_root: Path
+) -> dict[str, Any]:
+    path = run_root / "active-decision.json"
+    if path.exists():
+        decision = json.loads(path.read_text(encoding="utf-8"))
+        if decision.get("manifest_sha256") != prospective_decision.sha256_file(
+            args.manifest
+        ):
+            raise RuntimeError("existing active decision has a different manifest")
+        if decision.get("policy_sha256") != prospective_decision.sha256_file(args.policy):
+            raise RuntimeError("existing active decision has a different policy")
+        return decision
+    decision = prospective_decision.build_decision(
+        manifest_path=args.manifest,
+        policy_path=args.policy,
+        run_root=run_root,
+    )
+    prospective_decision.atomic_json(path, decision)
+    return decision
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("--manifest", type=Path, required=True)
+    result.add_argument("--policy", type=Path, required=True)
+    result.add_argument("--run-root", type=Path, required=True)
+    result.add_argument("--aituner-root", type=Path, required=True)
+    result.add_argument("--vllm-source", type=Path, required=True)
+    result.add_argument("--venv", type=Path, required=True)
+    result.add_argument("--model", type=Path, required=True)
+    result.add_argument("--client", type=Path, required=True)
+    result.add_argument("--dry-run", action="store_true")
+    return result
+
+
+def main() -> None:
+    args = parser().parse_args()
+    manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
+    validate_inputs(args, manifest)
+    action_controller.configure(args, manifest)
+    action_controller.base.MARKER = "active-intervention-prospective-v0"
+    if args.dry_run:
+        print(json.dumps(dry_run(args, manifest), indent=2, sort_keys=True))
+        return
+
+    args.run_root.mkdir(parents=True, exist_ok=True)
+    copied_manifest = args.run_root / "prospective-manifest.json"
+    if not copied_manifest.exists():
+        action_controller.atomic_json(copied_manifest, manifest)
+    state_path = args.run_root / "controller-state.json"
+    state = action_controller.load_state(
+        state_path, float(manifest["budget"]["hard_cap_h20_hours"])
+    )
+    state["schema"] = SCHEMA
+    state["status"] = "running"
+    action_controller.atomic_json(state_path, state)
+
+    configs = {str(item["id"]): dict(item) for item in manifest["configs"]}
+    config_indexes = {
+        str(item["id"]): index for index, item in enumerate(manifest["configs"])
+    }
+    source_id = str(manifest["source_config_id"])
+    action_controller.execute_session(
+        args=args,
+        manifest=manifest,
+        config=configs[source_id],
+        index=config_indexes[source_id],
+        state=state,
+        state_path=state_path,
+    )
+    decision = load_or_build_decision(args=args, run_root=args.run_root)
+    state["active_decision"] = {
+        "path": str(args.run_root / "active-decision.json"),
+        "status": decision["status"],
+        "outcome_only": {
+            key: decision["decisions"]["outcome_only"][key]
+            for key in ("selected_cutoff_s", "decision_kind", "selected_action")
+        },
+        "telemetry": {
+            key: decision["decisions"]["telemetry"][key]
+            for key in ("selected_cutoff_s", "decision_kind", "selected_action")
+        },
+    }
+    action_controller.atomic_json(state_path, state)
+    if decision["status"] != "SELECTED":
+        state["status"] = decision["status"].lower()
+        state["completed_at"] = action_controller.time.time()
+        action_controller.atomic_json(state_path, state)
+        action_controller.wait_all_idle()
+        print(json.dumps({"status": state["status"], "decision": decision["status"]}))
+        return
+
+    action_order = decision["decisions"]["telemetry"]["intervention_order"]
+    execution_order = [source_id]
+    for action_id in action_order:
+        target_id = str(manifest["actions"][action_id])
+        if target_id not in execution_order:
+            execution_order.append(target_id)
+    for config_id in configs:
+        if config_id not in execution_order:
+            execution_order.append(config_id)
+    state["execution_order"] = execution_order
+    action_controller.atomic_json(state_path, state)
+    for config_id in execution_order[1:]:
+        action_controller.execute_session(
+            args=args,
+            manifest=manifest,
+            config=configs[config_id],
+            index=config_indexes[config_id],
+            state=state,
+            state_path=state_path,
+        )
+    state["status"] = "complete"
+    state["completed_at"] = action_controller.time.time()
+    action_controller.atomic_json(state_path, state)
+    action_controller.wait_all_idle()
+    print(
+        json.dumps(
+            {
+                "status": state["status"],
+                "completed_sessions": state["completed_sessions"],
+                "gpu_hours_total": state["gpu_hours_total"],
+                "execution_order": execution_order,
+            },
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/active-intervention-v0/prospective_decision.py

@@ -0,0 +1,441 @@
+#!/usr/bin/env python3
+"""Choose measurement horizon and next intervention from a completed source run."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import math
+import os
+import statistics
+import sys
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+
+
+HERE = Path(__file__).resolve().parent
+COMMON_STATE = HERE.parent / "telemetry-residual"
+sys.path.insert(0, str(COMMON_STATE))
+
+from common_state import summarize_engine  # noqa: E402
+
+
+SCHEMA = "active-intervention-prospective-decision-v0"
+
+
+def load_module(name: str, path: Path):
+    spec = importlib.util.spec_from_file_location(name, path)
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+MODEL = load_module("active_intervention_prospective_model", HERE / "model.py")
+EXTRACT = load_module(
+    "active_intervention_prospective_extract", HERE / "extract_training.py"
+)
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix(path.suffix + ".tmp")
+    temporary.write_text(
+        json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+    )
+    os.replace(temporary, path)
+
+
+def numeric(values: Sequence[float]) -> dict[str, Any]:
+    finite = [float(value) for value in values]
+    if not finite or any(not math.isfinite(value) for value in finite):
+        raise ValueError("numeric summary requires finite values")
+    return {
+        "n": len(finite),
+        "min": min(finite),
+        "max": max(finite),
+        "distinct_n": len(set(finite)),
+    }
+
+
+def load_engine_records(source_root: Path) -> tuple[list[dict[str, Any]], Path]:
+    streams = sorted((source_root / "opprof").glob("*.jsonl"))
+    if len(streams) != 1:
+        raise ValueError(f"expected one source engine stream, found {len(streams)}")
+    records = [
+        row for row in EXTRACT.load_jsonl(streams[0]) if "step_index" in row
+    ]
+    if not records:
+        raise ValueError("source engine stream has no Layer-1 records")
+    return records, streams[0]
+
+
+def candidate_example(
+    *,
+    source_config: Mapping[str, Any],
+    target_config: Mapping[str, Any],
+    action_id: str,
+    offered_rate_per_gpu: float,
+    outcome: Mapping[str, float],
+    telemetry: Mapping[str, float],
+) -> dict[str, Any]:
+    return {
+        "source": {
+            "mns": int(source_config["mns"]),
+            "mbbt": int(source_config["mbbt"]),
+            "offered_rate_per_gpu": float(offered_rate_per_gpu),
+            "outcome": dict(outcome),
+            "telemetry": dict(telemetry),
+        },
+        "action": {
+            "id": action_id,
+            "target_mns": int(target_config["mns"]),
+            "target_mbbt": int(target_config["mbbt"]),
+        },
+    }
+
+
+def aggregate_checkpoint(
+    *,
+    models: Sequence[Any],
+    examples_by_action: Mapping[str, Sequence[Mapping[str, Any]]],
+    include_telemetry: bool,
+    confidence_z: float,
+    minimum_margin: float,
+) -> dict[str, Any]:
+    rows = []
+    for action_id, examples in sorted(examples_by_action.items()):
+        raw = []
+        for example in examples:
+            source = example["source"]
+            action = example["action"]
+            noop = (
+                int(source["mns"]) == int(action["target_mns"])
+                and int(source["mbbt"]) == int(action["target_mbbt"])
+            )
+            if noop:
+                raw.extend(0.0 for _model in models)
+                continue
+            names, values = MODEL.feature_vector(
+                example, include_telemetry=include_telemetry
+            )
+            if any(model.feature_names != tuple(names) for model in models):
+                raise ValueError("prospective feature schema does not match frozen model")
+            raw.extend(model.predict(values) for model in models)
+        clipped = np.clip(np.asarray(raw, dtype=np.float64), -1.0, 1.0)
+        prediction = {
+            "mean": float(clipped.mean()),
+            "std": float(clipped.std(ddof=0)),
+            "min": float(clipped.min()),
+            "max": float(clipped.max()),
+            "distinct_n": len(set(float(value) for value in clipped)),
+            "sample_n": int(clipped.size),
+        }
+        rows.append(
+            {
+                "action_id": action_id,
+                "prediction": prediction,
+                "lower": prediction["mean"] - confidence_z * prediction["std"],
+                "upper": prediction["mean"] + confidence_z * prediction["std"],
+            }
+        )
+    rows.sort(key=lambda row: (-row["prediction"]["mean"], row["action_id"]))
+    best, second = rows[:2]
+    margin = float(best["prediction"]["mean"] - second["prediction"]["mean"])
+    confident = bool(
+        margin >= minimum_margin and best["lower"] > second["upper"]
+    )
+    return {
+        "selected_action": best["action_id"],
+        "confident": confident,
+        "predicted_margin": margin,
+        "candidates": rows,
+    }
+
+
+def apply_measurement_and_acquisition(checkpoints: list[dict[str, Any]]) -> dict[str, Any]:
+    selected = checkpoints[-1]
+    stop_reason = "full_measurement_fallback"
+    for previous, current in zip(checkpoints, checkpoints[1:], strict=False):
+        if (
+            previous["confident"]
+            and current["confident"]
+            and previous["selected_action"] == current["selected_action"]
+        ):
+            selected = current
+            stop_reason = "two_consecutive_confident_checkpoints"
+            break
+    candidates = selected["candidates"]
+    mean_best = candidates[0]
+    non_noop = [row for row in candidates if row["action_id"] != "noop"]
+    if selected["confident"]:
+        chosen = mean_best
+        decision_kind = "exploit"
+    else:
+        positive_ucb = [row for row in non_noop if float(row["upper"]) > 0.0]
+        if positive_ucb:
+            chosen = max(
+                positive_ucb,
+                key=lambda row: (float(row["upper"]), row["action_id"]),
+            )
+            decision_kind = "diagnostic_ucb"
+        else:
+            chosen = next(row for row in candidates if row["action_id"] == "noop")
+            decision_kind = "abstain_no_positive_ucb"
+    remaining = [row for row in candidates if row["action_id"] != chosen["action_id"]]
+    remaining.sort(key=lambda row: (-float(row["upper"]), row["action_id"]))
+    order = [chosen["action_id"], *(row["action_id"] for row in remaining)]
+    return {
+        "selected_phase": selected["phase"],
+        "selected_cutoff_s": selected["cutoff_s"],
+        "measurement_stop_reason": stop_reason,
+        "decision_kind": decision_kind,
+        "selected_action": chosen["action_id"],
+        "intervention_order": order,
+        "selected_checkpoint": selected,
+        "checkpoints": checkpoints,
+    }
+
+
+def build_decision(
+    *, manifest_path: Path, policy_path: Path, run_root: Path
+) -> dict[str, Any]:
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    policy = json.loads(policy_path.read_text(encoding="utf-8"))
+    if manifest.get("schema") != "active-intervention-prospective-manifest-v0":
+        raise ValueError("unexpected prospective manifest schema")
+    if policy.get("schema") != "active-intervention-policy-v0":
+        raise ValueError("unexpected frozen policy schema")
+    if sha256_file(policy_path) != manifest["policy"]["sha256"]:
+        raise ValueError("frozen policy hash changed after manifest preparation")
+    configs = {str(item["id"]): item for item in manifest["configs"]}
+    source_id = str(manifest["source_config_id"])
+    source_config = configs[source_id]
+    source_root = run_root / "sessions" / source_id
+    engine_records, stream_path = load_engine_records(source_root)
+    phases = [f"{fraction:.2f}" for fraction in manifest["checkpoints"]["fractions"]]
+    confidence_z = float(policy["measurement_policy"]["confidence_z"])
+    minimum_margin = float(policy["measurement_policy"]["minimum_margin"])
+
+    examples: dict[str, dict[str, dict[str, Mapping[str, Any]]]] = {}
+    source_measurements: dict[str, dict[str, Any]] = {}
+    source_normalized = []
+    telemetry_values = []
+    for repetition in sorted(int(key) for key in manifest["repetitions"]):
+        item = manifest["repetitions"][str(repetition)]
+        result_root = source_root / f"rep{repetition}"
+        result = json.loads((result_root / "result.json").read_text(encoding="utf-8"))
+        if result["selection"]["request_id_order_sha256"] != item["selection"][
+            "request_id_order_sha256"
+        ]:
+            raise ValueError(f"source request hash mismatch: rep{repetition}")
+        requests = EXTRACT.load_jsonl(result_root / "requests.jsonl")
+        offered_rate = float(item["selection"]["offered_req_s_per_gpu"])
+        offered_total = offered_rate * int(manifest["engine"]["tp"])
+        source_normalized.append(
+            float(result["slo_pass_count"])
+            / float(manifest["engine"]["duration_s"])
+            / offered_total
+        )
+        start_ns = int(result["interval"]["start_mono_ns"])
+        examples[str(repetition)] = {}
+        source_measurements[str(repetition)] = {
+            "result": str(result_root / "result.json"),
+            "result_sha256": sha256_file(result_root / "result.json"),
+            "request_sha256": sha256_file(result_root / "requests.jsonl"),
+            "phases": {},
+        }
+        for phase, cutoff_s in zip(
+            phases, manifest["checkpoints"]["seconds"], strict=True
+        ):
+            outcome = EXTRACT.prefix_outcome(
+                requests, cutoff_s=float(cutoff_s), offered_total=offered_total
+            )
+            admitted_count = sum(
+                float(request["arrival_s"]) <= float(cutoff_s)
+                for request in requests
+            )
+            state = summarize_engine(
+                engine_records,
+                start_ns=start_ns,
+                end_ns=start_ns + round(float(cutoff_s) * 1e9),
+                request_count=admitted_count,
+            )
+            if not all(state["sanity"]["invariants"].values()):
+                raise ValueError(
+                    f"source engine state invariant failed: rep{repetition} {phase}"
+                )
+            telemetry = EXTRACT.telemetry_record(state)
+            telemetry_values.extend(float(value) for value in telemetry.values())
+            source_measurements[str(repetition)]["phases"][phase] = {
+                "cutoff_s": float(cutoff_s),
+                "outcome": outcome,
+                "telemetry": telemetry,
+                "engine_sanity": state["sanity"],
+            }
+            examples[str(repetition)][phase] = {
+                action_id: candidate_example(
+                    source_config=source_config,
+                    target_config=configs[str(target_id)],
+                    action_id=action_id,
+                    offered_rate_per_gpu=offered_rate,
+                    outcome=outcome,
+                    telemetry=telemetry,
+                )
+                for action_id, target_id in manifest["actions"].items()
+            }
+
+    decisions = {}
+    for mode, include_telemetry in (("outcome_only", False), ("telemetry", True)):
+        checkpoints = []
+        for phase, cutoff_s in zip(
+            phases, manifest["checkpoints"]["seconds"], strict=True
+        ):
+            models = MODEL.models_from_json(policy["phases"][phase][mode]["models"])
+            examples_by_action = {
+                action_id: [
+                    examples[str(repetition)][phase][action_id]
+                    for repetition in sorted(int(key) for key in manifest["repetitions"])
+                ]
+                for action_id in manifest["actions"]
+            }
+            checkpoint = aggregate_checkpoint(
+                models=models,
+                examples_by_action=examples_by_action,
+                include_telemetry=include_telemetry,
+                confidence_z=confidence_z,
+                minimum_margin=minimum_margin,
+            )
+            checkpoints.append(
+                {"phase": phase, "cutoff_s": float(cutoff_s), **checkpoint}
+            )
+        decisions[mode] = apply_measurement_and_acquisition(checkpoints)
+
+    ceiling = float(manifest["gates"]["source_ceiling_normalized_goodput"])
+    source_median = float(statistics.median(source_normalized))
+    status = "STOP_SOURCE_CEILING" if source_median >= ceiling else "SELECTED"
+    phase_admission_monotonic = all(
+        all(
+            left <= right + 1e-12
+            for left, right in zip(values, values[1:], strict=False)
+        )
+        for repetition in source_measurements.values()
+        for values in (
+            [
+                float(repetition["phases"][phase]["outcome"]["admitted_fraction"])
+                for phase in phases
+            ],
+        )
+    )
+    telemetry_ratio_keys = {
+        "prefill_token_fraction",
+        "kv_usage_mean",
+        "kv_usage_max",
+        "graph_none_share",
+        "graph_full_share",
+        "graph_padding_fraction",
+    }
+    telemetry_records = [
+        measurement["telemetry"]
+        for repetition in source_measurements.values()
+        for measurement in repetition["phases"].values()
+    ]
+    invariants = {
+        "three_source_repetitions": len(source_normalized) == 3,
+        "source_goodput_nonnegative": all(value >= 0.0 for value in source_normalized),
+        "source_goodput_bounded": all(
+            value <= 1.0 + 1e-12 for value in source_normalized
+        ),
+        "four_actions": set(manifest["actions"]) == {"noop", "mns", "mbbt", "joint"},
+        "four_checkpoints": len(phases) == 4,
+        "finite_telemetry": all(math.isfinite(value) for value in telemetry_values),
+        "nonnegative_telemetry": all(
+            float(value) >= 0.0
+            for record in telemetry_records
+            for key, value in record.items()
+            if key != "kv_usage_end_minus_start"
+        ),
+        "telemetry_ratios_bounded": all(
+            0.0 <= float(record[key]) <= 1.0 + 1e-12
+            for record in telemetry_records
+            for key in telemetry_ratio_keys
+        ),
+        "telemetry_not_all_identical": len(set(telemetry_values)) > 1,
+        "phase_admission_monotonic": phase_admission_monotonic,
+        "orders_are_permutations": all(
+            set(decisions[mode]["intervention_order"]) == set(manifest["actions"])
+            for mode in decisions
+        ),
+    }
+    red_flags = [name for name, passed in invariants.items() if not passed]
+    if red_flags:
+        status = "STOP_SANITY"
+    return {
+        "schema": SCHEMA,
+        "status": status,
+        "manifest": str(manifest_path),
+        "manifest_sha256": sha256_file(manifest_path),
+        "policy": str(policy_path),
+        "policy_sha256": sha256_file(policy_path),
+        "source_stream": str(stream_path),
+        "source_stream_sha256": sha256_file(stream_path),
+        "source_measurements": source_measurements,
+        "source_normalized_goodput": {
+            "values": source_normalized,
+            "median": source_median,
+            **numeric(source_normalized),
+        },
+        "decisions": decisions,
+        "sanity": {
+            "invariants": invariants,
+            "red_flags": red_flags,
+            "telemetry_values": numeric(telemetry_values),
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--policy", type=Path, required=True)
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    decision = build_decision(
+        manifest_path=args.manifest, policy_path=args.policy, run_root=args.run_root
+    )
+    atomic_json(args.output, decision)
+    print(
+        json.dumps(
+            {
+                "status": decision["status"],
+                "source_normalized_goodput": decision["source_normalized_goodput"],
+                "outcome_only": {
+                    key: decision["decisions"]["outcome_only"][key]
+                    for key in ("selected_cutoff_s", "decision_kind", "selected_action")
+                },
+                "telemetry": {
+                    key: decision["decisions"]["telemetry"][key]
+                    for key in ("selected_cutoff_s", "decision_kind", "selected_action")
+                },
+            },
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/active-intervention-v0/test_model.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def load_model():
+    spec = importlib.util.spec_from_file_location(
+        "active_intervention_model", HERE / "model.py"
+    )
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def example(model, decision: str, action: str, pressure: float, target: float):
+    outcome = {
+        name: 0.5 for name in model.PREFIX_FEATURES
+    }
+    telemetry = {name: 0.0 for name in model.TELEMETRY_FEATURES}
+    telemetry["queue_waiting_mean"] = pressure
+    telemetry["batch_size_mean"] = pressure
+    return {
+        "decision_id": decision,
+        "source": {
+            "mns": 16,
+            "mbbt": 8192,
+            "offered_rate_per_gpu": 2.0,
+            "outcome": outcome,
+            "telemetry": telemetry,
+        },
+        "action": {
+            "id": action,
+            "target_mns": 64 if action == "mns" else 16,
+            "target_mbbt": 8192 if action == "mns" else 16384,
+        },
+        "target_normalized_goodput": target,
+        "target_delta_normalized_goodput": target - 0.5,
+    }
+
+
+def main() -> None:
+    model = load_model()
+    examples = []
+    for index, pressure in enumerate((0.2, 0.5, 0.8), 1):
+        examples.extend(
+            (
+                example(model, f"d{index}", "mns", pressure, 0.5 + pressure / 2),
+                example(model, f"d{index}", "mbbt", pressure, 0.6 - pressure / 4),
+            )
+        )
+    fitted = model.fit_ridge(
+        examples, include_telemetry=True, regularization=1.0
+    )
+    encoded = fitted.to_json()
+    restored = model.RidgeModel.from_json(encoded)
+    names, values = model.feature_vector(examples[-2], include_telemetry=True)
+    assert tuple(names) == restored.feature_names
+    assert abs(fitted.predict(values) - restored.predict(values)) < 1e-12
+    ensemble = model.fit_jackknife_ensemble(
+        examples, include_telemetry=True, regularization=1.0
+    )
+    decision = model.select_action(
+        ensemble, examples[-2:], include_telemetry=True, minimum_margin=0.0
+    )
+    assert decision["selected_action"] == "mns"
+    assert all(-1.0 <= row["prediction"]["mean"] <= 1.0 for row in decision["candidates"])
+    noop = example(model, "noop", "noop", 0.8, 0.5)
+    noop["action"]["target_mbbt"] = 8192
+    prediction = model.ensemble_predict(
+        ensemble, noop, include_telemetry=True
+    )
+    assert prediction == {
+        "mean": 0.0,
+        "std": 0.0,
+        "min": 0.0,
+        "max": 0.0,
+        "distinct_n": 1,
+    }
+    print("active intervention model: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/active-intervention-v0/test_pipeline.py

@@ -0,0 +1,199 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import importlib.util
+import json
+import sys
+import tempfile
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def load(name: str, path: Path):
+    spec = importlib.util.spec_from_file_location(name, path)
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def write_json(path: Path, payload) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(json.dumps(payload) + "\n", encoding="utf-8")
+
+
+def write_jsonl(path: Path, rows) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(
+        "".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8"
+    )
+
+
+def engine_record(index: int, timestamp_ns: int) -> dict:
+    alternate = index % 2
+    return {
+        "step_index": index,
+        "submit_mono_ns": timestamp_ns,
+        "model_executed": True,
+        "scheduled_requests": 1 + alternate,
+        "decode_batch_size": alternate,
+        "prefill_tokens": 8 + alternate,
+        "decode_tokens": alternate,
+        "preemptions": 0,
+        "queues": {"waiting": alternate, "running": 1 + alternate},
+        "kv": {"usage": 0.1 + 0.01 * alternate},
+        "cudagraph": {
+            "runtime_mode": "FULL" if alternate else "NONE",
+            "bucket_tokens": 16,
+            "padding_tokens": alternate,
+        },
+        "dropped_records_before": 0,
+    }
+
+
+def main() -> None:
+    extractor = load("active_intervention_extract_test", HERE / "extract_training.py")
+    trainer = load("active_intervention_train_test", HERE / "train_policy.py")
+    with tempfile.TemporaryDirectory() as temporary:
+        root = Path(temporary)
+        run_root = root / "runs"
+        configs = [
+            {"id": "a_base", "mns": 16, "mbbt": 8192},
+            {"id": "a_mns", "mns": 64, "mbbt": 8192},
+            {"id": "a_mbbt", "mns": 16, "mbbt": 16384},
+            {"id": "b_base", "mns": 64, "mbbt": 2048},
+            {"id": "b_mns", "mns": 128, "mbbt": 2048},
+            {"id": "b_mbbt", "mns": 64, "mbbt": 8192},
+        ]
+        manifest = {
+            "engine": {"duration_s": 300.0, "tp": 4},
+            "configs": configs,
+            "repetitions": {
+                str(rep): {"selection": {"offered_req_s_per_gpu": 0.01}}
+                for rep in (1, 2, 3)
+            },
+            "regimes": {
+                "A": {
+                    "source": "a_base",
+                    "actions": {"mns": "a_mns", "mbbt": "a_mbbt"},
+                },
+                "B": {
+                    "source": "b_base",
+                    "actions": {"mns": "b_mns", "mbbt": "b_mbbt"},
+                },
+            },
+        }
+        manifest_path = root / "manifest.json"
+        write_json(manifest_path, manifest)
+
+        streams = []
+        source_starts: dict[tuple[str, int], int] = {}
+        for source_index, source_id in enumerate(("a_base", "b_base")):
+            rows = []
+            index = 0
+            for repetition in (1, 2, 3):
+                start_ns = int((source_index * 2000 + repetition * 400) * 1e9)
+                source_starts[(source_id, repetition)] = start_ns
+                for second in (1, 30, 76, 105, 151, 180, 226, 255):
+                    rows.append(engine_record(index, start_ns + int(second * 1e9)))
+                    index += 1
+            stream_path = root / f"{source_id}-stream.jsonl"
+            write_jsonl(stream_path, rows)
+            streams.append(
+                {
+                    "config_id": source_id,
+                    "stream": str(stream_path),
+                    "stream_sha256": extractor.sha256_file(stream_path),
+                }
+            )
+
+        request_rows = [
+            {
+                "request_id": f"r{index}",
+                "arrival_s": arrival,
+                "completed_elapsed_s": arrival + 10,
+                "slo_pass": index != 3,
+                "ttft_ms": 1000 + index * 100,
+                "tpot_ms": 20 + index,
+                "raw_input_tokens": 1000 + index * 100,
+            }
+            for index, arrival in enumerate((5.0, 80.0, 155.0, 230.0), 1)
+        ]
+        for source_id in ("a_base", "b_base"):
+            for repetition in (1, 2, 3):
+                write_jsonl(
+                    run_root
+                    / "sessions"
+                    / source_id
+                    / f"rep{repetition}"
+                    / "requests.jsonl",
+                    request_rows,
+                )
+
+        goodput = {
+            "a_base": 0.020,
+            "a_mns": 0.036,
+            "a_mbbt": 0.028,
+            "b_base": 0.032,
+            "b_mns": 0.030,
+            "b_mbbt": 0.038,
+        }
+        runs = []
+        for config in configs:
+            for repetition in (1, 2, 3):
+                item = {
+                    "config_id": config["id"],
+                    "repetition": repetition,
+                    "outcome": {
+                        "slo_goodput_req_s": goodput[config["id"]]
+                        + repetition * 0.0001
+                    },
+                }
+                if config["id"] in ("a_base", "b_base"):
+                    start_ns = source_starts[(config["id"], repetition)]
+                    item["state"] = {
+                        "interval": {
+                            "start_ns": start_ns,
+                            "end_ns": start_ns + int(300 * 1e9),
+                        }
+                    }
+                runs.append(item)
+        audit = {
+            "schema": "action-aware-constraint-pilot-audit-v0",
+            "sanity": {"red_flags": []},
+            "streams": streams,
+            "runs": runs,
+        }
+        audit_path = root / "audit.json"
+        write_json(audit_path, audit)
+
+        dataset = extractor.build_dataset(
+            audit_path=audit_path, manifest_path=manifest_path, run_root=run_root
+        )
+        assert dataset["status"] == "VALID"
+        assert len(dataset["examples"]) == 72
+        assert not dataset["sanity"]["red_flags"]
+        assert all(
+            "exclusive" not in feature
+            for example in dataset["examples"]
+            for feature in example["source"]["telemetry"]
+        )
+        dataset_path = root / "dataset.json"
+        write_json(dataset_path, dataset)
+        policy = trainer.build_policy(dataset_path)
+        assert policy["status"] in {
+            "RETROSPECTIVE_GPU_COST_SIGNAL",
+            "NO_RETROSPECTIVE_GPU_COST_SIGNAL",
+        }
+        assert policy["training"]["acceptable_regret"] == 0.02
+        assert policy["sequential_replay"]["outcome_only"]["decision_n"] == 6
+        assert policy["sequential_replay"]["telemetry"]["decision_n"] == 6
+        assert not policy["sanity"]["red_flags"]
+    print("active intervention pipeline: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/active-intervention-v0/test_prospective.py

@@ -0,0 +1,190 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import importlib.util
+import json
+import sys
+import tempfile
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def load(name: str, path: Path):
+    spec = importlib.util.spec_from_file_location(name, path)
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def write_json(path: Path, payload) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(json.dumps(payload) + "\n", encoding="utf-8")
+
+
+def main() -> None:
+    prepare = load("active_intervention_prepare_test", HERE / "prepare_prospective.py")
+    decision_module = load(
+        "active_intervention_decision_test", HERE / "prospective_decision.py"
+    )
+    analyzer = load("active_intervention_audit_test", HERE / "analyze_prospective.py")
+    with tempfile.TemporaryDirectory() as temporary:
+        root = Path(temporary)
+        source = root / "source.jsonl"
+        source.write_text(
+            "".join(
+                json.dumps(
+                    {
+                        "request_id": f"request-{index}",
+                        "timestamp": float(index),
+                        "sampling_u": index / 100.0,
+                    }
+                )
+                + "\n"
+                for index in range(60)
+            ),
+            encoding="utf-8",
+        )
+        partition = prepare.partition_trace(source, root / "partitions")
+        assert sum(item["rows"] for item in partition["partitions"].values()) == 60
+        ids = []
+        for item in partition["partitions"].values():
+            assert item["rows"] > 0
+            ids.extend(
+                json.loads(line)["request_id"]
+                for line in Path(item["path"]).read_text(encoding="utf-8").splitlines()
+            )
+        assert len(ids) == len(set(ids)) == 60
+
+        checkpoints = [
+            {
+                "phase": "0.25",
+                "cutoff_s": 75.0,
+                "selected_action": "joint",
+                "confident": True,
+                "candidates": [
+                    {"action_id": "joint", "upper": 0.5, "prediction": {"mean": 0.4}},
+                    {"action_id": "mns", "upper": 0.2, "prediction": {"mean": 0.1}},
+                    {"action_id": "mbbt", "upper": 0.1, "prediction": {"mean": 0.05}},
+                    {"action_id": "noop", "upper": 0.0, "prediction": {"mean": 0.0}},
+                ],
+            },
+            {
+                "phase": "0.50",
+                "cutoff_s": 150.0,
+                "selected_action": "joint",
+                "confident": True,
+                "candidates": [
+                    {"action_id": "joint", "upper": 0.45, "prediction": {"mean": 0.4}},
+                    {"action_id": "mns", "upper": 0.2, "prediction": {"mean": 0.1}},
+                    {"action_id": "mbbt", "upper": 0.1, "prediction": {"mean": 0.05}},
+                    {"action_id": "noop", "upper": 0.0, "prediction": {"mean": 0.0}},
+                ],
+            },
+        ]
+        selected = decision_module.apply_measurement_and_acquisition(checkpoints)
+        assert selected["selected_cutoff_s"] == 150.0
+        assert selected["selected_action"] == "joint"
+
+        configs = prepare.configs()
+        repetitions = {
+            str(rep): {
+                "selection": {
+                    "offered_req_s_per_gpu": 0.25,
+                    "request_id_order_sha256": f"hash-{rep}",
+                }
+            }
+            for rep in (1, 2, 3)
+        }
+        manifest = {
+            "schema": "active-intervention-prospective-manifest-v0",
+            "engine": {"duration_s": 300.0, "tp": 4},
+            "repetitions": repetitions,
+            "configs": configs,
+            "source_config_id": "source_mns32_mbbt4096",
+            "actions": {
+                "noop": "source_mns32_mbbt4096",
+                "mns": "mns64_mbbt4096",
+                "mbbt": "mns32_mbbt8192",
+                "joint": "joint_mns64_mbbt8192",
+            },
+            "gates": {
+                "acceptable_regret": 0.02,
+                "confirmation_trigger_gpu_cost_reduction": 0.10,
+                "contribution_gpu_cost_reduction": 0.20,
+            },
+        }
+        manifest_path = root / "manifest.json"
+        write_json(manifest_path, manifest)
+        run_root = root / "run"
+        scores = {
+            "source_mns32_mbbt4096": 0.5,
+            "mns64_mbbt4096": 0.8,
+            "mns32_mbbt8192": 0.7,
+            "joint_mns64_mbbt8192": 1.0,
+        }
+        sessions = {}
+        for config in configs:
+            config_id = config["id"]
+            sessions[config_id] = {"status": "complete", "gpu_hours": 1.2}
+            for repetition in (1, 2, 3):
+                result = {
+                    "selection": {
+                        "request_id_order_sha256": f"hash-{repetition}"
+                    },
+                    "slo_pass_count": round(scores[config_id] * 300),
+                    "pass_rate": scores[config_id],
+                    "interval": {"elapsed_s": 300.0},
+                }
+                write_json(
+                    run_root
+                    / "sessions"
+                    / config_id
+                    / f"rep{repetition}"
+                    / "result.json",
+                    result,
+                )
+        state = {
+            "status": "complete",
+            "gpu_hours_total": 4.8,
+            "sessions": sessions,
+        }
+        write_json(run_root / "controller-state.json", state)
+        mode_base = {
+            "selected_cutoff_s": 300.0,
+            "selected_action": "mns",
+            "decision_kind": "exploit",
+            "intervention_order": ["mns", "mbbt", "joint", "noop"],
+        }
+        mode_telemetry = {
+            "selected_cutoff_s": 150.0,
+            "selected_action": "joint",
+            "decision_kind": "exploit",
+            "intervention_order": ["joint", "mns", "mbbt", "noop"],
+        }
+        decision = {
+            "schema": "active-intervention-prospective-decision-v0",
+            "manifest_sha256": analyzer.sha256_file(manifest_path),
+            "decisions": {
+                "outcome_only": mode_base,
+                "telemetry": mode_telemetry,
+            },
+        }
+        decision_path = root / "decision.json"
+        write_json(decision_path, decision)
+        audit = analyzer.build_audit(
+            manifest_path=manifest_path,
+            decision_path=decision_path,
+            run_root=run_root,
+        )
+        assert audit["status"] == "TRIGGER_ACTUAL_EARLY_STOP_CONFIRMATION"
+        assert audit["comparison"]["telemetry_gpu_cost_reduction_fraction"] > 0.10
+        assert not audit["sanity"]["red_flags"]
+    print("active intervention prospective pipeline: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/active-intervention-v0/trace13-results.json

@@ -0,0 +1,151 @@
+{
+  "schema": "active-intervention-trace13-result-summary-v0",
+  "status": "STOP_NO_PROSPECTIVE_GPU_COST_SIGNAL",
+  "provenance": {
+    "aituner_commit": "39b767e384fc49da53b99ad06a3e2ca1b6ac37d6",
+    "vllm_commit": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05",
+    "manifest_sha256": "3bd25ae0ca040729a6351635f14447b3c789d4d86fb0fe4d65940735ad225a78",
+    "policy_sha256": "4f096d3a5f8c38771e956dfd576dd6cd5d5691286ab258028dbddbe07b20078c",
+    "decision_sha256": "86b7089151c480e18b5ae6ed65c4e4a3e11159dd0c16d5851312d4d9196d5ca2",
+    "controller_state_sha256": "d10d3cbe5ce20eacc1392f52e79d16ae23b5e15301123d5e85e48d53ae676cda",
+    "audit_sha256": "bd95b45e5d4cb93b5ad2f722b7dc96553d64eb8a5d5aae3a82f29c2d015fe3f6",
+    "remote_root": "/home/admin/cpfs/wjh/active-intervention-prospective-20260715"
+  },
+  "cost": {
+    "annotation_campaign_h20_hours": 5.0379046784506905,
+    "hard_cap_h20_hours": 6.0,
+    "outcome_only_cost_to_acceptable_h20_hours_lower_bound": 2.4284364508172893,
+    "telemetry_cost_to_acceptable_h20_hours_lower_bound": 2.4284364508172893,
+    "telemetry_gpu_cost_reduction_fraction": 0.0,
+    "session_h20_hours": {
+      "source_mns32_mbbt4096": 1.3566088432735868,
+      "mns64_mbbt4096": 1.256969277858734,
+      "mns32_mbbt8192": 1.254111782974667,
+      "joint_mns64_mbbt8192": 1.1702147743437026
+    }
+  },
+  "policy_comparison": {
+    "outcome_only": {
+      "measurement_cutoff_s": 300.0,
+      "selected_action": "joint",
+      "intervention_order": ["joint", "mns", "mbbt", "noop"]
+    },
+    "telemetry": {
+      "measurement_cutoff_s": 300.0,
+      "selected_action": "joint",
+      "intervention_order": ["joint", "mns", "mbbt", "noop"]
+    },
+    "action_changed": false,
+    "measurement_changed": false,
+    "confirmation_trigger": false,
+    "contribution_gate": false
+  },
+  "surface": {
+    "source_mns32_mbbt4096": {
+      "normalized_slo_goodput": [
+        0.40090909090909094,
+        0.3978787878787879,
+        0.42060606060606065
+      ],
+      "median": 0.40090909090909094
+    },
+    "mns64_mbbt4096": {
+      "normalized_slo_goodput": [1.0, 0.9996969696969698, 1.0],
+      "median": 1.0
+    },
+    "mns32_mbbt8192": {
+      "normalized_slo_goodput": [
+        0.44393939393939397,
+        0.41515151515151516,
+        0.4260606060606061
+      ],
+      "median": 0.42606060606060603
+    },
+    "joint_mns64_mbbt8192": {
+      "normalized_slo_goodput": [1.0, 1.0, 1.0],
+      "median": 1.0
+    }
+  },
+  "selected_checkpoint_prediction": {
+    "actual_median_effect": {
+      "noop": 0.0,
+      "mns": 0.5990909090909091,
+      "mbbt": 0.02515151515151509,
+      "joint": 0.5990909090909091
+    },
+    "outcome_only_predicted_effect": {
+      "noop": 0.0,
+      "mns": 0.2886250281729182,
+      "mbbt": 0.17933598309437812,
+      "joint": 0.3205015384324615
+    },
+    "telemetry_predicted_effect": {
+      "noop": 0.0,
+      "mns": 0.26117798146236215,
+      "mbbt": 0.09686132563074483,
+      "joint": 0.35190199346536294
+    },
+    "actual_joint_minus_mns": 0.0,
+    "outcome_only_joint_minus_mns": 0.0318765102595433,
+    "telemetry_joint_minus_mns": 0.09072401200300079
+  },
+  "engine_mechanism": {
+    "source_mns32_mbbt4096": {
+      "scheduler_records": 41086,
+      "waiting_fraction": 0.9312174463320839,
+      "mns_exclusive_fraction": 0.8536484447256973,
+      "mbbt_exclusive_fraction": 0.01114734946210388,
+      "both_fraction": 0.06642165214428272,
+      "running_utilization_mean": 0.9738878510928297,
+      "token_utilization_mean": 0.15694342925509905,
+      "kv_usage_mean": 0.027507593814715858,
+      "preemptions": 0
+    },
+    "mns64_mbbt4096": {
+      "scheduler_records": 37001,
+      "waiting_fraction": 0.053809356503878275,
+      "mns_exclusive_fraction": 0.0,
+      "mbbt_exclusive_fraction": 0.053809356503878275,
+      "running_utilization_mean": 0.5410364753655307,
+      "token_utilization_mean": 0.17425695631536986,
+      "kv_usage_mean": 0.030549112146415616,
+      "preemptions": 0
+    },
+    "mns32_mbbt8192": {
+      "scheduler_records": 41348,
+      "waiting_fraction": 0.9119425365192996,
+      "mns_exclusive_fraction": 0.9108542130211861,
+      "mbbt_exclusive_fraction": 0.0003627744993711909,
+      "running_utilization_mean": 0.9652567838831383,
+      "token_utilization_mean": 0.0779606414231039,
+      "kv_usage_mean": 0.027355900366122385,
+      "preemptions": 0
+    },
+    "joint_mns64_mbbt8192": {
+      "scheduler_records": 40416,
+      "waiting_fraction": 0.0088826207442597,
+      "mns_exclusive_fraction": 0.0,
+      "mbbt_exclusive_fraction": 0.0088826207442597,
+      "running_utilization_mean": 0.49403392220902614,
+      "token_utilization_mean": 0.07978070546782215,
+      "kv_usage_mean": 0.028003770774946098,
+      "preemptions": 0
+    }
+  },
+  "sanity": {
+    "surface_outcomes": {"n": 12, "min": 0.3978787878787879, "max": 1.0, "distinct_n": 8},
+    "session_h20_hours": {"n": 4, "min": 1.1702147743437026, "max": 1.3566088432735868, "distinct_n": 4},
+    "scheduler_records": {"n": 4, "min": 37001, "max": 41348, "distinct_n": 4},
+    "invariants": {
+      "controller_complete": true,
+      "four_sessions_complete": true,
+      "twelve_surface_outcomes": true,
+      "ratios_bounded": true,
+      "nonnegative_counts_and_costs": true,
+      "surface_not_all_identical": true,
+      "request_hashes_match": true,
+      "no_censored_runs": true
+    },
+    "red_flags": []
+  }
+}

runs/active-intervention-v0/train_policy.py

@@ -0,0 +1,519 @@
+#!/usr/bin/env python3
+"""Train and audit outcome-only versus telemetry action-response policies."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import math
+import os
+import sys
+from collections import defaultdict
+from pathlib import Path
+from typing import Any, Mapping, Sequence
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def _load_model():
+    spec = importlib.util.spec_from_file_location(
+        "active_intervention_model", HERE / "model.py"
+    )
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+MODEL = _load_model()
+REGULARIZATION = 10.0
+MINIMUM_MARGIN = 0.02
+CONFIDENCE_Z = 1.0
+ACCEPTABLE_REGRET = 0.02
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix(path.suffix + ".tmp")
+    temporary.write_text(
+        json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+    )
+    os.replace(temporary, path)
+
+
+def grouped(
+    examples: Sequence[Mapping[str, Any]], key: str
+) -> dict[str, list[Mapping[str, Any]]]:
+    result: dict[str, list[Mapping[str, Any]]] = defaultdict(list)
+    for example in examples:
+        result[str(example[key])].append(example)
+    return dict(result)
+
+
+def evaluate_grouped_cv(
+    examples: Sequence[Mapping[str, Any]],
+    *,
+    include_telemetry: bool,
+    holdout_key: str,
+) -> dict[str, Any]:
+    holdouts = grouped(examples, holdout_key)
+    decision_rows = []
+    for held_out, test_examples in sorted(holdouts.items()):
+        training = [example for example in examples if str(example[holdout_key]) != held_out]
+        if len({str(example["decision_id"]) for example in training}) < 2:
+            continue
+        model = MODEL.fit_ridge(
+            training,
+            include_telemetry=include_telemetry,
+            regularization=REGULARIZATION,
+        )
+        for decision_id, candidates in sorted(grouped(test_examples, "decision_id").items()):
+            predictions = []
+            for candidate in candidates:
+                source = candidate["source"]
+                action = candidate["action"]
+                noop = (
+                    int(action["target_mns"]) == int(source["mns"])
+                    and int(action["target_mbbt"]) == int(source["mbbt"])
+                )
+                if noop:
+                    prediction = 0.0
+                else:
+                    names, vector = MODEL.feature_vector(
+                        candidate, include_telemetry=include_telemetry
+                    )
+                    if tuple(names) != model.feature_names:
+                        raise ValueError("cross-validation feature schema mismatch")
+                    prediction = max(-1.0, min(1.0, model.predict(vector)))
+                predictions.append(
+                    {
+                        "action_id": str(candidate["action"]["id"]),
+                        "prediction": prediction,
+                        "real": float(candidate["target_normalized_goodput"]),
+                    }
+                )
+            predictions.sort(key=lambda row: (-row["prediction"], row["action_id"]))
+            selected = predictions[0]
+            oracle = max(row["real"] for row in predictions)
+            regret = 1.0 - selected["real"] / oracle if oracle > 0 else 0.0
+            best_actions = {
+                row["action_id"] for row in predictions if math.isclose(row["real"], oracle)
+            }
+            acceptable_actions = {
+                row["action_id"]
+                for row in predictions
+                if oracle <= 0
+                or 1.0 - float(row["real"]) / oracle <= ACCEPTABLE_REGRET + 1e-12
+            }
+            decision_rows.append(
+                {
+                    "holdout": held_out,
+                    "decision_id": decision_id,
+                    "selected_action": selected["action_id"],
+                    "best_actions": sorted(best_actions),
+                    "acceptable_actions": sorted(acceptable_actions),
+                    "correct": regret <= ACCEPTABLE_REGRET + 1e-12,
+                    "selected_real": selected["real"],
+                    "oracle_real": oracle,
+                    "regret": regret,
+                    "predictions": predictions,
+                }
+            )
+    if not decision_rows:
+        return {"status": "INSUFFICIENT_GROUPS", "decisions": []}
+    regrets = [float(row["regret"]) for row in decision_rows]
+    return {
+        "status": "VALID",
+        "holdout_key": holdout_key,
+        "acceptable_regret": ACCEPTABLE_REGRET,
+        "decision_n": len(decision_rows),
+        "correct_n": sum(bool(row["correct"]) for row in decision_rows),
+        "accuracy": sum(bool(row["correct"]) for row in decision_rows) / len(decision_rows),
+        "mean_regret": sum(regrets) / len(regrets),
+        "max_regret": max(regrets),
+        "decisions": decision_rows,
+    }
+
+
+def paired_delta(outcome: Mapping[str, Any], telemetry: Mapping[str, Any]) -> dict[str, Any]:
+    if outcome.get("status") != "VALID" or telemetry.get("status") != "VALID":
+        return {"status": "INSUFFICIENT_GROUPS"}
+    outcome_by_id = {row["decision_id"]: row for row in outcome["decisions"]}
+    telemetry_by_id = {row["decision_id"]: row for row in telemetry["decisions"]}
+    common = sorted(set(outcome_by_id) & set(telemetry_by_id))
+    rows = []
+    for decision_id in common:
+        before = outcome_by_id[decision_id]
+        after = telemetry_by_id[decision_id]
+        rows.append(
+            {
+                "decision_id": decision_id,
+                "outcome_action": before["selected_action"],
+                "telemetry_action": after["selected_action"],
+                "action_changed": before["selected_action"] != after["selected_action"],
+                "regret_delta": float(after["regret"]) - float(before["regret"]),
+                "telemetry_corrected": (not before["correct"]) and bool(after["correct"]),
+                "telemetry_harmed": bool(before["correct"]) and (not after["correct"]),
+            }
+        )
+    return {
+        "status": "VALID",
+        "decision_n": len(rows),
+        "action_changed_n": sum(row["action_changed"] for row in rows),
+        "corrected_n": sum(row["telemetry_corrected"] for row in rows),
+        "harmed_n": sum(row["telemetry_harmed"] for row in rows),
+        "mean_regret_delta": (
+            sum(float(row["regret_delta"]) for row in rows) / len(rows) if rows else 0.0
+        ),
+        "rows": rows,
+    }
+
+
+def evaluate_sequential_measurement_cv(
+    examples: Sequence[Mapping[str, Any]],
+    *,
+    include_telemetry: bool,
+    holdout_key: str,
+) -> dict[str, Any]:
+    """Replay a two-consecutive-confident-checkpoint measurement policy."""
+
+    phases = sorted({str(example["phase"]) for example in examples}, key=float)
+    holdouts = grouped(examples, holdout_key)
+    rows = []
+    full_duration_s = max(float(example["cutoff_s"]) for example in examples)
+    for held_out, test_examples in sorted(holdouts.items()):
+        training = [
+            example for example in examples if str(example[holdout_key]) != held_out
+        ]
+        if len({str(example["decision_id"]) for example in training}) < 3:
+            continue
+        phase_models = {}
+        for phase in phases:
+            phase_training = [
+                example for example in training if str(example["phase"]) == phase
+            ]
+            phase_models[phase] = MODEL.fit_jackknife_ensemble(
+                phase_training,
+                include_telemetry=include_telemetry,
+                regularization=REGULARIZATION,
+            )
+        for decision_id, decision_examples in sorted(
+            grouped(test_examples, "decision_id").items()
+        ):
+            checkpoints = []
+            by_phase = grouped(decision_examples, "phase")
+            for phase in phases:
+                candidates = by_phase[phase]
+                decision = MODEL.select_action(
+                    phase_models[phase],
+                    candidates,
+                    include_telemetry=include_telemetry,
+                    confidence_z=CONFIDENCE_Z,
+                    minimum_margin=MINIMUM_MARGIN,
+                )
+                checkpoints.append(
+                    {
+                        "phase": phase,
+                        "cutoff_s": float(candidates[0]["cutoff_s"]),
+                        **decision,
+                    }
+                )
+            selected_checkpoint = checkpoints[-1]
+            stop_reason = "full_measurement_fallback"
+            for previous, current in zip(checkpoints, checkpoints[1:], strict=False):
+                if (
+                    previous["confident"]
+                    and current["confident"]
+                    and previous["selected_action"] == current["selected_action"]
+                ):
+                    selected_checkpoint = current
+                    stop_reason = "two_consecutive_confident_checkpoints"
+                    break
+            candidates = by_phase[str(selected_checkpoint["phase"])]
+            real_by_action = {
+                str(candidate["action"]["id"]): float(
+                    candidate["target_normalized_goodput"]
+                )
+                for candidate in candidates
+            }
+            target_by_action = {
+                str(candidate["action"]["id"]): str(
+                    candidate["action"]["target_config_id"]
+                )
+                for candidate in candidates
+            }
+            selected_action = str(selected_checkpoint["selected_action"])
+            oracle = max(real_by_action.values())
+            selected_real = real_by_action[selected_action]
+            regret = 1.0 - selected_real / oracle if oracle > 0 else 0.0
+            source_tp = 4
+            target_s = 0.0 if selected_action == "noop" else full_duration_s
+            replay_gpu_seconds = source_tp * (
+                float(selected_checkpoint["cutoff_s"]) + target_s
+            )
+            rows.append(
+                {
+                    "holdout": held_out,
+                    "decision_id": decision_id,
+                    "selected_phase": str(selected_checkpoint["phase"]),
+                    "selected_cutoff_s": float(selected_checkpoint["cutoff_s"]),
+                    "stop_reason": stop_reason,
+                    "selected_action": selected_action,
+                    "selected_target_config_id": target_by_action[selected_action],
+                    "selected_real": selected_real,
+                    "oracle_real": oracle,
+                    "regret": regret,
+                    "acceptable": regret <= ACCEPTABLE_REGRET + 1e-12,
+                    "replay_gpu_seconds_lower_bound": replay_gpu_seconds,
+                    "checkpoints": checkpoints,
+                }
+            )
+    if not rows:
+        return {"status": "INSUFFICIENT_GROUPS", "decisions": []}
+    regrets = [float(row["regret"]) for row in rows]
+    cutoffs = [float(row["selected_cutoff_s"]) for row in rows]
+    costs = [float(row["replay_gpu_seconds_lower_bound"]) for row in rows]
+    return {
+        "status": "VALID",
+        "holdout_key": holdout_key,
+        "measurement_rule": "earliest two consecutive confident checkpoints; otherwise full",
+        "acceptable_regret": ACCEPTABLE_REGRET,
+        "decision_n": len(rows),
+        "acceptable_n": sum(bool(row["acceptable"]) for row in rows),
+        "mean_regret": sum(regrets) / len(regrets),
+        "max_regret": max(regrets),
+        "mean_cutoff_s": sum(cutoffs) / len(cutoffs),
+        "total_replay_gpu_seconds_lower_bound": sum(costs),
+        "total_replay_h20_hours_lower_bound": sum(costs) / 3600.0,
+        "decisions": rows,
+    }
+
+
+def paired_sequential_delta(
+    outcome: Mapping[str, Any], telemetry: Mapping[str, Any]
+) -> dict[str, Any]:
+    if outcome.get("status") != "VALID" or telemetry.get("status") != "VALID":
+        return {"status": "INSUFFICIENT_GROUPS"}
+    before_by_id = {row["decision_id"]: row for row in outcome["decisions"]}
+    after_by_id = {row["decision_id"]: row for row in telemetry["decisions"]}
+    rows = []
+    for decision_id in sorted(set(before_by_id) & set(after_by_id)):
+        before = before_by_id[decision_id]
+        after = after_by_id[decision_id]
+        rows.append(
+            {
+                "decision_id": decision_id,
+                "outcome_action": before["selected_action"],
+                "telemetry_action": after["selected_action"],
+                "outcome_cutoff_s": before["selected_cutoff_s"],
+                "telemetry_cutoff_s": after["selected_cutoff_s"],
+                "outcome_regret": before["regret"],
+                "telemetry_regret": after["regret"],
+                "regret_delta": float(after["regret"]) - float(before["regret"]),
+                "gpu_seconds_delta": float(
+                    after["replay_gpu_seconds_lower_bound"]
+                )
+                - float(before["replay_gpu_seconds_lower_bound"]),
+                "telemetry_corrected": (not before["acceptable"])
+                and bool(after["acceptable"]),
+                "telemetry_harmed": bool(before["acceptable"])
+                and (not after["acceptable"]),
+            }
+        )
+    outcome_cost = float(outcome["total_replay_gpu_seconds_lower_bound"])
+    telemetry_cost = float(telemetry["total_replay_gpu_seconds_lower_bound"])
+    return {
+        "status": "VALID",
+        "decision_n": len(rows),
+        "corrected_n": sum(row["telemetry_corrected"] for row in rows),
+        "harmed_n": sum(row["telemetry_harmed"] for row in rows),
+        "outcome_replay_gpu_seconds_lower_bound": outcome_cost,
+        "telemetry_replay_gpu_seconds_lower_bound": telemetry_cost,
+        "gpu_cost_reduction_fraction": (
+            1.0 - telemetry_cost / outcome_cost if outcome_cost > 0 else 0.0
+        ),
+        "rows": rows,
+    }
+
+
+def build_policy(dataset_path: Path) -> dict[str, Any]:
+    dataset = json.loads(dataset_path.read_text(encoding="utf-8"))
+    if dataset.get("status") != "VALID" or dataset["sanity"]["red_flags"]:
+        raise ValueError("training dataset is not valid")
+    examples = dataset["examples"]
+    phases = sorted({str(example["phase"]) for example in examples}, key=float)
+    phase_results = {}
+    incremental_candidates = []
+    for phase in phases:
+        selected = [example for example in examples if str(example["phase"]) == phase]
+        outcome_cv = evaluate_grouped_cv(
+            selected, include_telemetry=False, holdout_key="repetition"
+        )
+        telemetry_cv = evaluate_grouped_cv(
+            selected, include_telemetry=True, holdout_key="repetition"
+        )
+        outcome_regime = evaluate_grouped_cv(
+            selected, include_telemetry=False, holdout_key="regime"
+        )
+        telemetry_regime = evaluate_grouped_cv(
+            selected, include_telemetry=True, holdout_key="regime"
+        )
+        delta = paired_delta(outcome_cv, telemetry_cv)
+        outcome_models = MODEL.fit_jackknife_ensemble(
+            selected,
+            include_telemetry=False,
+            regularization=REGULARIZATION,
+        )
+        telemetry_models = MODEL.fit_jackknife_ensemble(
+            selected,
+            include_telemetry=True,
+            regularization=REGULARIZATION,
+        )
+        incremental = bool(
+            delta.get("status") == "VALID"
+            and int(delta["corrected_n"]) >= 1
+            and int(delta["harmed_n"]) == 0
+            and float(delta["mean_regret_delta"]) < -1e-12
+            and float(telemetry_cv["max_regret"]) <= 0.05
+            and telemetry_regime.get("status") == "VALID"
+            and float(telemetry_regime["mean_regret"])
+            <= float(outcome_regime["mean_regret"]) + 1e-12
+            and float(telemetry_regime["max_regret"]) <= 0.05
+        )
+        if incremental:
+            incremental_candidates.append(phase)
+        phase_results[phase] = {
+            "cutoff_s": float(selected[0]["cutoff_s"]),
+            "outcome_only": {
+                "leave_repetition_out": outcome_cv,
+                "leave_regime_out": outcome_regime,
+                "models": MODEL.models_to_json(outcome_models),
+            },
+            "telemetry": {
+                "leave_repetition_out": telemetry_cv,
+                "leave_regime_out": telemetry_regime,
+                "models": MODEL.models_to_json(telemetry_models),
+            },
+            "paired_incremental": delta,
+            "incremental_gate": incremental,
+        }
+    outcome_sequential = evaluate_sequential_measurement_cv(
+        examples, include_telemetry=False, holdout_key="repetition"
+    )
+    telemetry_sequential = evaluate_sequential_measurement_cv(
+        examples, include_telemetry=True, holdout_key="repetition"
+    )
+    sequential_delta = paired_sequential_delta(
+        outcome_sequential, telemetry_sequential
+    )
+    retrospective_cost_gate = bool(
+        sequential_delta.get("status") == "VALID"
+        and int(sequential_delta["harmed_n"]) == 0
+        and int(telemetry_sequential["acceptable_n"])
+        >= int(outcome_sequential["acceptable_n"])
+        and float(telemetry_sequential["max_regret"]) <= 0.05
+        and float(sequential_delta["gpu_cost_reduction_fraction"]) >= 0.10
+    )
+    status = (
+        "RETROSPECTIVE_GPU_COST_SIGNAL"
+        if retrospective_cost_gate
+        else "NO_RETROSPECTIVE_GPU_COST_SIGNAL"
+    )
+    target_values = [float(example["target_normalized_goodput"]) for example in examples]
+    effect_values = [
+        float(example["target_delta_normalized_goodput"]) for example in examples
+    ]
+    invariants = {
+        "four_phases": len(phases) == 4,
+        "targets_bounded": all(0.0 <= value <= 1.0 for value in target_values),
+        "targets_not_all_identical": len(set(target_values)) > 1,
+        "effects_bounded": all(-1.0 <= value <= 1.0 for value in effect_values),
+        "effects_not_all_identical": len(set(effect_values)) > 1,
+        "models_present_every_phase": all(
+            phase_results[phase][mode]["models"]
+            for phase in phases
+            for mode in ("outcome_only", "telemetry")
+        ),
+    }
+    red_flags = [name for name, passed in invariants.items() if not passed]
+    if red_flags:
+        raise RuntimeError(f"policy sanity failed: {red_flags}")
+    return {
+        "schema": "active-intervention-policy-v0",
+        "status": status,
+        "training": {
+            "dataset": str(dataset_path),
+            "dataset_sha256": sha256_file(dataset_path),
+            "examples": len(examples),
+            "decisions": len({example["decision_id"] for example in examples}),
+            "regularization": REGULARIZATION,
+            "confidence_z": CONFIDENCE_Z,
+            "minimum_margin": MINIMUM_MARGIN,
+            "acceptable_regret": ACCEPTABLE_REGRET,
+        },
+        "measurement_policy": {
+            "rule": "earliest two consecutive confident checkpoints; otherwise full",
+            "checkpoints": [phase_results[phase]["cutoff_s"] for phase in phases],
+            "confidence_z": CONFIDENCE_Z,
+            "minimum_margin": MINIMUM_MARGIN,
+        },
+        "sequential_replay": {
+            "outcome_only": outcome_sequential,
+            "telemetry": telemetry_sequential,
+            "paired_delta": sequential_delta,
+            "retrospective_gpu_cost_gate": retrospective_cost_gate,
+            "minimum_cost_reduction_fraction": 0.10,
+        },
+        "phases": phase_results,
+        "sanity": {
+            "invariants": invariants,
+            "red_flags": red_flags,
+            "target_normalized_goodput": {
+                "n": len(target_values),
+                "min": min(target_values),
+                "max": max(target_values),
+                "distinct_n": len(set(target_values)),
+            },
+            "target_delta_normalized_goodput": {
+                "n": len(effect_values),
+                "min": min(effect_values),
+                "max": max(effect_values),
+                "distinct_n": len(set(effect_values)),
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--dataset", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    policy = build_policy(args.dataset)
+    atomic_json(args.output, policy)
+    print(
+        json.dumps(
+            {
+                "status": policy["status"],
+                "measurement_policy": policy["measurement_policy"],
+                "sanity": policy["sanity"],
+            },
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_existing.py

@@ -0,0 +1,508 @@
+#!/usr/bin/env python3
+"""Retrospective headroom audit for a fidelity-aware tuning harness.
+
+This analysis intentionally separates two questions:
+
+1. How many real cell evaluations does a simulator top-k shortlist already
+   need to recover the real optimum on the frozen SimFid surface?
+2. On the P6 anchor ladder, do Layer-1 engine features predict the next
+   anchor's feasibility better than outcome-only features from the same
+   current anchor?
+
+The second question is diagnostic rather than decision-bearing: it uses a
+small, already-observed single-workload surface and full current-anchor
+summaries.  It is a premise check for a future prospective early-probe study.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable
+
+import numpy as np
+
+
+SCHEMA = "fidelity-headroom-v1"
+DEFAULT_REGULARIZATION = 1.0
+REGULARIZATION_SENSITIVITY = (0.1, 1.0, 10.0)
+BOOTSTRAP_SEED = 20260714
+BOOTSTRAP_REPLICATES = 10_000
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def numeric(values: Iterable[float | int]) -> dict[str, Any]:
+    array = [float(value) for value in values]
+    return {
+        "n": len(array),
+        "min": min(array) if array else None,
+        "max": max(array) if array else None,
+        "distinct_n": len(set(array)),
+    }
+
+
+def score_buckets(scores: dict[str, float], tolerance: float) -> dict[str, int]:
+    if tolerance <= 0:
+        raise ValueError("score tolerance must be positive")
+    return {cell: math.floor(float(score) / tolerance) for cell, score in scores.items()}
+
+
+def topk_curve(
+    real_scores: dict[str, float],
+    simulated_scores: dict[str, float],
+    tolerance: float,
+) -> dict[str, Any]:
+    if set(real_scores) != set(simulated_scores):
+        raise ValueError("real and simulator score cells differ")
+    buckets = score_buckets(simulated_scores, tolerance)
+    ordered = sorted(
+        simulated_scores,
+        key=lambda cell: (-buckets[cell], -float(simulated_scores[cell]), cell),
+    )
+    real_best = max(float(value) for value in real_scores.values())
+    points = []
+    for nominal_k in range(1, len(ordered) + 1):
+        cutoff_bucket = buckets[ordered[nominal_k - 1]]
+        candidates = [cell for cell in ordered if buckets[cell] >= cutoff_bucket]
+        selected = max(candidates, key=lambda cell: (float(real_scores[cell]), cell))
+        selected_score = float(real_scores[selected])
+        points.append(
+            {
+                "nominal_k": nominal_k,
+                "expanded_k": len(candidates),
+                "candidates": candidates,
+                "selected_cell_after_real_final": selected,
+                "selected_real_score": selected_score,
+                "real_regret": 1.0 - selected_score / real_best,
+            }
+        )
+
+    minimum_k = {}
+    for name, threshold in (("zero", 1e-15), ("one_percent", 0.01), ("five_percent", 0.05)):
+        eligible = [point for point in points if point["real_regret"] <= threshold]
+        minimum_k[name] = (
+            {
+                "nominal_k": eligible[0]["nominal_k"],
+                "expanded_k": eligible[0]["expanded_k"],
+            }
+            if eligible
+            else None
+        )
+    return {
+        "real_best": real_best,
+        "minimum_k": minimum_k,
+        "points": points,
+    }
+
+
+@dataclass(frozen=True)
+class Transition:
+    cell: str
+    current_anchor: float
+    next_anchor: float
+    external: tuple[float, ...]
+    instrumentation: tuple[float, ...]
+    next_feasible: int
+
+
+EXTERNAL_FEATURES = (
+    "log_current_rate_per_gpu",
+    "log_next_over_current_rate",
+    "log2_tp",
+    "log2_mns",
+    "current_pass_rate",
+    "ttft_max_over_6s",
+    "tpot_max_over_50ms",
+    "exact_output_fraction",
+    "early_stopped",
+)
+
+INSTRUMENTATION_FEATURES = (
+    "waiting_mean",
+    "waiting_max",
+    "decode_batch_mean",
+    "decode_batch_cv",
+    "kv_usage_mean",
+    "kv_usage_max",
+    "graph_none_share",
+    "graph_full_share",
+    "padding_fraction",
+    "prefill_token_fraction",
+    "model_steps_per_second",
+)
+
+
+def _finite(value: float | int | None) -> float:
+    if value is None:
+        return 0.0
+    result = float(value)
+    if not math.isfinite(result):
+        raise ValueError(f"non-finite feature: {value}")
+    return result
+
+
+def build_transitions(phase6: dict[str, Any]) -> list[Transition]:
+    transitions = []
+    for cell, cell_result in sorted(phase6["cells"].items()):
+        anchors = sorted(cell_result["anchors"], key=lambda item: float(item["anchor"]))
+        for current, following in zip(anchors, anchors[1:]):
+            if following["accepted_feasible"] is None:
+                continue
+            primary = current["primary"]
+            next_primary = following["primary"]
+            layer = current["layer1"]
+            rate = float(primary["selection"]["offered_req_s_per_gpu"])
+            next_rate = float(next_primary["selection"]["offered_req_s_per_gpu"])
+            selected_count = int(primary["selection"]["count"])
+            if rate <= 0 or next_rate <= 0 or selected_count <= 0:
+                raise ValueError("rates and selected counts must be positive")
+            external = (
+                math.log(rate),
+                math.log(next_rate / rate),
+                math.log2(float(cell_result["tp"])),
+                math.log2(float(cell_result["mns"])),
+                float(primary["pass_rate"]),
+                _finite(primary["ttft_ms"]["max"]) / 6000.0,
+                _finite(primary["tpot_ms"]["max"]) / 50.0,
+                float(primary["exact_output_count"]) / selected_count,
+                float(bool(primary["early_stopped"])),
+            )
+            graph_shares = layer.get("graph_mode_shares", {})
+            prefill_tokens = _finite(layer["prefill_tokens"])
+            decode_tokens = _finite(layer["decode_tokens"])
+            instrumentation = (
+                _finite(layer["waiting_mean"]),
+                _finite(layer["waiting_max"]),
+                _finite(layer["decode_B_mean"]),
+                _finite(layer["decode_B_cv"]),
+                _finite(layer["kv_usage_mean"]),
+                _finite(layer["kv_usage_max"]),
+                float(graph_shares.get("NONE", 0.0)),
+                float(graph_shares.get("FULL", 0.0)),
+                _finite(layer["padding_fraction"]),
+                prefill_tokens / max(1.0, prefill_tokens + decode_tokens),
+                _finite(layer["model_steps"]) / float(primary["interval"]["elapsed_s"]),
+            )
+            transitions.append(
+                Transition(
+                    cell=cell,
+                    current_anchor=float(current["anchor"]),
+                    next_anchor=float(following["anchor"]),
+                    external=external,
+                    instrumentation=instrumentation,
+                    next_feasible=int(bool(following["accepted_feasible"])),
+                )
+            )
+    return transitions
+
+
+def _sigmoid(values: np.ndarray) -> np.ndarray:
+    clipped = np.clip(values, -30.0, 30.0)
+    return 1.0 / (1.0 + np.exp(-clipped))
+
+
+def _fit_logistic(x: np.ndarray, y: np.ndarray, regularization: float) -> np.ndarray:
+    weights = np.zeros(x.shape[1], dtype=np.float64)
+    penalty = np.eye(x.shape[1], dtype=np.float64)
+    penalty[0, 0] = 0.0
+    for _ in range(100):
+        probability = _sigmoid(x @ weights)
+        gradient = x.T @ (probability - y) / len(y)
+        gradient += regularization * penalty @ weights / len(y)
+        curvature = probability * (1.0 - probability)
+        hessian = (x.T * curvature) @ x / len(y)
+        hessian += regularization * penalty / len(y)
+        step = np.linalg.lstsq(hessian, gradient, rcond=None)[0]
+        weights -= step
+        if float(np.max(np.abs(step))) < 1e-9:
+            break
+    return weights
+
+
+def _classification_metrics(y: np.ndarray, probability: np.ndarray) -> dict[str, Any]:
+    if np.any(probability < 0.0) or np.any(probability > 1.0):
+        raise ValueError("classification probabilities must be in [0, 1]")
+    prediction = probability >= 0.5
+    true_positive = int(np.sum(prediction & (y == 1)))
+    true_negative = int(np.sum(~prediction & (y == 0)))
+    false_positive = int(np.sum(prediction & (y == 0)))
+    false_negative = int(np.sum(~prediction & (y == 1)))
+    positive_total = true_positive + false_negative
+    negative_total = true_negative + false_positive
+    balanced = 0.5 * (
+        true_positive / positive_total + true_negative / negative_total
+    )
+    clipped = np.clip(probability, 1e-12, 1.0 - 1e-12)
+    return {
+        "accuracy": float(np.mean(prediction == y)),
+        "balanced_accuracy": float(balanced),
+        "brier": float(np.mean((probability - y) ** 2)),
+        "log_loss": float(np.mean(-(y * np.log(clipped) + (1 - y) * np.log(1 - clipped)))),
+        "confusion": {
+            "true_positive": true_positive,
+            "true_negative": true_negative,
+            "false_positive": false_positive,
+            "false_negative": false_negative,
+        },
+    }
+
+
+def _mcnemar_exact_p(outcome_only_correct: int, instrumentation_only_correct: int) -> float:
+    discordant = outcome_only_correct + instrumentation_only_correct
+    if discordant == 0:
+        return 1.0
+    tail = sum(
+        math.comb(discordant, value)
+        for value in range(min(outcome_only_correct, instrumentation_only_correct) + 1)
+    ) / (2**discordant)
+    return min(1.0, 2.0 * tail)
+
+
+def grouped_predictions(
+    transitions: list[Transition],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> tuple[np.ndarray, np.ndarray, list[str]]:
+    probabilities = []
+    labels = []
+    test_cells = []
+    for held_out in sorted({transition.cell for transition in transitions}):
+        train = [transition for transition in transitions if transition.cell != held_out]
+        test = [transition for transition in transitions if transition.cell == held_out]
+
+        def row(transition: Transition) -> np.ndarray:
+            values = transition.external
+            if instrumentation_aware:
+                values += transition.instrumentation
+            return np.asarray((1.0, *values), dtype=np.float64)
+
+        x_train = np.stack([row(transition) for transition in train])
+        x_test = np.stack([row(transition) for transition in test])
+        y_train = np.asarray([transition.next_feasible for transition in train], dtype=np.float64)
+        mean = x_train[:, 1:].mean(axis=0)
+        standard_deviation = x_train[:, 1:].std(axis=0)
+        standard_deviation[standard_deviation < 1e-8] = 1.0
+        x_train[:, 1:] = (x_train[:, 1:] - mean) / standard_deviation
+        x_test[:, 1:] = (x_test[:, 1:] - mean) / standard_deviation
+        weights = _fit_logistic(x_train, y_train, regularization)
+        probabilities.extend(_sigmoid(x_test @ weights).tolist())
+        labels.extend(transition.next_feasible for transition in test)
+        test_cells.extend(held_out for _ in test)
+    return (
+        np.asarray(labels, dtype=np.int64),
+        np.asarray(probabilities, dtype=np.float64),
+        test_cells,
+    )
+
+
+def _group_bootstrap_delta(
+    y: np.ndarray,
+    outcome_probability: np.ndarray,
+    instrumentation_probability: np.ndarray,
+    cells: list[str],
+) -> dict[str, Any]:
+    groups = sorted(set(cells))
+    indices = {group: np.asarray([i for i, cell in enumerate(cells) if cell == group]) for group in groups}
+    random = np.random.default_rng(BOOTSTRAP_SEED)
+    accuracy_deltas = []
+    brier_deltas = []
+    for _ in range(BOOTSTRAP_REPLICATES):
+        sampled = random.choice(groups, size=len(groups), replace=True)
+        selected = np.concatenate([indices[group] for group in sampled])
+        selected_y = y[selected]
+        outcome = outcome_probability[selected]
+        instrumentation = instrumentation_probability[selected]
+        accuracy_deltas.append(
+            float(np.mean((instrumentation >= 0.5) == selected_y))
+            - float(np.mean((outcome >= 0.5) == selected_y))
+        )
+        brier_deltas.append(
+            float(np.mean((instrumentation - selected_y) ** 2))
+            - float(np.mean((outcome - selected_y) ** 2))
+        )
+    return {
+        "semantics": "group bootstrap over cells; diagnostic confidence interval",
+        "replicates": BOOTSTRAP_REPLICATES,
+        "seed": BOOTSTRAP_SEED,
+        "accuracy_delta_instrumentation_minus_outcome": {
+            "point": float(np.mean((instrumentation_probability >= 0.5) == y))
+            - float(np.mean((outcome_probability >= 0.5) == y)),
+            "ci95": [float(x) for x in np.percentile(accuracy_deltas, [2.5, 97.5])],
+        },
+        "brier_delta_instrumentation_minus_outcome": {
+            "point": float(np.mean((instrumentation_probability - y) ** 2))
+            - float(np.mean((outcome_probability - y) ** 2)),
+            "ci95": [float(x) for x in np.percentile(brier_deltas, [2.5, 97.5])],
+        },
+    }
+
+
+def transition_analysis(transitions: list[Transition]) -> dict[str, Any]:
+    sensitivity = {}
+    headline_payload = None
+    for regularization in REGULARIZATION_SENSITIVITY:
+        y, outcome_probability, cells = grouped_predictions(
+            transitions,
+            instrumentation_aware=False,
+            regularization=regularization,
+        )
+        instrumentation_y, instrumentation_probability, instrumentation_cells = grouped_predictions(
+            transitions,
+            instrumentation_aware=True,
+            regularization=regularization,
+        )
+        if not np.array_equal(y, instrumentation_y) or cells != instrumentation_cells:
+            raise AssertionError("model folds or labels differ")
+        outcome_correct = (outcome_probability >= 0.5) == y
+        instrumentation_correct = (instrumentation_probability >= 0.5) == y
+        payload = {
+            "outcome_only": _classification_metrics(y, outcome_probability),
+            "instrumentation_aware": _classification_metrics(y, instrumentation_probability),
+            "paired_correctness": {
+                "both_correct": int(np.sum(outcome_correct & instrumentation_correct)),
+                "outcome_only_correct": int(np.sum(outcome_correct & ~instrumentation_correct)),
+                "instrumentation_only_correct": int(np.sum(~outcome_correct & instrumentation_correct)),
+                "both_wrong": int(np.sum(~outcome_correct & ~instrumentation_correct)),
+            },
+            "bootstrap": _group_bootstrap_delta(
+                y,
+                outcome_probability,
+                instrumentation_probability,
+                cells,
+            ),
+        }
+        payload["paired_correctness"]["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+            payload["paired_correctness"]["outcome_only_correct"],
+            payload["paired_correctness"]["instrumentation_only_correct"],
+        )
+        sensitivity[str(regularization)] = payload
+        if regularization == DEFAULT_REGULARIZATION:
+            headline_payload = payload
+    assert headline_payload is not None
+    labels = [transition.next_feasible for transition in transitions]
+    accuracy_deltas = [
+        value["instrumentation_aware"]["accuracy"] - value["outcome_only"]["accuracy"]
+        for value in sensitivity.values()
+    ]
+    brier_deltas = [
+        value["instrumentation_aware"]["brier"] - value["outcome_only"]["brier"]
+        for value in sensitivity.values()
+    ]
+    return {
+        "status": "RETROSPECTIVE_DIAGNOSTIC_ONLY",
+        "estimand": "next-anchor feasibility from the full current-anchor summary",
+        "split": "leave-one-cell-out",
+        "model": "L2 logistic regression with train-fold standardization",
+        "external_features": list(EXTERNAL_FEATURES),
+        "instrumentation_features": list(INSTRUMENTATION_FEATURES),
+        "headline_regularization": DEFAULT_REGULARIZATION,
+        "headline": headline_payload,
+        "regularization_sensitivity": sensitivity,
+        "sensitivity_summary": {
+            "accuracy_delta_min_max": [min(accuracy_deltas), max(accuracy_deltas)],
+            "brier_delta_min_max": [min(brier_deltas), max(brier_deltas)],
+            "incremental_signal_verdict": "NEEDS_PROSPECTIVE_EVIDENCE",
+        },
+        "label_sanity": {
+            **numeric(labels),
+            "positive": sum(labels),
+            "negative": len(labels) - sum(labels),
+        },
+    }
+
+
+def analyze(simfid_path: Path, phase6_path: Path) -> dict[str, Any]:
+    simfid = json.loads(simfid_path.read_text())
+    phase6 = json.loads(phase6_path.read_text())
+    real_scores = {cell: float(score) for cell, score in simfid["real_scores"].items()}
+    topk = {}
+    for reading, payload in sorted(simfid["analyses"].items()):
+        tie = payload["metrics"]["tie_buckets"]["simulator"]
+        topk[reading] = topk_curve(
+            real_scores,
+            {cell: float(score) for cell, score in payload["simulated_scores"].items()},
+            float(tie["tolerance"]),
+        )
+    transitions = build_transitions(phase6)
+    transition_result = transition_analysis(transitions)
+    red_flags = []
+    if len(real_scores) != 12:
+        red_flags.append("unexpected_simfid_cell_count")
+    if len(transitions) == 0 or len(set(x.next_feasible for x in transitions)) != 2:
+        red_flags.append("transition_labels_missing_or_single_class")
+    if any(not math.isfinite(value) or value < 0 for value in real_scores.values()):
+        red_flags.append("invalid_real_score")
+    return {
+        "schema": SCHEMA,
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": "retrospective single-workload premise audit; not prospective contribution evidence",
+        "provenance": {
+            "simfid_metrics": str(simfid_path.resolve()),
+            "simfid_sha256": sha256_file(simfid_path),
+            "phase6_metrics": str(phase6_path.resolve()),
+            "phase6_sha256": sha256_file(phase6_path),
+        },
+        "topk_headroom": topk,
+        "next_anchor_prediction": transition_result,
+        "decision": {
+            "current_surface_can_show_selection_contribution": False,
+            "reason": (
+                "The strongest frozen-calibrated SLO reading reaches zero real regret "
+                "after real evaluation of its first two-cell tie bucket. A method that "
+                "requires one calibration probe and one final verification cannot use "
+                "this single task to demonstrate fewer real cell evaluations."
+            ),
+            "prospective_target": (
+                "Test whether internal features from a short, shared real probe reduce "
+                "the number or duration of full frontier evaluations relative to an "
+                "outcome-only model given the same probe."
+            ),
+        },
+        "sanity": {
+            "real_scores": numeric(real_scores.values()),
+            "simulator_readings": len(topk),
+            "transitions": len(transitions),
+            "transition_cells": len({transition.cell for transition in transitions}),
+            "red_flags": red_flags,
+            "invariants": {
+                "same_cells_all_readings": all(
+                    set(payload["simulated_scores"]) == set(real_scores)
+                    for payload in simfid["analyses"].values()
+                ),
+                "scores_nonnegative": all(value >= 0 for value in real_scores.values()),
+                "transition_features_finite": all(
+                    all(math.isfinite(value) for value in (*item.external, *item.instrumentation))
+                    for item in transitions
+                ),
+                "probabilities_bounded": True,
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--simfid-metrics", type=Path, required=True)
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(args.simfid_metrics, args.phase6_metrics)
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(json.dumps({"status": result["status"], "output": str(args.output)}, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_pilot.py

@@ -0,0 +1,336 @@
+#!/usr/bin/env python3
+"""Evaluate frozen outcome-only and instrumentation-aware policies on P1."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from analyze_existing import _classification_metrics, _mcnemar_exact_p
+from analyze_prefixes import (
+    PrefixExample,
+    _load_jsonl,
+    _prefix_features,
+    numeric,
+    policy_metrics,
+    predict_frozen_model,
+    sha256_file,
+)
+
+
+def result_path(run_root: Path, cell: str, level: str, replicate: int) -> Path:
+    return run_root / "cells" / cell / f"{level}-rep{replicate}" / "result.json"
+
+
+def requests_path(run_root: Path, cell: str, level: str, replicate: int) -> Path:
+    return run_root / "cells" / cell / f"{level}-rep{replicate}" / "requests.jsonl"
+
+
+def selection_for(
+    manifest: dict[str, Any], cell: str, level: str, replicate: int
+) -> dict[str, Any]:
+    role = f"{level}{replicate}"
+    return manifest["cells"][cell]["targets"][level]["selections"][role]
+
+
+def campaign_gpu_accounting(
+    primary_state_path: Path, prior_state_paths: tuple[Path, ...] = ()
+) -> dict[str, Any]:
+    attempts = []
+    for role, path in (
+        [("prior_failure", path) for path in prior_state_paths]
+        + [("primary", primary_state_path)]
+    ):
+        state = json.loads(path.read_text(encoding="utf-8"))
+        gpu_hours = float(state["gpu_hours_total"])
+        attempts.append(
+            {
+                "role": role,
+                "path": str(path.resolve()),
+                "sha256": sha256_file(path),
+                "status": state["status"],
+                "h20_hours": gpu_hours,
+            }
+        )
+    total = sum(attempt["h20_hours"] for attempt in attempts)
+    primary = json.loads(primary_state_path.read_text(encoding="utf-8"))
+    hard_cap = float(primary["hard_cap_h20_hours"])
+    return {
+        "attempts": attempts,
+        "aggregate_h20_hours": total,
+        "hard_cap_h20_hours": hard_cap,
+        "invariants": {
+            "costs_nonnegative": all(
+                attempt["h20_hours"] >= 0.0 for attempt in attempts
+            ),
+            "aggregate_below_cap": 0.0 <= total < hard_cap,
+        },
+    }
+
+
+def build_pilot_examples(
+    manifest: dict[str, Any], run_root: Path, cutoff_s: float
+) -> tuple[list[PrefixExample], list[dict[str, Any]], list[str]]:
+    examples = []
+    details = []
+    red_flags = []
+    for cell, config in sorted(manifest["cells"].items()):
+        stream_path = next((run_root / "cells" / cell / "opprof").glob("*.jsonl"))
+        stream = _load_jsonl(stream_path, require_key="submit_mono_ns")
+        for level in ("low", "high"):
+            results = [
+                json.loads(result_path(run_root, cell, level, replicate).read_text())
+                for replicate in (1, 2, 3)
+            ]
+            votes = [bool(result["feasible"]) for result in results]
+            adjudicated = sum(votes) >= 2
+            primary = results[0]
+            requests = _load_jsonl(requests_path(run_root, cell, level, 1))
+            exact_timestamps = sum(
+                request.get("completed_elapsed_s") is not None for request in requests
+            )
+            actual_outcomes = sum(
+                request.get("completed_mono_ns") is not None for request in requests
+            )
+            if exact_timestamps != actual_outcomes:
+                red_flags.append(f"timestamp_count_mismatch_{cell}_{level}")
+            expected = selection_for(manifest, cell, level, 1)
+            if int(primary["selection"]["count"]) != int(expected["selected_count"]):
+                red_flags.append(f"selection_count_mismatch_{cell}_{level}")
+            for result_key, manifest_key in (
+                ("request_id_order_sha256", "request_id_order_sha256"),
+                ("arrival_order_sha256", "arrival_order_sha256"),
+                ("raw_length_order_sha256", "input_length_order_sha256"),
+            ):
+                if primary["selection"][result_key] != expected[manifest_key]:
+                    red_flags.append(f"selection_hash_mismatch_{cell}_{level}_{result_key}")
+            start_ns = int(primary["interval"]["start_mono_ns"])
+            end_ns = start_ns + int(cutoff_s * 1e9)
+            records = [
+                record
+                for record in stream
+                if record.get("model_executed")
+                and start_ns <= int(record["submit_mono_ns"]) <= end_ns
+            ]
+            outcome, instrumentation, completion_source = _prefix_features(
+                primary=primary,
+                tp=int(config["tp"]),
+                max_num_seqs=int(config["mns"]),
+                requests=requests,
+                records=records,
+                cutoff_s=cutoff_s,
+            )
+            example = PrefixExample(
+                cell=cell,
+                anchor=float(primary["anchor"]),
+                cutoff_s=cutoff_s,
+                tp=int(config["tp"]),
+                full_elapsed_s=float(primary["interval"]["elapsed_s"]),
+                feasible=int(adjudicated),
+                primary_feasible=int(bool(primary["feasible"])),
+                outcome=outcome,
+                instrumentation=instrumentation,
+                completion_time_source=completion_source,
+            )
+            examples.append(example)
+            details.append(
+                {
+                    "cell": cell,
+                    "level": level,
+                    "anchor_rep1": primary["anchor"],
+                    "selected_count_rep1": primary["selection"]["count"],
+                    "votes": votes,
+                    "pass_rates": [result["pass_rate"] for result in results],
+                    "adjudicated_feasible": adjudicated,
+                    "primary_feasible": bool(primary["feasible"]),
+                    "actual_timestamped_outcomes": actual_outcomes,
+                    "selected_outcomes": len(requests),
+                    "prefix_layer1_records": len(records),
+                    "completion_time_source": completion_source,
+                }
+            )
+    return examples, details, red_flags
+
+
+def analyze(
+    manifest_path: Path,
+    model_path: Path,
+    run_root: Path,
+    prior_state_paths: tuple[Path, ...] = (),
+) -> dict[str, Any]:
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    models = json.loads(model_path.read_text(encoding="utf-8"))
+    state_path = run_root / "controller-state.json"
+    state = json.loads(state_path.read_text(encoding="utf-8"))
+    gpu_accounting = campaign_gpu_accounting(state_path, prior_state_paths)
+    cutoff_s = float(models["cutoff_s"])
+    threshold = float(models["accept_probability"])
+    examples, details, red_flags = build_pilot_examples(manifest, run_root, cutoff_s)
+    labels = np.asarray([example.feasible for example in examples], dtype=np.int64)
+    outcome_probability = predict_frozen_model(models["models"]["outcome_only"], examples)
+    instrumentation_probability = predict_frozen_model(
+        models["models"]["instrumentation_aware"], examples
+    )
+    outcome_policy = policy_metrics(
+        examples, labels, outcome_probability, threshold
+    )
+    instrumentation_policy = policy_metrics(
+        examples, labels, instrumentation_probability, threshold
+    )
+    outcome_correct = (outcome_probability >= 0.5) == labels
+    instrumentation_correct = (instrumentation_probability >= 0.5) == labels
+    paired = {
+        "both_correct": int(np.sum(outcome_correct & instrumentation_correct)),
+        "outcome_only_correct": int(np.sum(outcome_correct & ~instrumentation_correct)),
+        "instrumentation_only_correct": int(np.sum(~outcome_correct & instrumentation_correct)),
+        "both_wrong": int(np.sum(~outcome_correct & ~instrumentation_correct)),
+    }
+    paired["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+        paired["outcome_only_correct"], paired["instrumentation_only_correct"]
+    )
+    for detail, outcome_p, instrumentation_p in zip(
+        details, outcome_probability, instrumentation_probability
+    ):
+        detail["outcome_probability_feasible"] = float(outcome_p)
+        detail["instrumentation_probability_feasible"] = float(instrumentation_p)
+
+    positive = int(np.sum(labels))
+    negative = len(labels) - positive
+    if state["status"] != "complete" or int(state["completed_cells"]) != 6:
+        red_flags.append("campaign_incomplete")
+    if positive < 3 or negative < 3:
+        red_flags.append("insufficient_label_balance")
+    if any(
+        detail["actual_timestamped_outcomes"] == 0 for detail in details
+    ):
+        red_flags.append("no_exact_request_timestamps")
+    if not all(gpu_accounting["invariants"].values()):
+        red_flags.append("hard_cap_exceeded")
+
+    outcome_errors = outcome_policy["false_accept"] + outcome_policy["false_reject"]
+    instrumentation_errors = (
+        instrumentation_policy["false_accept"]
+        + instrumentation_policy["false_reject"]
+    )
+    outcome_decisions = outcome_policy["early_accept"] + outcome_policy["early_reject"]
+    instrumentation_decisions = (
+        instrumentation_policy["early_accept"]
+        + instrumentation_policy["early_reject"]
+    )
+    outcome_reduction = outcome_policy["valid_cost_reduction_fraction"]
+    instrumentation_reduction = instrumentation_policy["valid_cost_reduction_fraction"]
+    cost_delta = (
+        instrumentation_reduction - outcome_reduction
+        if outcome_reduction is not None and instrumentation_reduction is not None
+        else None
+    )
+    data_valid = not red_flags
+    safety_gate = instrumentation_errors == 0 and instrumentation_errors <= outcome_errors
+    incremental_gate = (
+        instrumentation_decisions - outcome_decisions >= 3
+        or (cost_delta is not None and cost_delta >= 0.15)
+    )
+    pilot_pass = data_valid and safety_gate and incremental_gate
+
+    return {
+        "schema": "fidelity-prefix-pilot-result-v1",
+        "status": "PILOT_PASS" if pilot_pass else "PILOT_FAIL",
+        "scope": "held-out single-task gate; not paper-facing contribution evidence",
+        "provenance": {
+            "manifest": str(manifest_path.resolve()),
+            "manifest_sha256": sha256_file(manifest_path),
+            "frozen_models": str(model_path.resolve()),
+            "frozen_models_sha256": sha256_file(model_path),
+            "controller_state": str(state_path.resolve()),
+            "controller_state_sha256": sha256_file(state_path),
+        },
+        "cutoff_s": cutoff_s,
+        "threshold": threshold,
+        "examples": details,
+        "outcome_only": {
+            "classification": _classification_metrics(labels, outcome_probability),
+            "policy": outcome_policy,
+        },
+        "instrumentation_aware": {
+            "classification": _classification_metrics(labels, instrumentation_probability),
+            "policy": instrumentation_policy,
+        },
+        "paired_correctness": paired,
+        "gate": {
+            "data_valid": data_valid,
+            "safety_gate": safety_gate,
+            "incremental_gate": incremental_gate,
+            "additional_early_decisions": instrumentation_decisions - outcome_decisions,
+            "valid_cost_reduction_fraction_delta": cost_delta,
+            "opens_expanded_p2": pilot_pass,
+        },
+        "gpu": {
+            "primary_attempt_h20_hours": state["gpu_hours_total"],
+            **gpu_accounting,
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "labels": {
+                **numeric(labels.tolist()),
+                "positive": positive,
+                "negative": negative,
+            },
+            "full_elapsed_s": numeric(example.full_elapsed_s for example in examples),
+            "remaining_h20_hours": numeric(
+                example.remaining_h20_hours for example in examples
+            ),
+            "outcome_probability": numeric(outcome_probability.tolist()),
+            "instrumentation_probability": numeric(
+                instrumentation_probability.tolist()
+            ),
+            "invariants": {
+                "examples_12": len(examples) == 12,
+                "cells_6": len({example.cell for example in examples}) == 6,
+                "ratios_bounded": bool(
+                    np.all((outcome_probability >= 0) & (outcome_probability <= 1))
+                    and np.all(
+                        (instrumentation_probability >= 0)
+                        & (instrumentation_probability <= 1)
+                    )
+                ),
+                "costs_nonnegative": all(
+                    example.remaining_h20_hours >= 0 for example in examples
+                ),
+                "all_cell_validations": all(
+                    all(cell["validation"]["invariants"].values())
+                    for cell in state["cells"].values()
+                ),
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--frozen-models", type=Path, required=True)
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("--prior-state", type=Path, action="append", default=[])
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(
+        args.manifest,
+        args.frozen_models,
+        args.run_root,
+        tuple(args.prior_state),
+    )
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(json.dumps({
+        "status": result["status"],
+        "gate": result["gate"],
+        "sanity_red_flags": result["sanity"]["red_flags"],
+    }, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_pilot_e2e.py

@@ -0,0 +1,429 @@
+#!/usr/bin/env python3
+"""Replay the P1 simulator shortlist under full and prefix policies."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import subprocess
+from pathlib import Path
+from typing import Any
+
+from analyze_prefixes import numeric, sha256_file
+
+
+AITUNER_ROOT = Path(__file__).resolve().parents[2]
+FROZEN_K = 2
+CUTOFF_S = 5.0
+THRESHOLD = 0.95
+
+
+def git_capture(*arguments: str) -> str:
+    return subprocess.run(
+        ["git", "-C", str(AITUNER_ROOT), *arguments],
+        check=True,
+        text=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    ).stdout
+
+
+def setup_costs(state: dict[str, Any]) -> dict[str, float]:
+    result = {}
+    for cell, payload in state["cells"].items():
+        tp = int(payload["tp"])
+        annotation_intervals = sum(
+            float(run["elapsed_s"]) * tp / 3600.0
+            for run in payload["runs"]
+            if run["role"] not in {"low1", "high1"}
+        )
+        primary_intervals = sum(
+            float(run["elapsed_s"]) * tp / 3600.0
+            for run in payload["runs"]
+            if run["role"] in {"low1", "high1"}
+        )
+        setup = float(payload["gpu_hours"]) - annotation_intervals - primary_intervals
+        if setup < -1e-12:
+            raise ValueError(f"negative inferred setup cost: {cell}={setup}")
+        result[cell] = max(0.0, setup)
+    return result
+
+
+def build_candidates(
+    manifest: dict[str, Any],
+    state: dict[str, Any],
+    strong: dict[str, Any],
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+    baseline_probability = strong["headline"]["sim_plus_outcome"]["probability"]
+    instrument_probability = strong["headline"][
+        "sim_plus_outcome_plus_instrumentation"
+    ]["probability"]
+    setup = setup_costs(state)
+    anchors = []
+    for detail, baseline_p, instrument_p in zip(
+        strong["pilot_examples"], baseline_probability, instrument_probability
+    ):
+        cell = str(detail["cell"])
+        level = str(detail["level"])
+        role = f"{level}1"
+        selection = manifest["cells"][cell]["targets"][level]["selections"][role]
+        run = next(
+            item for item in state["cells"][cell]["runs"] if item["role"] == role
+        )
+        tp = int(state["cells"][cell]["tp"])
+        full_cost = float(run["elapsed_s"]) * tp / 3600.0
+        prefix_cost = min(CUTOFF_S, float(run["elapsed_s"])) * tp / 3600.0
+        anchors.append(
+            {
+                "cell": cell,
+                "level": level,
+                "role": role,
+                "tp": tp,
+                "real_feasible": bool(detail["adjudicated_feasible"]),
+                "real_goodput_req_s_per_gpu": float(
+                    selection["offered_req_s_per_gpu"]
+                ),
+                "sim_feasible": bool(detail["sim_slo_feasible"]),
+                "sim_pass_rate": float(detail["sim_slo_pass_rate"]),
+                "sim_throughput_req_s_per_gpu": float(
+                    detail["sim_completed_throughput_per_gpu"]
+                ),
+                "baseline_probability": float(baseline_p),
+                "instrument_probability": float(instrument_p),
+                "setup_h20_hours": setup[cell],
+                "full_trial_h20_hours": full_cost,
+                "prefix_h20_hours": prefix_cost,
+            }
+        )
+
+    candidates = []
+    for cell in sorted(manifest["cells"]):
+        feasible = [
+            anchor for anchor in anchors if anchor["cell"] == cell and anchor["sim_feasible"]
+        ]
+        if not feasible:
+            continue
+        candidates.append(
+            max(feasible, key=lambda anchor: anchor["sim_throughput_req_s_per_gpu"])
+        )
+    candidates.sort(
+        key=lambda anchor: (
+            -anchor["sim_throughput_req_s_per_gpu"],
+            anchor["cell"],
+        )
+    )
+    return anchors, candidates
+
+
+def expanded_top_k(candidates: list[dict[str, Any]], k: int) -> list[dict[str, Any]]:
+    if not candidates or k <= 0:
+        return []
+    boundary = candidates[min(k, len(candidates)) - 1][
+        "sim_throughput_req_s_per_gpu"
+    ]
+    return [
+        candidate
+        for candidate in candidates
+        if candidate["sim_throughput_req_s_per_gpu"] >= boundary - 1e-12
+    ]
+
+
+def selected_result(
+    evaluated: list[dict[str, Any]], feasible_key: str
+) -> tuple[str | None, float | None]:
+    feasible = [candidate for candidate in evaluated if candidate[feasible_key]]
+    if not feasible:
+        return None, None
+    best = max(feasible, key=lambda candidate: candidate["real_goodput_req_s_per_gpu"])
+    return str(best["cell"]), float(best["real_goodput_req_s_per_gpu"])
+
+
+def replay(
+    shortlist: list[dict[str, Any]],
+    *,
+    probability_key: str | None,
+    oracle_goodput: float,
+    common_failure_h20_hours: float,
+) -> dict[str, Any]:
+    evaluated = []
+    online_cost = 0.0
+    early_accept = 0
+    early_reject = 0
+    false_accept = 0
+    false_reject = 0
+    for candidate in shortlist:
+        current = dict(candidate)
+        online_cost += current["setup_h20_hours"]
+        if probability_key is None:
+            predicted_feasible = current["real_feasible"]
+            online_cost += current["full_trial_h20_hours"]
+            action = "full"
+        else:
+            probability = float(current[probability_key])
+            if probability >= THRESHOLD:
+                predicted_feasible = True
+                early_accept += 1
+                online_cost += current["prefix_h20_hours"]
+                action = "early_accept"
+                false_accept += int(not current["real_feasible"])
+            elif probability <= 1.0 - THRESHOLD:
+                predicted_feasible = False
+                early_reject += 1
+                online_cost += current["prefix_h20_hours"]
+                action = "early_reject"
+                false_reject += int(current["real_feasible"])
+            else:
+                predicted_feasible = current["real_feasible"]
+                online_cost += current["full_trial_h20_hours"]
+                action = "continue_full"
+        current["policy_feasible"] = predicted_feasible
+        current["action"] = action
+        evaluated.append(current)
+    selected_cell, selected_goodput = selected_result(evaluated, "policy_feasible")
+    regret = (
+        1.0 - selected_goodput / oracle_goodput
+        if selected_goodput is not None and oracle_goodput > 0
+        else None
+    )
+    return {
+        "selected_cell": selected_cell,
+        "selected_real_goodput_req_s_per_gpu": selected_goodput,
+        "real_regret": regret,
+        "online_h20_hours": online_cost,
+        "conservative_h20_hours_with_prior_failure": (
+            online_cost + common_failure_h20_hours
+        ),
+        "early_accept": early_accept,
+        "early_reject": early_reject,
+        "false_accept": false_accept,
+        "false_reject": false_reject,
+        "evaluated": [
+            {
+                "cell": item["cell"],
+                "level": item["level"],
+                "action": item["action"],
+                "real_feasible": item["real_feasible"],
+                "policy_feasible": item["policy_feasible"],
+            }
+            for item in evaluated
+        ],
+    }
+
+
+def analyze(
+    manifest_path: Path,
+    state_path: Path,
+    prior_state_path: Path,
+    strong_path: Path,
+) -> dict[str, Any]:
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    state = json.loads(state_path.read_text(encoding="utf-8"))
+    prior = json.loads(prior_state_path.read_text(encoding="utf-8"))
+    strong = json.loads(strong_path.read_text(encoding="utf-8"))
+    anchors, candidates = build_candidates(manifest, state, strong)
+    oracle_anchor = max(
+        (anchor for anchor in anchors if anchor["real_feasible"]),
+        key=lambda anchor: anchor["real_goodput_req_s_per_gpu"],
+    )
+    oracle_goodput = float(oracle_anchor["real_goodput_req_s_per_gpu"])
+    common_failure = float(prior["gpu_hours_total"])
+    by_k = {}
+    for k in (1, 2, 3, 6):
+        shortlist = expanded_top_k(candidates, k)
+        full = replay(
+            shortlist,
+            probability_key=None,
+            oracle_goodput=oracle_goodput,
+            common_failure_h20_hours=common_failure,
+        )
+        baseline = replay(
+            shortlist,
+            probability_key="baseline_probability",
+            oracle_goodput=oracle_goodput,
+            common_failure_h20_hours=common_failure,
+        )
+        instrument = replay(
+            shortlist,
+            probability_key="instrument_probability",
+            oracle_goodput=oracle_goodput,
+            common_failure_h20_hours=common_failure,
+        )
+        for result in (baseline, instrument):
+            result["online_cost_reduction_vs_full"] = (
+                1.0 - result["online_h20_hours"] / full["online_h20_hours"]
+            )
+            result["conservative_cost_reduction_vs_full"] = 1.0 - (
+                result["conservative_h20_hours_with_prior_failure"]
+                / full["conservative_h20_hours_with_prior_failure"]
+            )
+        by_k[str(k)] = {
+            "actual_shortlist_size": len(shortlist),
+            "shortlist": [candidate["cell"] for candidate in shortlist],
+            "sim_top_k_plus_real_final": full,
+            "sim_plus_outcome": baseline,
+            "sim_plus_outcome_plus_instrumentation": instrument,
+        }
+
+    frozen = by_k[str(FROZEN_K)]
+    full = frozen["sim_top_k_plus_real_final"]
+    baseline = frozen["sim_plus_outcome"]
+    instrument = frozen["sim_plus_outcome_plus_instrumentation"]
+    baseline_safe = baseline["false_accept"] == 0 and baseline["false_reject"] == 0
+    instrument_safe = (
+        instrument["false_accept"] == 0 and instrument["false_reject"] == 0
+    )
+    incremental_reduction = (
+        1.0 - instrument["online_h20_hours"] / baseline["online_h20_hours"]
+        if baseline_safe and instrument_safe and baseline["online_h20_hours"] > 0
+        else None
+    )
+    contribution_gate = {
+        "frozen_k": FROZEN_K,
+        "instrument_safe": instrument_safe,
+        "outcome_baseline_safe": baseline_safe,
+        "instrument_regret_at_most_5pct": (
+            instrument["real_regret"] is not None
+            and instrument["real_regret"] <= 0.05
+        ),
+        "instrument_cost_reduction_vs_full_at_least_30pct": (
+            instrument["online_cost_reduction_vs_full"] >= 0.30
+        ),
+        "instrument_cost_reduction_vs_outcome_at_least_20pct": (
+            incremental_reduction is not None and incremental_reduction >= 0.20
+        ),
+        "incremental_reduction_vs_outcome": incremental_reduction,
+    }
+    contribution_gate["passes"] = all(
+        contribution_gate[key]
+        for key in (
+            "instrument_safe",
+            "outcome_baseline_safe",
+            "instrument_regret_at_most_5pct",
+            "instrument_cost_reduction_vs_full_at_least_30pct",
+            "instrument_cost_reduction_vs_outcome_at_least_20pct",
+        )
+    )
+
+    red_flags = []
+    if state["status"] != "complete" or int(state["completed_cells"]) != 6:
+        red_flags.append("pilot_incomplete")
+    if strong["status"] != "PASS" or strong["sanity"]["red_flags"]:
+        red_flags.append("strong_input_invalid")
+    if len(anchors) != 12 or len(candidates) != 6:
+        red_flags.append("unexpected_surface_size")
+    probabilities = [
+        value
+        for anchor in anchors
+        for value in (anchor["baseline_probability"], anchor["instrument_probability"])
+    ]
+    costs = [
+        value
+        for anchor in anchors
+        for value in (
+            anchor["setup_h20_hours"],
+            anchor["full_trial_h20_hours"],
+            anchor["prefix_h20_hours"],
+        )
+    ]
+    if not all(0.0 <= value <= 1.0 for value in probabilities):
+        red_flags.append("probability_out_of_range")
+    if not all(value >= 0.0 and math.isfinite(value) for value in costs):
+        red_flags.append("invalid_cost")
+
+    return {
+        "schema": "fidelity-pilot-e2e-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": "held-out P1 replay; gate diagnostic, not paper-facing evidence",
+        "ranking": [
+            {
+                "rank": rank,
+                "cell": candidate["cell"],
+                "level": candidate["level"],
+                "sim_throughput_req_s_per_gpu": candidate[
+                    "sim_throughput_req_s_per_gpu"
+                ],
+                "real_feasible": candidate["real_feasible"],
+                "real_goodput_req_s_per_gpu": candidate[
+                    "real_goodput_req_s_per_gpu"
+                ],
+            }
+            for rank, candidate in enumerate(candidates, start=1)
+        ],
+        "real_oracle": {
+            "cell": oracle_anchor["cell"],
+            "level": oracle_anchor["level"],
+            "goodput_req_s_per_gpu": oracle_goodput,
+        },
+        "by_k": by_k,
+        "contribution_gate": contribution_gate,
+        "analysis": {
+            "script": str(Path(__file__).resolve()),
+            "script_sha256": sha256_file(Path(__file__).resolve()),
+            "aituner_git_head": git_capture("rev-parse", "HEAD").strip(),
+            "aituner_git_status_short": git_capture("status", "--short"),
+        },
+        "provenance": {
+            "manifest": str(manifest_path.resolve()),
+            "manifest_sha256": sha256_file(manifest_path),
+            "controller_state": str(state_path.resolve()),
+            "controller_state_sha256": sha256_file(state_path),
+            "prior_state": str(prior_state_path.resolve()),
+            "prior_state_sha256": sha256_file(prior_state_path),
+            "strong_metrics": str(strong_path.resolve()),
+            "strong_metrics_sha256": sha256_file(strong_path),
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "anchors": numeric([1 for _ in anchors]),
+            "candidates": numeric([1 for _ in candidates]),
+            "probabilities": numeric(probabilities),
+            "costs_h20_hours": numeric(costs),
+            "invariants": {
+                "anchors_12": len(anchors) == 12,
+                "candidates_6": len(candidates) == 6,
+                "probabilities_bounded": all(
+                    0.0 <= value <= 1.0 for value in probabilities
+                ),
+                "costs_nonnegative": all(value >= 0.0 for value in costs),
+                "per_config_not_all_identical": len(
+                    {candidate["sim_throughput_req_s_per_gpu"] for candidate in candidates}
+                )
+                > 1,
+                "tie_expansion_applied": True,
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--controller-state", type=Path, required=True)
+    parser.add_argument("--prior-state", type=Path, required=True)
+    parser.add_argument("--strong-metrics", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(
+        args.manifest,
+        args.controller_state,
+        args.prior_state,
+        args.strong_metrics,
+    )
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(
+        json.dumps(
+            {
+                "status": result["status"],
+                "red_flags": result["sanity"]["red_flags"],
+                "contribution_gate": result["contribution_gate"],
+            },
+            sort_keys=True,
+        )
+    )
+    if result["status"] != "PASS":
+        raise RuntimeError(result["sanity"]["red_flags"])
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_prefixes.py

@@ -0,0 +1,629 @@
+#!/usr/bin/env python3
+"""Retrospective, leakage-bounded audit of short real-probe prefixes.
+
+The outcome-only and instrumentation-aware models receive the same trial
+prefix.  The latter differs only by Layer-1 engine state.  Existing Phase-6
+request artifacts predate exact completion timestamps, so their completion
+time is reconstructed from arrival + TTFT + token intervals and is explicitly
+marked approximate.  New artifacts use ``completed_elapsed_s`` directly.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable
+
+import numpy as np
+
+from analyze_existing import (
+    DEFAULT_REGULARIZATION,
+    REGULARIZATION_SENSITIVITY,
+    _classification_metrics,
+    _fit_logistic,
+    _group_bootstrap_delta,
+    _mcnemar_exact_p,
+    _sigmoid,
+)
+
+
+SCHEMA = "fidelity-prefix-v1"
+DEFAULT_CUTOFFS = (5.0, 10.0, 15.0, 20.0)
+POLICY_THRESHOLDS = (0.8, 0.9, 0.95)
+
+OUTCOME_FEATURES = (
+    "log_offered_rate_per_gpu",
+    "log2_tp",
+    "log2_max_num_seqs",
+    "admitted_fraction",
+    "completed_over_admitted",
+    "completed_pass_rate",
+    "completed_fail_fraction_of_total",
+    "outstanding_over_admitted",
+    "ttft_max_over_slo_max",
+    "ttft_mean_over_slo_max",
+    "tpot_max_over_slo",
+    "tpot_mean_over_slo",
+    "admitted_input_tokens_mean_over_limit",
+)
+
+INSTRUMENTATION_FEATURES = (
+    "model_steps_per_second",
+    "waiting_mean",
+    "waiting_max",
+    "waiting_nonzero_share",
+    "running_mean",
+    "running_max",
+    "decode_batch_mean",
+    "decode_batch_max",
+    "decode_batch_cv",
+    "kv_usage_mean",
+    "kv_usage_max",
+    "kv_usage_end_minus_start",
+    "graph_none_share",
+    "graph_full_share",
+    "padding_fraction",
+    "prefill_token_fraction",
+    "preemptions",
+)
+
+
+@dataclass(frozen=True)
+class PrefixExample:
+    cell: str
+    anchor: float
+    cutoff_s: float
+    tp: int
+    full_elapsed_s: float
+    feasible: int
+    primary_feasible: int
+    outcome: tuple[float, ...]
+    instrumentation: tuple[float, ...]
+    completion_time_source: str
+
+    @property
+    def remaining_h20_hours(self) -> float:
+        return self.tp * max(0.0, self.full_elapsed_s - self.cutoff_s) / 3600.0
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def numeric(values: Iterable[float | int]) -> dict[str, Any]:
+    array = [float(value) for value in values]
+    return {
+        "n": len(array),
+        "min": min(array) if array else None,
+        "max": max(array) if array else None,
+        "distinct_n": len(set(array)),
+    }
+
+
+def _cv(values: list[float]) -> float:
+    if not values:
+        return 0.0
+    array = np.asarray(values, dtype=np.float64)
+    mean = float(array.mean())
+    return float(array.std(ddof=0) / mean) if mean else 0.0
+
+
+def completion_elapsed_s(request: dict[str, Any]) -> tuple[float | None, str]:
+    exact = request.get("completed_elapsed_s")
+    if exact is not None:
+        value = float(exact)
+        if value < 0 or not math.isfinite(value):
+            raise ValueError(f"invalid completed_elapsed_s={exact}")
+        return value, "exact_monotonic"
+    if not request.get("success"):
+        return None, "unobserved_failure"
+    required = (
+        request.get("arrival_s"),
+        request.get("ttft_ms"),
+        request.get("tpot_ms"),
+        request.get("completion_tokens"),
+    )
+    if any(value is None for value in required):
+        return None, "unobserved_failure"
+    arrival_s, ttft_ms, tpot_ms, completion_tokens = required
+    value = float(arrival_s) + (
+        float(ttft_ms) + max(int(completion_tokens) - 1, 0) * float(tpot_ms)
+    ) / 1000.0
+    if value < 0 or not math.isfinite(value):
+        raise ValueError(f"invalid reconstructed completion time={value}")
+    return value, "reconstructed_from_latency"
+
+
+def _load_jsonl(path: Path, *, require_key: str | None = None) -> list[dict[str, Any]]:
+    records = []
+    with path.open(encoding="utf-8") as source:
+        for line in source:
+            item = json.loads(line)
+            if require_key is None or require_key in item:
+                records.append(item)
+    return records
+
+
+def _anchor_directory(cell_root: Path, anchor: float) -> Path:
+    matches = []
+    for result_path in cell_root.glob("anchor-*/result.json"):
+        payload = json.loads(result_path.read_text(encoding="utf-8"))
+        if math.isclose(float(payload["anchor"]), anchor, rel_tol=0.0, abs_tol=1e-15):
+            matches.append(result_path.parent)
+    if len(matches) != 1:
+        raise ValueError(f"expected one primary directory for anchor {anchor}: {matches}")
+    return matches[0]
+
+
+def _prefix_features(
+    *,
+    primary: dict[str, Any],
+    tp: int,
+    max_num_seqs: int,
+    requests: list[dict[str, Any]],
+    records: list[dict[str, Any]],
+    cutoff_s: float,
+) -> tuple[tuple[float, ...], tuple[float, ...], str]:
+    admitted = [request for request in requests if float(request["arrival_s"]) <= cutoff_s]
+    completed = []
+    sources = set()
+    for request in requests:
+        completed_s, source = completion_elapsed_s(request)
+        if completed_s is None or completed_s > cutoff_s:
+            continue
+        completed.append(request)
+        sources.add(source)
+    if not admitted or not records:
+        raise ValueError("prefix has no admitted requests or Layer-1 records")
+    if any(request not in admitted for request in completed):
+        raise ValueError("completed request was not admitted inside prefix")
+
+    total = len(requests)
+    passed = sum(bool(request["slo_pass"]) for request in completed)
+    ttft = [float(request["ttft_ms"]) for request in completed if request["ttft_ms"] is not None]
+    tpot = [float(request["tpot_ms"]) for request in completed if request["tpot_ms"] is not None]
+    offered_rate = float(primary["selection"]["offered_req_s_per_gpu"])
+    if offered_rate <= 0 or total <= 0:
+        raise ValueError("offered rate and selected request count must be positive")
+
+    outcome = (
+        math.log(offered_rate),
+        math.log2(float(tp)),
+        math.log2(float(max_num_seqs)),
+        len(admitted) / total,
+        len(completed) / len(admitted),
+        passed / max(1, len(completed)),
+        (len(completed) - passed) / total,
+        (len(admitted) - len(completed)) / len(admitted),
+        max(ttft, default=0.0) / 6000.0,
+        float(np.mean(ttft)) / 6000.0 if ttft else 0.0,
+        max(tpot, default=0.0) / 50.0,
+        float(np.mean(tpot)) / 50.0 if tpot else 0.0,
+        float(np.mean([float(request["raw_input_tokens"]) for request in admitted])) / 8192.0,
+    )
+
+    waiting = [float(record["queues"]["waiting"]) for record in records]
+    running = [float(record["queues"]["running"]) for record in records]
+    decode_batch = [float(record["decode_batch_size"]) for record in records]
+    kv_usage = [float(record["kv"]["usage"]) for record in records]
+    graph_modes = [str(record["cudagraph"]["runtime_mode"]) for record in records]
+    bucket_tokens = sum(int(record["cudagraph"]["bucket_tokens"]) for record in records)
+    padding_tokens = sum(int(record["cudagraph"]["padding_tokens"]) for record in records)
+    prefill_tokens = sum(int(record["prefill_tokens"]) for record in records)
+    decode_tokens = sum(int(record["decode_tokens"]) for record in records)
+    instrumentation = (
+        len(records) / cutoff_s,
+        float(np.mean(waiting)),
+        max(waiting),
+        sum(value > 0 for value in waiting) / len(waiting),
+        float(np.mean(running)),
+        max(running),
+        float(np.mean(decode_batch)),
+        max(decode_batch),
+        _cv(decode_batch),
+        float(np.mean(kv_usage)),
+        max(kv_usage),
+        kv_usage[-1] - kv_usage[0],
+        graph_modes.count("NONE") / len(graph_modes),
+        graph_modes.count("FULL") / len(graph_modes),
+        padding_tokens / max(1, bucket_tokens),
+        prefill_tokens / max(1, prefill_tokens + decode_tokens),
+        float(sum(int(record["preemptions"]) for record in records)),
+    )
+    completion_source = "+".join(sorted(sources)) if sources else "none_completed"
+    return outcome, instrumentation, completion_source
+
+
+def build_examples(
+    phase6: dict[str, Any],
+    raw_root: Path,
+    cutoff_s: float,
+) -> list[PrefixExample]:
+    examples = []
+    for cell, cell_result in sorted(phase6["cells"].items()):
+        cell_root = raw_root / cell
+        stream_path = next((cell_root / "opprof").glob("*.jsonl"))
+        stream = _load_jsonl(stream_path, require_key="submit_mono_ns")
+        for anchor in cell_result["anchors"]:
+            primary = anchor["primary"]
+            full_elapsed_s = float(primary["interval"]["elapsed_s"])
+            if full_elapsed_s + 1e-9 < cutoff_s:
+                continue
+            anchor_value = float(primary["anchor"])
+            anchor_root = _anchor_directory(cell_root, anchor_value)
+            requests = _load_jsonl(anchor_root / "requests.jsonl")
+            start_ns = int(primary["interval"]["start_mono_ns"])
+            end_ns = start_ns + int(cutoff_s * 1e9)
+            records = [
+                record
+                for record in stream
+                if record.get("model_executed")
+                and start_ns <= int(record["submit_mono_ns"]) <= end_ns
+            ]
+            outcome, instrumentation, source = _prefix_features(
+                primary=primary,
+                tp=int(cell_result["tp"]),
+                max_num_seqs=int(cell_result["mns"]),
+                requests=requests,
+                records=records,
+                cutoff_s=cutoff_s,
+            )
+            examples.append(
+                PrefixExample(
+                    cell=cell,
+                    anchor=anchor_value,
+                    cutoff_s=cutoff_s,
+                    tp=int(cell_result["tp"]),
+                    full_elapsed_s=full_elapsed_s,
+                    feasible=int(bool(anchor["accepted_feasible"])),
+                    primary_feasible=int(bool(primary["feasible"])),
+                    outcome=outcome,
+                    instrumentation=instrumentation,
+                    completion_time_source=source,
+                )
+            )
+    return examples
+
+
+def grouped_predictions(
+    examples: list[PrefixExample],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> tuple[np.ndarray, np.ndarray, list[str]]:
+    probabilities = []
+    labels = []
+    groups = []
+    for held_out in sorted({example.cell for example in examples}):
+        train = [example for example in examples if example.cell != held_out]
+        test = [example for example in examples if example.cell == held_out]
+
+        def row(example: PrefixExample) -> np.ndarray:
+            values = example.outcome
+            if instrumentation_aware:
+                values += example.instrumentation
+            return np.asarray((1.0, *values), dtype=np.float64)
+
+        x_train = np.stack([row(example) for example in train])
+        x_test = np.stack([row(example) for example in test])
+        y_train = np.asarray([example.feasible for example in train], dtype=np.float64)
+        if len(set(y_train.tolist())) != 2:
+            raise ValueError(f"training fold for {held_out} has a single label")
+        mean = x_train[:, 1:].mean(axis=0)
+        standard_deviation = x_train[:, 1:].std(axis=0)
+        standard_deviation[standard_deviation < 1e-8] = 1.0
+        x_train[:, 1:] = (x_train[:, 1:] - mean) / standard_deviation
+        x_test[:, 1:] = (x_test[:, 1:] - mean) / standard_deviation
+        weights = _fit_logistic(x_train, y_train, regularization)
+        probabilities.extend(_sigmoid(x_test @ weights).tolist())
+        labels.extend(example.feasible for example in test)
+        groups.extend(held_out for _ in test)
+    return (
+        np.asarray(labels, dtype=np.int64),
+        np.asarray(probabilities, dtype=np.float64),
+        groups,
+    )
+
+
+def fit_frozen_model(
+    examples: list[PrefixExample],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> dict[str, Any]:
+    def row(example: PrefixExample) -> np.ndarray:
+        values = example.outcome
+        if instrumentation_aware:
+            values += example.instrumentation
+        return np.asarray((1.0, *values), dtype=np.float64)
+
+    matrix = np.stack([row(example) for example in examples])
+    labels = np.asarray([example.feasible for example in examples], dtype=np.float64)
+    if len(set(labels.tolist())) != 2:
+        raise ValueError("frozen model requires both feasibility labels")
+    mean = matrix[:, 1:].mean(axis=0)
+    standard_deviation = matrix[:, 1:].std(axis=0)
+    standard_deviation[standard_deviation < 1e-8] = 1.0
+    standardized = matrix.copy()
+    standardized[:, 1:] = (standardized[:, 1:] - mean) / standard_deviation
+    weights = _fit_logistic(standardized, labels, regularization)
+    probabilities = _sigmoid(standardized @ weights)
+    names = list(OUTCOME_FEATURES)
+    if instrumentation_aware:
+        names.extend(INSTRUMENTATION_FEATURES)
+    return {
+        "instrumentation_aware": instrumentation_aware,
+        "regularization": regularization,
+        "feature_names": names,
+        "feature_mean": mean.tolist(),
+        "feature_standard_deviation": standard_deviation.tolist(),
+        "weights_with_intercept_first": weights.tolist(),
+        "training_classification": _classification_metrics(labels, probabilities),
+    }
+
+
+def predict_frozen_model(
+    model: dict[str, Any],
+    examples: list[PrefixExample],
+) -> np.ndarray:
+    instrumentation_aware = bool(model["instrumentation_aware"])
+    rows = []
+    for example in examples:
+        values = example.outcome
+        if instrumentation_aware:
+            values += example.instrumentation
+        rows.append((1.0, *values))
+    matrix = np.asarray(rows, dtype=np.float64)
+    mean = np.asarray(model["feature_mean"], dtype=np.float64)
+    standard_deviation = np.asarray(
+        model["feature_standard_deviation"], dtype=np.float64
+    )
+    weights = np.asarray(model["weights_with_intercept_first"], dtype=np.float64)
+    if matrix.shape[1] != len(weights) or matrix.shape[1] - 1 != len(mean):
+        raise ValueError("frozen model feature dimensions do not match examples")
+    matrix[:, 1:] = (matrix[:, 1:] - mean) / standard_deviation
+    return _sigmoid(matrix @ weights)
+
+
+def policy_metrics(
+    examples: list[PrefixExample],
+    labels: np.ndarray,
+    probabilities: np.ndarray,
+    threshold: float,
+) -> dict[str, Any]:
+    accept = probabilities >= threshold
+    reject = probabilities <= 1.0 - threshold
+    decide = accept | reject
+    prediction = accept.astype(np.int64)
+    correct = prediction == labels
+    remaining = np.asarray(
+        [example.remaining_h20_hours for example in examples], dtype=np.float64
+    )
+    full_cost = sum(example.tp * example.full_elapsed_s / 3600.0 for example in examples)
+    saved = float(np.sum(remaining[decide]))
+    correct_saved = float(np.sum(remaining[decide & correct]))
+    invalid_saved = float(np.sum(remaining[decide & ~correct]))
+
+    def describe(mask: np.ndarray) -> list[dict[str, Any]]:
+        return [
+            {
+                "cell": example.cell,
+                "anchor": example.anchor,
+                "label_feasible": bool(label),
+                "probability_feasible": float(probability),
+                "remaining_h20_hours": example.remaining_h20_hours,
+            }
+            for example, label, probability, selected in zip(
+                examples, labels, probabilities, mask
+            )
+            if selected
+        ]
+
+    return {
+        "threshold": threshold,
+        "early_accept": int(np.sum(accept)),
+        "early_reject": int(np.sum(reject)),
+        "abstain_continue_full": int(np.sum(~decide)),
+        "false_accept": int(np.sum(accept & (labels == 0))),
+        "false_reject": int(np.sum(reject & (labels == 1))),
+        "false_accept_examples": describe(accept & (labels == 0)),
+        "false_reject_examples": describe(reject & (labels == 1)),
+        "decision_coverage": float(np.mean(decide)),
+        "full_trial_h20_hours": float(full_cost),
+        "remaining_h20_hours_at_cutoff": float(np.sum(remaining)),
+        "saved_h20_hours_if_decisions_used": saved,
+        "correctly_saved_h20_hours": correct_saved,
+        "invalidly_saved_h20_hours": invalid_saved,
+        "valid_zero_error_policy": bool(np.all(correct[decide])),
+        "valid_cost_reduction_fraction": (
+            correct_saved / full_cost if invalid_saved == 0.0 and full_cost else None
+        ),
+    }
+
+
+def analyze_cutoff(examples: list[PrefixExample]) -> dict[str, Any]:
+    sensitivity = {}
+    headline = None
+    for regularization in REGULARIZATION_SENSITIVITY:
+        labels, outcome_probability, groups = grouped_predictions(
+            examples,
+            instrumentation_aware=False,
+            regularization=regularization,
+        )
+        instrument_labels, instrument_probability, instrument_groups = grouped_predictions(
+            examples,
+            instrumentation_aware=True,
+            regularization=regularization,
+        )
+        if not np.array_equal(labels, instrument_labels) or groups != instrument_groups:
+            raise AssertionError("paired folds or labels differ")
+        if groups != [example.cell for example in examples]:
+            raise AssertionError("prediction order differs from example order")
+        outcome_correct = (outcome_probability >= 0.5) == labels
+        instrument_correct = (instrument_probability >= 0.5) == labels
+        result = {
+            "outcome_only": {
+                "classification": _classification_metrics(labels, outcome_probability),
+                "policies": [
+                    policy_metrics(examples, labels, outcome_probability, threshold)
+                    for threshold in POLICY_THRESHOLDS
+                ],
+            },
+            "instrumentation_aware": {
+                "classification": _classification_metrics(labels, instrument_probability),
+                "policies": [
+                    policy_metrics(examples, labels, instrument_probability, threshold)
+                    for threshold in POLICY_THRESHOLDS
+                ],
+            },
+            "paired_correctness": {
+                "both_correct": int(np.sum(outcome_correct & instrument_correct)),
+                "outcome_only_correct": int(np.sum(outcome_correct & ~instrument_correct)),
+                "instrumentation_only_correct": int(np.sum(~outcome_correct & instrument_correct)),
+                "both_wrong": int(np.sum(~outcome_correct & ~instrument_correct)),
+            },
+            "bootstrap": _group_bootstrap_delta(
+                labels,
+                outcome_probability,
+                instrument_probability,
+                groups,
+            ),
+        }
+        paired = result["paired_correctness"]
+        paired["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+            paired["outcome_only_correct"], paired["instrumentation_only_correct"]
+        )
+        sensitivity[str(regularization)] = result
+        if regularization == DEFAULT_REGULARIZATION:
+            headline = result
+    assert headline is not None
+    labels = [example.feasible for example in examples]
+    return {
+        "examples": len(examples),
+        "cells": len({example.cell for example in examples}),
+        "label_sanity": {
+            **numeric(labels),
+            "positive": sum(labels),
+            "negative": len(labels) - sum(labels),
+            "primary_adjudicated_disagreements": sum(
+                example.feasible != example.primary_feasible for example in examples
+            ),
+        },
+        "completion_time_sources": {
+            source: sum(example.completion_time_source == source for example in examples)
+            for source in sorted({example.completion_time_source for example in examples})
+        },
+        "headline_regularization": DEFAULT_REGULARIZATION,
+        "headline": headline,
+        "regularization_sensitivity": sensitivity,
+        "remaining_h20_hours": numeric(
+            example.remaining_h20_hours for example in examples
+        ),
+    }
+
+
+def analyze(
+    phase6_path: Path,
+    raw_root: Path,
+    cutoffs: tuple[float, ...],
+) -> dict[str, Any]:
+    phase6 = json.loads(phase6_path.read_text(encoding="utf-8"))
+    by_cutoff = {}
+    red_flags = []
+    for cutoff in cutoffs:
+        examples = build_examples(phase6, raw_root, cutoff)
+        if len({example.feasible for example in examples}) != 2:
+            red_flags.append(f"single_label_at_{cutoff:g}s")
+            continue
+        by_cutoff[f"{cutoff:g}"] = analyze_cutoff(examples)
+        if len({example.cell for example in examples}) != 12:
+            red_flags.append(f"incomplete_cells_at_{cutoff:g}s")
+        if not all(
+            math.isfinite(value)
+            for example in examples
+            for value in (*example.outcome, *example.instrumentation)
+        ):
+            red_flags.append(f"nonfinite_features_at_{cutoff:g}s")
+
+    headline_deltas = {
+        cutoff: {
+            "accuracy": (
+                result["headline"]["instrumentation_aware"]["classification"]["accuracy"]
+                - result["headline"]["outcome_only"]["classification"]["accuracy"]
+            ),
+            "brier": (
+                result["headline"]["instrumentation_aware"]["classification"]["brier"]
+                - result["headline"]["outcome_only"]["classification"]["brier"]
+            ),
+        }
+        for cutoff, result in by_cutoff.items()
+    }
+    return {
+        "schema": SCHEMA,
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": (
+            "retrospective single-workload prefix diagnostic; model selection, "
+            "threshold choice, and contribution claims require held-out prospective tasks"
+        ),
+        "estimand": (
+            "2-of-3 adjudicated anchor feasibility from the first primary trial's "
+            "identical short real prefix"
+        ),
+        "split": "leave-one-configuration-cell-out",
+        "model": "same L2 logistic model and folds; instrumentation model appends Layer-1 features",
+        "outcome_features": list(OUTCOME_FEATURES),
+        "instrumentation_features": list(INSTRUMENTATION_FEATURES),
+        "provenance": {
+            "phase6_metrics": str(phase6_path.resolve()),
+            "phase6_metrics_sha256": sha256_file(phase6_path),
+            "raw_root": str(raw_root.resolve()),
+        },
+        "cutoffs_s": list(cutoffs),
+        "cutoffs": by_cutoff,
+        "headline_incremental_deltas": headline_deltas,
+        "decision": {
+            "contribution_established": False,
+            "reason": (
+                "This dataset contains one workload and reconstructed rather than exact request "
+                "completion times. Three TP4 primary trials also disagree with their 2-of-3 "
+                "labels. It can reject a missing-signal premise but cannot establish "
+                "generalization or a paper-facing cost reduction."
+            ),
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "cutoff_count": len(by_cutoff),
+            "invariants": {
+                "cutoffs_positive": all(cutoff > 0 for cutoff in cutoffs),
+                "paired_same_model_family": True,
+                "probabilities_checked_in_unit_interval": True,
+                "full_trial_label_not_used_as_feature": True,
+                "records_strictly_prefix_sliced": True,
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--raw-root", type=Path, required=True)
+    parser.add_argument("--cutoffs", type=float, nargs="+", default=DEFAULT_CUTOFFS)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(args.phase6_metrics, args.raw_root, tuple(args.cutoffs))
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    print(json.dumps({"status": result["status"], "output": str(args.output)}, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_strong_baseline.py

@@ -0,0 +1,298 @@
+#!/usr/bin/env python3
+"""Audit telemetry against a simulator-aware outcome calibration baseline.
+
+This is a retrospective headroom check.  It strengthens the earlier
+outcome-only baseline by giving both nested models the same per-anchor
+Frontier throughput and SLO predictions.  The only additional inputs to the
+larger model are real engine Layer-1 features.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from analyze_existing import (
+    DEFAULT_REGULARIZATION,
+    REGULARIZATION_SENSITIVITY,
+    _classification_metrics,
+    _fit_logistic,
+    _group_bootstrap_delta,
+    _mcnemar_exact_p,
+    _sigmoid,
+)
+from analyze_prefixes import (
+    INSTRUMENTATION_FEATURES,
+    OUTCOME_FEATURES,
+    PrefixExample,
+    build_examples,
+    numeric,
+    policy_metrics,
+    sha256_file,
+)
+
+
+SIMULATOR_FEATURES = (
+    "log_sim_completed_throughput_per_gpu",
+    "sim_slo_pass_rate",
+    "sim_slo_feasible",
+)
+
+
+def load_simulator_features(raw_root: Path) -> tuple[dict[tuple[str, float], tuple[float, ...]], str]:
+    features: dict[tuple[str, float], tuple[float, ...]] = {}
+    digest = hashlib.sha256()
+    paths = sorted(raw_root.glob("*/trial-0001/run_manifest.json"))
+    for manifest_path in paths:
+        manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+        run = manifest["run"]
+        if run["mode"] != "frozen-calibrated":
+            continue
+        scorer_path = manifest_path.parent / "scorer_output.json"
+        scorer = json.loads(scorer_path.read_text(encoding="utf-8"))
+        key = (str(run["cell_id"]), float(run["sampling_u"]))
+        if key in features:
+            raise ValueError(f"duplicate frozen simulator run: {key}")
+        throughput = float(scorer["throughput_requests_per_second_per_gpu"])
+        pass_rate = float(scorer["slo"]["pass_rate"])
+        if throughput <= 0 or not 0.0 <= pass_rate <= 1.0:
+            raise ValueError(f"invalid simulator output: {key}")
+        features[key] = (
+            math.log(throughput),
+            pass_rate,
+            float(bool(scorer["slo"]["feasible"])),
+        )
+        for path in (manifest_path, scorer_path):
+            digest.update(str(path.relative_to(raw_root)).encode())
+            digest.update(path.read_bytes())
+    return features, digest.hexdigest()
+
+
+def simulator_row(
+    example: PrefixExample,
+    features: dict[tuple[str, float], tuple[float, ...]],
+) -> tuple[float, ...]:
+    matches = [
+        values
+        for (cell, anchor), values in features.items()
+        if cell == example.cell
+        and math.isclose(anchor, example.anchor, rel_tol=0.0, abs_tol=1e-12)
+    ]
+    if len(matches) != 1:
+        raise ValueError(
+            f"expected one simulator match for {example.cell}/{example.anchor}: {len(matches)}"
+        )
+    return matches[0]
+
+
+def grouped_predictions(
+    examples: list[PrefixExample],
+    simulator: dict[tuple[str, float], tuple[float, ...]],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> tuple[np.ndarray, np.ndarray, list[str]]:
+    probabilities: list[float] = []
+    labels: list[int] = []
+    groups: list[str] = []
+    for held_out in sorted({example.cell for example in examples}):
+        train = [example for example in examples if example.cell != held_out]
+        test = [example for example in examples if example.cell == held_out]
+
+        def row(example: PrefixExample) -> np.ndarray:
+            values = example.outcome + simulator_row(example, simulator)
+            if instrumentation_aware:
+                values += example.instrumentation
+            return np.asarray((1.0, *values), dtype=np.float64)
+
+        x_train = np.stack([row(example) for example in train])
+        x_test = np.stack([row(example) for example in test])
+        y_train = np.asarray([example.feasible for example in train], dtype=np.float64)
+        mean = x_train[:, 1:].mean(axis=0)
+        standard_deviation = x_train[:, 1:].std(axis=0)
+        standard_deviation[standard_deviation < 1e-8] = 1.0
+        x_train[:, 1:] = (x_train[:, 1:] - mean) / standard_deviation
+        x_test[:, 1:] = (x_test[:, 1:] - mean) / standard_deviation
+        weights = _fit_logistic(x_train, y_train, regularization)
+        probabilities.extend(_sigmoid(x_test @ weights).tolist())
+        labels.extend(example.feasible for example in test)
+        groups.extend(held_out for _ in test)
+    return (
+        np.asarray(labels, dtype=np.int64),
+        np.asarray(probabilities, dtype=np.float64),
+        groups,
+    )
+
+
+def analyze(
+    phase6_path: Path,
+    phase6_raw_root: Path,
+    simulator_raw_root: Path,
+    simulator_metrics_path: Path,
+) -> dict[str, Any]:
+    phase6 = json.loads(phase6_path.read_text(encoding="utf-8"))
+    examples = build_examples(phase6, phase6_raw_root, 5.0)
+    simulator, simulator_raw_sha256 = load_simulator_features(simulator_raw_root)
+    red_flags = []
+    try:
+        matched = [simulator_row(example, simulator) for example in examples]
+    except ValueError as error:
+        matched = []
+        red_flags.append(str(error))
+
+    sensitivity = {}
+    if matched:
+        for regularization in REGULARIZATION_SENSITIVITY:
+            labels, baseline_probability, groups = grouped_predictions(
+                examples,
+                simulator,
+                instrumentation_aware=False,
+                regularization=regularization,
+            )
+            instrument_labels, instrument_probability, instrument_groups = grouped_predictions(
+                examples,
+                simulator,
+                instrumentation_aware=True,
+                regularization=regularization,
+            )
+            if not np.array_equal(labels, instrument_labels) or groups != instrument_groups:
+                raise AssertionError("nested baseline folds differ")
+            baseline_correct = (baseline_probability >= 0.5) == labels
+            instrument_correct = (instrument_probability >= 0.5) == labels
+            paired = {
+                "both_correct": int(np.sum(baseline_correct & instrument_correct)),
+                "sim_outcome_only_correct": int(
+                    np.sum(baseline_correct & ~instrument_correct)
+                ),
+                "instrumentation_only_correct": int(
+                    np.sum(~baseline_correct & instrument_correct)
+                ),
+                "both_wrong": int(np.sum(~baseline_correct & ~instrument_correct)),
+            }
+            paired["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+                paired["sim_outcome_only_correct"],
+                paired["instrumentation_only_correct"],
+            )
+            sensitivity[str(regularization)] = {
+                "sim_plus_outcome": {
+                    "classification": _classification_metrics(labels, baseline_probability),
+                    "policy_0p95": policy_metrics(
+                        examples, labels, baseline_probability, 0.95
+                    ),
+                },
+                "sim_plus_outcome_plus_instrumentation": {
+                    "classification": _classification_metrics(labels, instrument_probability),
+                    "policy_0p95": policy_metrics(
+                        examples, labels, instrument_probability, 0.95
+                    ),
+                },
+                "paired_correctness": paired,
+                "group_bootstrap": _group_bootstrap_delta(
+                    labels,
+                    baseline_probability,
+                    instrument_probability,
+                    groups,
+                ),
+            }
+
+    headline = sensitivity.get(str(DEFAULT_REGULARIZATION))
+    simulator_pass_rates = [row[1] for row in matched]
+    labels = [example.feasible for example in examples]
+    if len(examples) != 37:
+        red_flags.append("examples_not_37")
+    if len(simulator) != 92:
+        red_flags.append("frozen_simulator_runs_not_92")
+    if len(set(labels)) != 2:
+        red_flags.append("single_label")
+    if matched and not all(0.0 <= value <= 1.0 for value in simulator_pass_rates):
+        red_flags.append("simulator_pass_rate_out_of_range")
+
+    return {
+        "schema": "fidelity-strong-baseline-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": "retrospective one-task headroom audit; not contribution evidence",
+        "comparison": (
+            "same 5-second prefix, folds, logistic family, regularization, and frozen "
+            "Frontier outputs; the only nested difference is real Layer-1 engine state"
+        ),
+        "features": {
+            "shared_outcome": list(OUTCOME_FEATURES),
+            "shared_simulator": list(SIMULATOR_FEATURES),
+            "instrumentation_only": list(INSTRUMENTATION_FEATURES),
+        },
+        "headline_regularization": DEFAULT_REGULARIZATION,
+        "headline": headline,
+        "regularization_sensitivity": sensitivity,
+        "provenance": {
+            "phase6_metrics": str(phase6_path.resolve()),
+            "phase6_metrics_sha256": sha256_file(phase6_path),
+            "phase6_raw_root": str(phase6_raw_root.resolve()),
+            "simulator_metrics": str(simulator_metrics_path.resolve()),
+            "simulator_metrics_sha256": sha256_file(simulator_metrics_path),
+            "simulator_raw_root": str(simulator_raw_root.resolve()),
+            "frozen_simulator_manifest_scorer_set_sha256": simulator_raw_sha256,
+        },
+        "decision": {
+            "contribution_established": False,
+            "prospective_requirement": (
+                "repeat sim+outcome versus sim+outcome+instrumentation on complete held-out tasks"
+            ),
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "examples": numeric([1 for _ in examples]),
+            "labels": {
+                **numeric(labels),
+                "positive": sum(labels),
+                "negative": len(labels) - sum(labels),
+            },
+            "matched_simulator_pass_rate": numeric(simulator_pass_rates),
+            "frozen_simulator_runs": len(simulator),
+            "invariants": {
+                "all_examples_matched_once": len(matched) == len(examples),
+                "same_nested_folds": True,
+                "simulator_ratios_bounded": all(
+                    0.0 <= value <= 1.0 for value in simulator_pass_rates
+                ),
+                "labels_not_identical": len(set(labels)) == 2,
+                "per_config_results_not_all_identical": len(set(simulator_pass_rates)) > 1,
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--phase6-raw-root", type=Path, required=True)
+    parser.add_argument("--simulator-raw-root", type=Path, required=True)
+    parser.add_argument("--simulator-metrics", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(
+        args.phase6_metrics,
+        args.phase6_raw_root,
+        args.simulator_raw_root,
+        args.simulator_metrics,
+    )
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(
+        json.dumps(
+            {
+                "status": result["status"],
+                "output": str(args.output),
+                "red_flags": result["sanity"]["red_flags"],
+            },
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/analyze_strong_pilot.py

@@ -0,0 +1,507 @@
+#!/usr/bin/env python3
+"""Exploratory P1 audit against the strengthened simulator-aware baseline.
+
+P1 was already running when the strong baseline was added, so this script is
+not paper-facing prospective evidence.  It trains only on the historical
+Phase-6 task and evaluates the exact P1 primary probes.  Both nested models
+receive identical Frontier predictions; engine telemetry is the sole feature
+difference.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import subprocess
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+
+from analyze_existing import (
+    DEFAULT_REGULARIZATION,
+    REGULARIZATION_SENSITIVITY,
+    _classification_metrics,
+    _fit_logistic,
+    _mcnemar_exact_p,
+    _sigmoid,
+)
+from analyze_pilot import build_pilot_examples, campaign_gpu_accounting
+from analyze_prefixes import (
+    INSTRUMENTATION_FEATURES,
+    OUTCOME_FEATURES,
+    PrefixExample,
+    build_examples,
+    numeric,
+    policy_metrics,
+    sha256_file,
+)
+from analyze_strong_baseline import (
+    SIMULATOR_FEATURES,
+    load_simulator_features,
+    simulator_row,
+)
+
+
+AITUNER_ROOT = Path(__file__).resolve().parents[2]
+
+
+def git_capture(*arguments: str) -> str:
+    return subprocess.run(
+        ["git", "-C", str(AITUNER_ROOT), *arguments],
+        check=True,
+        text=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    ).stdout
+
+
+def load_pilot_simulator(
+    path: Path,
+) -> tuple[dict[tuple[str, str], tuple[float, ...]], list[str]]:
+    payload = json.loads(path.read_text(encoding="utf-8"))
+    red_flags = []
+    if payload.get("status") != "PASS":
+        red_flags.append("pilot_simulator_not_pass")
+    features: dict[tuple[str, str], tuple[float, ...]] = {}
+    for item in payload.get("results", []):
+        key = (str(item["cell"]), str(item["role"]))
+        if key in features:
+            red_flags.append(f"duplicate_pilot_simulator_{key[0]}_{key[1]}")
+            continue
+        scorer = item["scorer"]
+        throughput = float(scorer["throughput_requests_per_second_per_gpu"])
+        pass_rate = float(scorer["slo"]["pass_rate"])
+        if throughput <= 0:
+            red_flags.append(f"nonpositive_pilot_simulator_throughput_{key[0]}_{key[1]}")
+        if not 0.0 <= pass_rate <= 1.0:
+            red_flags.append(f"pilot_simulator_ratio_out_of_range_{key[0]}_{key[1]}")
+        features[key] = (
+            math.log(throughput),
+            pass_rate,
+            float(bool(scorer["slo"]["feasible"])),
+        )
+    if len(features) != 12:
+        red_flags.append("pilot_simulator_entries_not_12")
+    return features, red_flags
+
+
+def fit_model(
+    examples: list[PrefixExample],
+    simulator: list[tuple[float, ...]],
+    *,
+    instrumentation_aware: bool,
+    regularization: float,
+) -> dict[str, Any]:
+    rows = []
+    for example, simulator_features in zip(examples, simulator):
+        values = example.outcome + simulator_features
+        if instrumentation_aware:
+            values += example.instrumentation
+        rows.append((1.0, *values))
+    matrix = np.asarray(rows, dtype=np.float64)
+    labels = np.asarray([example.feasible for example in examples], dtype=np.float64)
+    mean = matrix[:, 1:].mean(axis=0)
+    standard_deviation = matrix[:, 1:].std(axis=0)
+    standard_deviation[standard_deviation < 1e-8] = 1.0
+    standardized = matrix.copy()
+    standardized[:, 1:] = (standardized[:, 1:] - mean) / standard_deviation
+    weights = _fit_logistic(standardized, labels, regularization)
+    return {
+        "instrumentation_aware": instrumentation_aware,
+        "regularization": regularization,
+        "feature_mean": mean,
+        "feature_standard_deviation": standard_deviation,
+        "weights": weights,
+    }
+
+
+def predict_model(
+    model: dict[str, Any],
+    examples: list[PrefixExample],
+    simulator: list[tuple[float, ...]],
+) -> np.ndarray:
+    rows = []
+    for example, simulator_features in zip(examples, simulator):
+        values = example.outcome + simulator_features
+        if model["instrumentation_aware"]:
+            values += example.instrumentation
+        rows.append((1.0, *values))
+    matrix = np.asarray(rows, dtype=np.float64)
+    matrix[:, 1:] = (
+        matrix[:, 1:] - model["feature_mean"]
+    ) / model["feature_standard_deviation"]
+    return _sigmoid(matrix @ model["weights"])
+
+
+def covariate_shift(
+    training_examples: list[PrefixExample],
+    training_simulator: list[tuple[float, ...]],
+    pilot_examples: list[PrefixExample],
+    pilot_simulator: list[tuple[float, ...]],
+    *,
+    instrumentation_aware: bool,
+) -> dict[str, Any]:
+    def matrix(
+        examples: list[PrefixExample], simulator: list[tuple[float, ...]]
+    ) -> np.ndarray:
+        rows = []
+        for example, simulator_features in zip(examples, simulator):
+            values = example.outcome + simulator_features
+            if instrumentation_aware:
+                values += example.instrumentation
+            rows.append(values)
+        return np.asarray(rows, dtype=np.float64)
+
+    training = matrix(training_examples, training_simulator)
+    pilot = matrix(pilot_examples, pilot_simulator)
+    mean = training.mean(axis=0)
+    standard_deviation = training.std(axis=0)
+    standard_deviation[standard_deviation < 1e-8] = 1.0
+    absolute_z = np.abs((pilot - mean) / standard_deviation)
+    names = [*OUTCOME_FEATURES, *SIMULATOR_FEATURES]
+    if instrumentation_aware:
+        names.extend(INSTRUMENTATION_FEATURES)
+    return {
+        "values": numeric(absolute_z.ravel().tolist()),
+        "count_gt_3": int(np.sum(absolute_z > 3.0)),
+        "count_gt_5": int(np.sum(absolute_z > 5.0)),
+        "total_feature_values": int(absolute_z.size),
+        "per_feature_max_abs_z": {
+            name: float(value) for name, value in zip(names, absolute_z.max(axis=0))
+        },
+    }
+
+
+def comparison(
+    training_examples: list[PrefixExample],
+    training_simulator: list[tuple[float, ...]],
+    pilot_examples: list[PrefixExample],
+    pilot_simulator: list[tuple[float, ...]],
+    regularization: float,
+) -> dict[str, Any]:
+    labels = np.asarray([example.feasible for example in pilot_examples], dtype=np.int64)
+    baseline_model = fit_model(
+        training_examples,
+        training_simulator,
+        instrumentation_aware=False,
+        regularization=regularization,
+    )
+    instrument_model = fit_model(
+        training_examples,
+        training_simulator,
+        instrumentation_aware=True,
+        regularization=regularization,
+    )
+    baseline_probability = predict_model(
+        baseline_model, pilot_examples, pilot_simulator
+    )
+    instrument_probability = predict_model(
+        instrument_model, pilot_examples, pilot_simulator
+    )
+    baseline_correct = (baseline_probability >= 0.5) == labels
+    instrument_correct = (instrument_probability >= 0.5) == labels
+    paired = {
+        "both_correct": int(np.sum(baseline_correct & instrument_correct)),
+        "sim_outcome_only_correct": int(
+            np.sum(baseline_correct & ~instrument_correct)
+        ),
+        "instrumentation_only_correct": int(
+            np.sum(~baseline_correct & instrument_correct)
+        ),
+        "both_wrong": int(np.sum(~baseline_correct & ~instrument_correct)),
+    }
+    paired["mcnemar_exact_two_sided_p"] = _mcnemar_exact_p(
+        paired["sim_outcome_only_correct"], paired["instrumentation_only_correct"]
+    )
+    return {
+        "sim_plus_outcome": {
+            "classification": _classification_metrics(labels, baseline_probability),
+            "policy_0p95": policy_metrics(
+                pilot_examples, labels, baseline_probability, 0.95
+            ),
+            "probability": baseline_probability.tolist(),
+        },
+        "sim_plus_outcome_plus_instrumentation": {
+            "classification": _classification_metrics(labels, instrument_probability),
+            "policy_0p95": policy_metrics(
+                pilot_examples, labels, instrument_probability, 0.95
+            ),
+            "probability": instrument_probability.tolist(),
+        },
+        "paired_correctness": paired,
+    }
+
+
+def analyze(
+    phase6_path: Path,
+    phase6_raw_root: Path,
+    training_simulator_root: Path,
+    pilot_manifest_path: Path,
+    pilot_run_root: Path,
+    pilot_simulator_path: Path,
+    prior_state_paths: tuple[Path, ...] = (),
+) -> dict[str, Any]:
+    phase6 = json.loads(phase6_path.read_text(encoding="utf-8"))
+    pilot_manifest = json.loads(pilot_manifest_path.read_text(encoding="utf-8"))
+    pilot_state_path = pilot_run_root / "controller-state.json"
+    pilot_state = json.loads(pilot_state_path.read_text(encoding="utf-8"))
+    gpu_accounting = campaign_gpu_accounting(
+        pilot_state_path, prior_state_paths
+    )
+    training_examples = build_examples(phase6, phase6_raw_root, 5.0)
+    training_simulator_map, training_simulator_sha256 = load_simulator_features(
+        training_simulator_root
+    )
+    training_simulator = [
+        simulator_row(example, training_simulator_map)
+        for example in training_examples
+    ]
+    pilot_examples, pilot_details, red_flags = build_pilot_examples(
+        pilot_manifest, pilot_run_root, 5.0
+    )
+    pilot_simulator_map, simulator_red_flags = load_pilot_simulator(
+        pilot_simulator_path
+    )
+    red_flags.extend(simulator_red_flags)
+    pilot_simulator = []
+    for example, detail in zip(pilot_examples, pilot_details):
+        role = f"{detail['level']}1"
+        key = (example.cell, role)
+        if key not in pilot_simulator_map:
+            red_flags.append(f"missing_pilot_simulator_{example.cell}_{role}")
+            pilot_simulator.append((0.0, 0.0, 0.0))
+        else:
+            pilot_simulator.append(pilot_simulator_map[key])
+
+    sensitivity = {}
+    if not red_flags:
+        for regularization in REGULARIZATION_SENSITIVITY:
+            sensitivity[str(regularization)] = comparison(
+                training_examples,
+                training_simulator,
+                pilot_examples,
+                pilot_simulator,
+                regularization,
+            )
+    headline = sensitivity.get(str(DEFAULT_REGULARIZATION))
+    labels = [example.feasible for example in pilot_examples]
+    simulator_pass_rates = [row[1] for row in pilot_simulator]
+    simulator_labels = [int(row[2]) for row in pilot_simulator]
+    if len(training_examples) != 37:
+        red_flags.append("training_examples_not_37")
+    if len(pilot_examples) != 12:
+        red_flags.append("pilot_examples_not_12")
+    if len(set(labels)) != 2:
+        red_flags.append("pilot_single_label")
+    if len(set(simulator_pass_rates)) <= 1:
+        red_flags.append("pilot_simulator_results_identical")
+    if pilot_state.get("status") != "complete" or int(
+        pilot_state.get("completed_cells", 0)
+    ) != 6:
+        red_flags.append("pilot_campaign_incomplete")
+    if any(detail["actual_timestamped_outcomes"] == 0 for detail in pilot_details):
+        red_flags.append("pilot_no_exact_request_timestamps")
+    all_cell_validations = all(
+        cell.get("validation") is not None
+        and all(cell["validation"]["invariants"].values())
+        for cell in pilot_state.get("cells", {}).values()
+    )
+    if not all_cell_validations:
+        red_flags.append("pilot_cell_validation_failed")
+    if not all(gpu_accounting["invariants"].values()):
+        red_flags.append("pilot_hard_cap_exceeded")
+    covariate_diagnostics = {
+        "sim_plus_outcome": covariate_shift(
+            training_examples,
+            training_simulator,
+            pilot_examples,
+            pilot_simulator,
+            instrumentation_aware=False,
+        ),
+        "sim_plus_outcome_plus_instrumentation": covariate_shift(
+            training_examples,
+            training_simulator,
+            pilot_examples,
+            pilot_simulator,
+            instrumentation_aware=True,
+        ),
+    }
+
+    if headline is None:
+        decision = {
+            "strong_incremental_gate": False,
+            "reason": "analysis red flag prevented nested comparison",
+        }
+    else:
+        baseline_policy = headline["sim_plus_outcome"]["policy_0p95"]
+        instrument_policy = headline[
+            "sim_plus_outcome_plus_instrumentation"
+        ]["policy_0p95"]
+        baseline_errors = baseline_policy["false_accept"] + baseline_policy["false_reject"]
+        instrument_errors = (
+            instrument_policy["false_accept"] + instrument_policy["false_reject"]
+        )
+        baseline_reduction = baseline_policy["valid_cost_reduction_fraction"]
+        instrument_reduction = instrument_policy["valid_cost_reduction_fraction"]
+        reduction_delta = (
+            instrument_reduction - baseline_reduction
+            if baseline_reduction is not None and instrument_reduction is not None
+            else None
+        )
+        per_lambda_safe_and_better = []
+        for item in sensitivity.values():
+            baseline = item["sim_plus_outcome"]["policy_0p95"]
+            instrument = item["sim_plus_outcome_plus_instrumentation"]["policy_0p95"]
+            base_errors = baseline["false_accept"] + baseline["false_reject"]
+            inst_errors = instrument["false_accept"] + instrument["false_reject"]
+            base_reduction = baseline["valid_cost_reduction_fraction"]
+            inst_reduction = instrument["valid_cost_reduction_fraction"]
+            per_lambda_safe_and_better.append(
+                inst_errors == 0
+                and inst_errors <= base_errors
+                and base_reduction is not None
+                and inst_reduction is not None
+                and inst_reduction > base_reduction
+            )
+        decision = {
+            "strong_incremental_gate": bool(
+                not red_flags
+                and instrument_errors == 0
+                and instrument_errors <= baseline_errors
+                and reduction_delta is not None
+                and reduction_delta >= 0.15
+            ),
+            "regularization_robust": all(per_lambda_safe_and_better),
+            "valid_cost_reduction_fraction_delta": reduction_delta,
+            "scope": "exploratory task; may choose P2 design but cannot establish contribution",
+        }
+
+    return {
+        "schema": "fidelity-strong-pilot-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "scope": (
+            "post-amendment exploratory P1 audit; strong model was not frozen before "
+            "partial P1 outcomes, so this is not prospective contribution evidence"
+        ),
+        "features": {
+            "shared_outcome": list(OUTCOME_FEATURES),
+            "shared_simulator": list(SIMULATOR_FEATURES),
+            "instrumentation_only": list(INSTRUMENTATION_FEATURES),
+        },
+        "headline_regularization": DEFAULT_REGULARIZATION,
+        "headline": headline,
+        "regularization_sensitivity": sensitivity,
+        "simulator_only": {
+            "classification": _classification_metrics(
+                np.asarray(labels, dtype=np.int64),
+                np.asarray(simulator_labels, dtype=np.float64),
+            )
+            if labels
+            else None,
+            "predicted_feasible": simulator_labels,
+        },
+        "pilot_examples": [
+            {
+                **detail,
+                "sim_completed_throughput_per_gpu": math.exp(simulator[0]),
+                "sim_slo_pass_rate": simulator[1],
+                "sim_slo_feasible": bool(simulator[2]),
+            }
+            for detail, simulator in zip(pilot_details, pilot_simulator)
+        ],
+        "covariate_shift_diagnostic": covariate_diagnostics,
+        "decision": decision,
+        "gpu": {
+            "primary_attempt_h20_hours": pilot_state["gpu_hours_total"],
+            **gpu_accounting,
+        },
+        "analysis": {
+            "script": str(Path(__file__).resolve()),
+            "script_sha256": sha256_file(Path(__file__).resolve()),
+            "aituner_git_head": git_capture("rev-parse", "HEAD").strip(),
+            "aituner_git_status_short": git_capture("status", "--short"),
+        },
+        "provenance": {
+            "phase6_metrics": str(phase6_path.resolve()),
+            "phase6_metrics_sha256": sha256_file(phase6_path),
+            "phase6_raw_root": str(phase6_raw_root.resolve()),
+            "training_simulator_root": str(training_simulator_root.resolve()),
+            "training_simulator_manifest_scorer_set_sha256": training_simulator_sha256,
+            "pilot_manifest": str(pilot_manifest_path.resolve()),
+            "pilot_manifest_sha256": sha256_file(pilot_manifest_path),
+            "pilot_run_root": str(pilot_run_root.resolve()),
+            "pilot_controller_state": str(pilot_state_path.resolve()),
+            "pilot_controller_state_sha256": sha256_file(pilot_state_path),
+            "pilot_simulator": str(pilot_simulator_path.resolve()),
+            "pilot_simulator_sha256": sha256_file(pilot_simulator_path),
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "training_examples": numeric([1 for _ in training_examples]),
+            "pilot_labels": {
+                **numeric(labels),
+                "positive": sum(labels),
+                "negative": len(labels) - sum(labels),
+            },
+            "pilot_simulator_pass_rate": numeric(simulator_pass_rates),
+            "invariants": {
+                "training_examples_37": len(training_examples) == 37,
+                "pilot_examples_12": len(pilot_examples) == 12,
+                "pilot_cells_6": len({example.cell for example in pilot_examples}) == 6,
+                "pilot_both_labels": len(set(labels)) == 2,
+                "simulator_ratios_bounded": all(
+                    0.0 <= value <= 1.0 for value in simulator_pass_rates
+                ),
+                "per_config_not_all_identical": len(set(simulator_pass_rates)) > 1,
+                "all_prefixes_exact_monotonic": all(
+                    example.completion_time_source in {"exact_monotonic", "none_completed"}
+                    for example in pilot_examples
+                ),
+                "all_cell_validations": all_cell_validations,
+                "gpu_cost_nonnegative_below_cap": (
+                    all(gpu_accounting["invariants"].values())
+                ),
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--phase6-raw-root", type=Path, required=True)
+    parser.add_argument("--training-simulator-root", type=Path, required=True)
+    parser.add_argument("--pilot-manifest", type=Path, required=True)
+    parser.add_argument("--pilot-run-root", type=Path, required=True)
+    parser.add_argument("--pilot-simulator", type=Path, required=True)
+    parser.add_argument("--prior-state", type=Path, action="append", default=[])
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    result = analyze(
+        args.phase6_metrics,
+        args.phase6_raw_root,
+        args.training_simulator_root,
+        args.pilot_manifest,
+        args.pilot_run_root,
+        args.pilot_simulator,
+        tuple(args.prior_state),
+    )
+    args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    print(
+        json.dumps(
+            {
+                "status": result["status"],
+                "red_flags": result["sanity"]["red_flags"],
+                "decision": result["decision"],
+            },
+            sort_keys=True,
+        )
+    )
+    if result["status"] != "PASS":
+        raise RuntimeError(result["sanity"]["red_flags"])
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/freeze_models.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+"""Freeze the training-task prefix models before prospective GPU work."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from analyze_prefixes import (
+    DEFAULT_REGULARIZATION,
+    POLICY_THRESHOLDS,
+    build_examples,
+    fit_frozen_model,
+    sha256_file,
+)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase6-metrics", type=Path, required=True)
+    parser.add_argument("--prefix-metrics", type=Path, required=True)
+    parser.add_argument("--raw-root", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+
+    cutoff_s = 5.0
+    threshold = 0.95
+    if threshold not in POLICY_THRESHOLDS:
+        raise AssertionError("frozen threshold is outside audited policy thresholds")
+    phase6 = json.loads(args.phase6_metrics.read_text(encoding="utf-8"))
+    examples = build_examples(phase6, args.raw_root, cutoff_s)
+    payload = {
+        "schema": "fidelity-prefix-model-v1",
+        "status": "FROZEN_BEFORE_PROSPECTIVE_RUN",
+        "cutoff_s": cutoff_s,
+        "accept_probability": threshold,
+        "reject_probability": 1.0 - threshold,
+        "regularization": DEFAULT_REGULARIZATION,
+        "label": "same-placement 2-of-3 adjudicated anchor feasibility",
+        "training_split_role": "historical training only; never headline test",
+        "training_examples": [
+            {
+                "cell": example.cell,
+                "anchor": example.anchor,
+                "label_feasible": bool(example.feasible),
+                "primary_feasible": bool(example.primary_feasible),
+                "completion_time_source": example.completion_time_source,
+            }
+            for example in examples
+        ],
+        "models": {
+            "outcome_only": fit_frozen_model(
+                examples,
+                instrumentation_aware=False,
+                regularization=DEFAULT_REGULARIZATION,
+            ),
+            "instrumentation_aware": fit_frozen_model(
+                examples,
+                instrumentation_aware=True,
+                regularization=DEFAULT_REGULARIZATION,
+            ),
+        },
+        "provenance": {
+            "phase6_metrics": str(args.phase6_metrics.resolve()),
+            "phase6_metrics_sha256": sha256_file(args.phase6_metrics),
+            "prefix_metrics": str(args.prefix_metrics.resolve()),
+            "prefix_metrics_sha256": sha256_file(args.prefix_metrics),
+            "raw_root": str(args.raw_root.resolve()),
+        },
+        "sanity": {
+            "n": len(examples),
+            "positive": sum(example.feasible for example in examples),
+            "negative": sum(not example.feasible for example in examples),
+            "cells": len({example.cell for example in examples}),
+            "invariants": {
+                "n_37": len(examples) == 37,
+                "cells_12": len({example.cell for example in examples}) == 12,
+                "both_labels": len({example.feasible for example in examples}) == 2,
+                "cutoff_5s": cutoff_s == 5.0,
+                "threshold_0.95": threshold == 0.95,
+            },
+        },
+    }
+    if not all(payload["sanity"]["invariants"].values()):
+        raise RuntimeError(f"model freeze invariants failed: {payload['sanity']}")
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    print(json.dumps({"status": payload["status"], "output": str(args.output)}))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/frozen-models.json

@@ -0,0 +1,515 @@
+{
+  "accept_probability": 0.95,
+  "cutoff_s": 5.0,
+  "label": "same-placement 2-of-3 adjudicated anchor feasibility",
+  "models": {
+    "instrumentation_aware": {
+      "feature_mean": [
+        0.8984976998643891,
+        0.8378378378378378,
+        4.324324324324325,
+        0.07552086023066117,
+        0.6758807403968693,
+        0.9459459459459459,
+        0.0,
+        0.3241192596031305,
+        0.04468545442770093,
+        0.025590516908533558,
+        0.23873649352596996,
+        0.1943716628122394,
+        0.4321792125178198,
+        112.70270270270272,
+        0.21087752102856197,
+        0.918918918918919,
+        0.055470351361483664,
+        4.904239530899751,
+        10.162162162162161,
+        4.822982150502539,
+        10.135135135135135,
+        0.43557131397798415,
+        0.031387890158936414,
+        0.05804311436894179,
+        0.03298678556030958,
+        0.030437119300455177,
+        0.9503065396705037,
+        0.07127076398319926,
+        0.6234198543231205,
+        0.0
+      ],
+      "feature_names": [
+        "log_offered_rate_per_gpu",
+        "log2_tp",
+        "log2_max_num_seqs",
+        "admitted_fraction",
+        "completed_over_admitted",
+        "completed_pass_rate",
+        "completed_fail_fraction_of_total",
+        "outstanding_over_admitted",
+        "ttft_max_over_slo_max",
+        "ttft_mean_over_slo_max",
+        "tpot_max_over_slo",
+        "tpot_mean_over_slo",
+        "admitted_input_tokens_mean_over_limit",
+        "model_steps_per_second",
+        "waiting_mean",
+        "waiting_max",
+        "waiting_nonzero_share",
+        "running_mean",
+        "running_max",
+        "decode_batch_mean",
+        "decode_batch_max",
+        "decode_batch_cv",
+        "kv_usage_mean",
+        "kv_usage_max",
+        "kv_usage_end_minus_start",
+        "graph_none_share",
+        "graph_full_share",
+        "padding_fraction",
+        "prefill_token_fraction",
+        "preemptions"
+      ],
+      "feature_standard_deviation": [
+        0.2953332526155246,
+        0.8546696378833459,
+        1.1402715194448103,
+        0.006588255148989237,
+        0.2751217635728275,
+        0.22612433149569594,
+        1.0,
+        0.27512176357282747,
+        0.048292427420964075,
+        0.02574874589991541,
+        0.1635381690436309,
+        0.14098674719611365,
+        0.02516276437103069,
+        61.39272994412853,
+        0.7234949448561444,
+        2.198013579605131,
+        0.18326586413988316,
+        2.4542471212960844,
+        6.08726412391018,
+        2.4074006634033043,
+        6.067913672185017,
+        0.12556414020947543,
+        0.03256962310836033,
+        0.054049610008010444,
+        0.048321850100969746,
+        0.04298231458641556,
+        0.041906068064246155,
+        0.08212268757576466,
+        0.4089385238422411,
+        1.0
+      ],
+      "instrumentation_aware": true,
+      "regularization": 1.0,
+      "training_classification": {
+        "accuracy": 0.972972972972973,
+        "balanced_accuracy": 0.9444444444444444,
+        "brier": 0.02820726479488704,
+        "confusion": {
+          "false_negative": 0,
+          "false_positive": 1,
+          "true_negative": 8,
+          "true_positive": 28
+        },
+        "log_loss": 0.11247563885308659
+      },
+      "weights_with_intercept_first": [
+        2.109507425802979,
+        -0.8372240489271802,
+        -0.2476229678897366,
+        0.18172257646801393,
+        -0.07076358054975332,
+        0.3035586906752765,
+        0.08500005412355496,
+        -7.754818242684634e-26,
+        -0.3035586906752766,
+        0.4014234393196892,
+        0.513218716194957,
+        -0.35161457106287,
+        0.10558147889556725,
+        0.5674345291616134,
+        0.15895995157114373,
+        -0.4274260624362057,
+        -0.048791959001756195,
+        -0.37221380985270663,
+        -0.35527537277290255,
+        0.20582736797173468,
+        -0.35837576944545413,
+        0.2342062515631318,
+        0.45071068059490843,
+        0.3326948315186803,
+        0.2698892549960913,
+        0.017868065865726347,
+        -0.1540209080477302,
+        0.3412427440368233,
+        0.5831011876762794,
+        -0.583920360300169,
+        0.0
+      ]
+    },
+    "outcome_only": {
+      "feature_mean": [
+        0.8984976998643891,
+        0.8378378378378378,
+        4.324324324324325,
+        0.07552086023066117,
+        0.6758807403968693,
+        0.9459459459459459,
+        0.0,
+        0.3241192596031305,
+        0.04468545442770093,
+        0.025590516908533558,
+        0.23873649352596996,
+        0.1943716628122394,
+        0.4321792125178198
+      ],
+      "feature_names": [
+        "log_offered_rate_per_gpu",
+        "log2_tp",
+        "log2_max_num_seqs",
+        "admitted_fraction",
+        "completed_over_admitted",
+        "completed_pass_rate",
+        "completed_fail_fraction_of_total",
+        "outstanding_over_admitted",
+        "ttft_max_over_slo_max",
+        "ttft_mean_over_slo_max",
+        "tpot_max_over_slo",
+        "tpot_mean_over_slo",
+        "admitted_input_tokens_mean_over_limit"
+      ],
+      "feature_standard_deviation": [
+        0.2953332526155246,
+        0.8546696378833459,
+        1.1402715194448103,
+        0.006588255148989237,
+        0.2751217635728275,
+        0.22612433149569594,
+        1.0,
+        0.27512176357282747,
+        0.048292427420964075,
+        0.02574874589991541,
+        0.1635381690436309,
+        0.14098674719611365,
+        0.02516276437103069
+      ],
+      "instrumentation_aware": false,
+      "regularization": 1.0,
+      "training_classification": {
+        "accuracy": 0.9459459459459459,
+        "balanced_accuracy": 0.8888888888888888,
+        "brier": 0.051887373873176545,
+        "confusion": {
+          "false_negative": 0,
+          "false_positive": 2,
+          "true_negative": 7,
+          "true_positive": 28
+        },
+        "log_loss": 0.184988719119571
+      },
+      "weights_with_intercept_first": [
+        1.8996338126233983,
+        -1.1536861934230125,
+        -0.3806404559018098,
+        0.5901136731733696,
+        0.022432085012851908,
+        0.5805554730881304,
+        0.25786307099613026,
+        -8.077935669463161e-27,
+        -0.5805554730881304,
+        -0.15413292402348447,
+        0.0986842306063204,
+        -0.5181573573074624,
+        0.06283513013708956,
+        0.911619634884147
+      ]
+    }
+  },
+  "provenance": {
+    "phase6_metrics": "/home/gahow/phd/aituner/runs/opprof-phase6/phase6/metrics.json",
+    "phase6_metrics_sha256": "290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16",
+    "prefix_metrics": "/home/gahow/phd/aituner/runs/fidelity-headroom/prefix-metrics.json",
+    "prefix_metrics_sha256": "cda821bcde1ae8427507aa4f03a1c116ccc7f7b8b717f73ca587bee3670a0340",
+    "raw_root": "/home/gahow/phd/aituner/runs/opprof-phase6/phase6/solo-authoritative/cells"
+  },
+  "regularization": 1.0,
+  "reject_probability": 0.050000000000000044,
+  "sanity": {
+    "cells": 12,
+    "invariants": {
+      "both_labels": true,
+      "cells_12": true,
+      "cutoff_5s": true,
+      "n_37": true,
+      "threshold_0.95": true
+    },
+    "n": 37,
+    "negative": 9,
+    "positive": 28
+  },
+  "schema": "fidelity-prefix-model-v1",
+  "status": "FROZEN_BEFORE_PROSPECTIVE_RUN",
+  "training_examples": [
+    {
+      "anchor": 0.24609375,
+      "cell": "tp1_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.2421875,
+      "cell": "tp1_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.24609375,
+      "cell": "tp1_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns32",
+      "completion_time_source": "none_completed",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.2421875,
+      "cell": "tp1_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.24609375,
+      "cell": "tp1_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns64",
+      "completion_time_source": "none_completed",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.21875,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.2265625,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.23046875,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.234375,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.25,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp1_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.4921875,
+      "cell": "tp2_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.49609375,
+      "cell": "tp2_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp2_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.75,
+      "cell": "tp2_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.75390625,
+      "cell": "tp2_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.5,
+      "cell": "tp2_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.75,
+      "cell": "tp2_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.4921875,
+      "cell": "tp2_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.49609375,
+      "cell": "tp2_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.033182214016,
+      "cell": "tp4_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.033717411016,
+      "cell": "tp4_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns16",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.033717411016,
+      "cell": "tp4_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns32",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.033717411016,
+      "cell": "tp4_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": false
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns64",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.016055910008,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.016591107009,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.017126304009,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": true,
+      "primary_feasible": true
+    },
+    {
+      "anchor": 0.034252608017,
+      "cell": "tp4_mns8",
+      "completion_time_source": "reconstructed_from_latency",
+      "label_feasible": false,
+      "primary_feasible": false
+    }
+  ],
+  "training_split_role": "historical training only; never headline test"
+}

runs/fidelity-headroom/pilot_controller.py

@@ -0,0 +1,461 @@
+#!/usr/bin/env python3
+"""Serialized dash0 controller for the exact-timestamp prefix pilot."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shlex
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+
+HERE = Path(__file__).resolve().parent
+PHASE6 = HERE.parent / "opprof-phase6"
+sys.path.insert(0, str(PHASE6))
+
+import opprof_phase6_controller as base  # noqa: E402
+
+
+ORDER = (
+    "tp1_mns8",
+    "tp1_mns64",
+    "tp2_mns8",
+    "tp2_mns64",
+    "tp4_mns16",
+    "tp4_mns64",
+)
+CELL_ESTIMATE_H20_HOURS = {1: 0.20, 2: 0.40, 4: 0.80}
+SAFETY_H20_HOURS = 0.20
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    base.atomic_json(path, payload)
+
+
+def wait_all_idle(timeout_s: float = 30.0) -> None:
+    deadline = time.monotonic() + timeout_s
+    last_error: Exception | None = None
+    while time.monotonic() < deadline:
+        try:
+            base.assert_all_idle()
+            return
+        except RuntimeError as error:
+            last_error = error
+            time.sleep(1.0)
+    raise last_error or RuntimeError("GPU idle timeout")
+
+
+def configure_base(args: argparse.Namespace, manifest: dict[str, Any]) -> None:
+    base.WORKDIR = args.run_root.parent
+    base.RUN_ROOT = args.run_root
+    base.STATE = args.run_root / "controller-state.json"
+    base.SOURCE = args.vllm_source
+    base.VENV = args.venv
+    base.AITUNER = args.aituner_root
+    base.MODEL = args.model
+    base.CLIENT = args.client
+    base.GPU_LIMIT = float(manifest["execution"]["hard_cap_h20_hours"])
+    base.MARKER = "fidelity-prefix-pilot-20260714"
+    base.CELLS = {
+        cell: {"tp": int(config["tp"]), "mns": int(config["mns"])}
+        for cell, config in manifest["cells"].items()
+    }
+
+
+def load_state(path: Path, hard_cap: float) -> dict[str, Any]:
+    if path.exists():
+        return json.loads(path.read_text(encoding="utf-8"))
+    return {
+        "schema": "fidelity-prefix-pilot-state-v1",
+        "status": "initialized",
+        "hard_cap_h20_hours": hard_cap,
+        "gpu_hours_total": 0.0,
+        "completed_cells": 0,
+        "cells": {},
+        "failures": [],
+        "started_at": time.time(),
+    }
+
+
+def save_state(path: Path, state: dict[str, Any]) -> None:
+    atomic_json(path, state)
+
+
+def append_echo(run_root: Path, line: str) -> None:
+    run_root.mkdir(parents=True, exist_ok=True)
+    with (run_root / "launch-echo.log").open("a", encoding="utf-8") as target:
+        target.write(line + "\n")
+    print(line, flush=True)
+
+
+def remaining_projection(manifest: dict[str, Any], index: int) -> float:
+    return sum(
+        CELL_ESTIMATE_H20_HOURS[int(manifest["cells"][cell]["tp"])]
+        for cell in ORDER[index:]
+    ) + SAFETY_H20_HOURS
+
+
+def start_server(
+    *,
+    cell: str,
+    index: int,
+    run_root: Path,
+) -> dict[str, Any]:
+    config = base.CELLS[cell]
+    gpus = tuple(range(int(config["tp"])))
+    cell_root = run_root / "cells" / cell
+    cell_root.mkdir(parents=True, exist_ok=True)
+    port = 8900 + index
+    command = base.server_command(cell, gpus, port)
+    with (cell_root / "commands.log").open("a", encoding="utf-8") as log:
+        log.write(f"SERVER {shlex.join(command)}\n")
+    server_log = (cell_root / "server.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update(
+        {
+            "CUDA_VISIBLE_DEVICES": ",".join(map(str, gpus)),
+            "VLLM_OPPROF_DIR": str(cell_root / "opprof"),
+            "OPPROF_PHASE6_MARKER": base.MARKER,
+            "AITUNER_ROOT": str(base.AITUNER),
+            "HF_HUB_OFFLINE": "1",
+            "TRANSFORMERS_OFFLINE": "1",
+            "PYTHONUNBUFFERED": "1",
+        }
+    )
+    server = subprocess.Popen(
+        command,
+        cwd=base.SOURCE,
+        env=environment,
+        stdout=server_log,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    base.OWNED_PGIDS.add(server.pid)
+    return {
+        "cell": cell,
+        "gpus": gpus,
+        "port": port,
+        "dir": cell_root,
+        "server": server,
+        "server_handle": server_log,
+        "spawned_at": time.time(),
+        "results": [],
+    }
+
+
+def selection_for(
+    manifest: dict[str, Any], cell: str, role: str
+) -> tuple[str, dict[str, Any]]:
+    level = "low" if role == "burnin" or role.startswith("low") else "high"
+    return level, manifest["cells"][cell]["targets"][level]["selections"][role]
+
+
+def client_command(
+    entry: dict[str, Any],
+    *,
+    role: str,
+    selection: dict[str, Any],
+    output: Path,
+    warmup: bool,
+) -> list[str]:
+    config = base.CELLS[entry["cell"]]
+    return [
+        "taskset",
+        "-c",
+        base.cpu_mask(entry["gpus"]),
+        str(base.VENV / "bin/python"),
+        str(base.CLIENT),
+        "warmup" if warmup else "run-anchor",
+        "--study",
+        str(selection["study"]),
+        "--cell",
+        entry["cell"],
+        "--anchor",
+        str(selection["anchor"]),
+        "--tp",
+        str(config["tp"]),
+        "--mns",
+        str(config["mns"]),
+        "--base-url",
+        f"http://127.0.0.1:{entry['port']}",
+        "--result-dir",
+        str(output),
+    ]
+
+
+def run_client(
+    *,
+    entry: dict[str, Any],
+    role: str,
+    selection: dict[str, Any],
+    output: Path,
+    state: dict[str, Any],
+    warmup: bool = False,
+) -> dict[str, Any]:
+    command = client_command(
+        entry, role=role, selection=selection, output=output, warmup=warmup
+    )
+    with (entry["dir"] / "commands.log").open("a", encoding="utf-8") as log:
+        log.write(f"CLIENT role={role} {shlex.join(command)}\n")
+    handle = (output.parent / f"{output.name}.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update({"AITUNER_ROOT": str(base.AITUNER), "PYTHONUNBUFFERED": "1"})
+    process = subprocess.Popen(
+        command,
+        cwd=base.WORKDIR,
+        env=environment,
+        stdout=handle,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    deadline = time.monotonic() + 180.0
+    try:
+        while process.poll() is None:
+            if time.monotonic() > deadline:
+                process.terminate()
+                raise TimeoutError(f"client timeout: {entry['cell']} {role}")
+            if entry["server"].poll() is not None:
+                raise RuntimeError(f"server exited during {entry['cell']} {role}")
+            base.assert_no_other_compute()
+            if state["gpu_hours_total"] + base.live_gpu_hours([entry]) >= base.GPU_LIMIT:
+                process.terminate()
+                raise RuntimeError("pilot H20-hour hard cap reached")
+            time.sleep(1.0)
+    finally:
+        handle.close()
+    if process.returncode:
+        raise RuntimeError(
+            f"client failed: cell={entry['cell']} role={role} rc={process.returncode}"
+        )
+    result = json.loads((output / "result.json").read_text(encoding="utf-8"))
+    validate_result_selection(
+        result=result,
+        selection=selection,
+        cell=entry["cell"],
+        role=role,
+        warmup=warmup,
+    )
+    entry["results"].append(
+        {"anchor": float(selection["anchor"]), "dir": str(output), "kind": result["kind"]}
+    )
+    return result
+
+
+def validate_result_selection(
+    *,
+    result: dict[str, Any],
+    selection: dict[str, Any],
+    cell: str,
+    role: str,
+    warmup: bool,
+) -> None:
+    if warmup:
+        if result["kind"] != "warmup" or int(result["selection"]["count"]) != 16:
+            raise RuntimeError(f"invalid warmup selection: {cell} {role}")
+        for key in ("warmup_16", "warmup_exact_16", "warmup_long"):
+            if not result["invariants"].get(key, False):
+                raise RuntimeError(f"warmup invariant {key} failed: {cell} {role}")
+        return
+
+    if int(result["selection"]["count"]) != int(selection["selected_count"]):
+        raise RuntimeError(f"selection count mismatch: {cell} {role}")
+    for key in (
+        "request_id_order_sha256",
+        "arrival_order_sha256",
+        "raw_length_order_sha256",
+    ):
+        manifest_key = (
+            "input_length_order_sha256" if key == "raw_length_order_sha256" else key
+        )
+        if result["selection"][key] != selection[manifest_key]:
+            raise RuntimeError(f"selection hash mismatch {key}: {cell} {role}")
+
+
+def execute_cell(
+    *,
+    index: int,
+    cell: str,
+    manifest: dict[str, Any],
+    run_root: Path,
+    state_path: Path,
+    state: dict[str, Any],
+) -> None:
+    if state["cells"].get(cell, {}).get("status") == "complete":
+        return
+    projection = remaining_projection(manifest, index)
+    if state["gpu_hours_total"] + projection > base.GPU_LIMIT:
+        state["status"] = "budget_projection_stop"
+        state["budget_stop"] = {
+            "before_cell": cell,
+            "spent_h20_hours": state["gpu_hours_total"],
+            "remaining_projection_h20_hours": projection,
+            "hard_cap_h20_hours": base.GPU_LIMIT,
+        }
+        save_state(state_path, state)
+        raise RuntimeError(f"projected pilot cost exceeds hard cap before {cell}")
+
+    config = manifest["cells"][cell]
+    echo = (
+        f"PILOT_CELL_ECHO cell={cell} tp={config['tp']} mns={config['mns']} "
+        f"gpus=0-{int(config['tp']) - 1} workload={manifest['source']['window_id']} "
+        f"roles=burnin+low1/high1/low2/high2/low3/high3 "
+        f"spent_h20h={state['gpu_hours_total']:.6f} "
+        f"remaining_projection_h20h={projection:.3f} cap_h20h={base.GPU_LIMIT:.1f} "
+        f"manifest={run_root / 'pilot-manifest.json'}"
+    )
+    append_echo(run_root, echo)
+    wait_all_idle()
+    cell_state = {
+        "status": "starting",
+        "tp": int(config["tp"]),
+        "mns": int(config["mns"]),
+        "started_at": time.time(),
+        "runs": [],
+    }
+    state["status"] = "running"
+    state["cells"][cell] = cell_state
+    save_state(state_path, state)
+    entry = start_server(cell=cell, index=index, run_root=run_root)
+    failure: Exception | None = None
+    try:
+        base.wait_ready(entry)
+        _level, burnin = selection_for(manifest, cell, "burnin")
+        cell_state["status"] = "warmup"
+        save_state(state_path, state)
+        warmup = run_client(
+            entry=entry,
+            role="burnin",
+            selection=burnin,
+            output=entry["dir"] / "warmup",
+            state=state,
+            warmup=True,
+        )
+        cell_state["warmup"] = {
+            "exact_output_count": warmup["exact_output_count"],
+            "long_gt4096": warmup["selection"]["long_gt4096"],
+        }
+        cell_state["status"] = "burnin"
+        save_state(state_path, state)
+        burnin_result = run_client(
+            entry=entry,
+            role="burnin",
+            selection=burnin,
+            output=entry["dir"] / "burnin",
+            state=state,
+        )
+        cell_state["burnin"] = {
+            "pass_rate": burnin_result["pass_rate"],
+            "feasible": burnin_result["feasible"],
+        }
+        role_order = manifest["execution"][
+            "even_cell_order" if index % 2 == 0 else "odd_cell_order"
+        ]
+        cell_state["status"] = "measured"
+        cell_state["role_order"] = role_order
+        save_state(state_path, state)
+        for role in role_order:
+            level, selection = selection_for(manifest, cell, role)
+            result = run_client(
+                entry=entry,
+                role=role,
+                selection=selection,
+                output=entry["dir"] / f"{level}-rep{role[-1]}",
+                state=state,
+            )
+            cell_state["runs"].append(
+                {
+                    "role": role,
+                    "level": level,
+                    "anchor": selection["anchor"],
+                    "selected_count": selection["selected_count"],
+                    "pass_rate": result["pass_rate"],
+                    "feasible": result["feasible"],
+                    "elapsed_s": result["interval"]["elapsed_s"],
+                }
+            )
+            save_state(state_path, state)
+        cell_state["status"] = "stopping"
+        save_state(state_path, state)
+    except Exception as error:  # noqa: BLE001
+        failure = error
+    finally:
+        try:
+            base.stop_entry(entry)
+        except Exception as error:  # noqa: BLE001
+            failure = failure or error
+        time.sleep(2.0)
+        try:
+            wait_all_idle()
+        except Exception as error:  # noqa: BLE001
+            failure = failure or error
+
+    cell_hours = base.live_gpu_hours([entry])
+    state["gpu_hours_total"] += cell_hours
+    cell_state["gpu_hours"] = cell_hours
+    if failure is not None:
+        cell_state["status"] = "failed"
+        cell_state["failure"] = repr(failure)
+        state["status"] = "failed"
+        state["failures"].append({"cell": cell, "failure": repr(failure)})
+        save_state(state_path, state)
+        raise failure
+    validation = base.validate_cell(entry)
+    cell_state["validation"] = validation
+    cell_state["status"] = "complete"
+    cell_state["completed_at"] = time.time()
+    state["completed_cells"] += 1
+    save_state(state_path, state)
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("--manifest", type=Path, required=True)
+    result.add_argument("--run-root", type=Path, required=True)
+    result.add_argument("--aituner-root", type=Path, required=True)
+    result.add_argument("--vllm-source", type=Path, required=True)
+    result.add_argument("--venv", type=Path, required=True)
+    result.add_argument("--model", type=Path, required=True)
+    result.add_argument("--client", type=Path, required=True)
+    return result
+
+
+def main() -> None:
+    args = parser().parse_args()
+    manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
+    if manifest["status"] != "PASS":
+        raise RuntimeError("pilot manifest did not pass preflight")
+    args.run_root.mkdir(parents=True, exist_ok=True)
+    copied_manifest = args.run_root / "pilot-manifest.json"
+    if not copied_manifest.exists():
+        atomic_json(copied_manifest, manifest)
+    configure_base(args, manifest)
+    state_path = args.run_root / "controller-state.json"
+    state = load_state(state_path, base.GPU_LIMIT)
+    state["status"] = "running"
+    save_state(state_path, state)
+    for index, cell in enumerate(ORDER):
+        execute_cell(
+            index=index,
+            cell=cell,
+            manifest=manifest,
+            run_root=args.run_root,
+            state_path=state_path,
+            state=state,
+        )
+    state["status"] = "complete"
+    state["completed_at"] = time.time()
+    save_state(state_path, state)
+    print(json.dumps({
+        "status": state["status"],
+        "completed_cells": state["completed_cells"],
+        "gpu_hours_total": state["gpu_hours_total"],
+    }, sort_keys=True))
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/prepare_pilot.py

@@ -0,0 +1,355 @@
+#!/usr/bin/env python3
+"""Materialize session-disjoint pilot repeats and freeze attainable anchors.
+
+The private outputs retain prompt text and stay on the experiment host.  The
+public manifest contains only aggregate counts, hashes, paths, and parameters.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+import os
+import sys
+from pathlib import Path
+from typing import Any
+
+
+AITUNER_ROOT = Path(os.environ.get("AITUNER_ROOT", Path(__file__).resolve().parents[2]))
+sys.path.insert(0, str(AITUNER_ROOT / "src"))
+
+from aituner.spec import load_study_spec  # noqa: E402
+from aituner.trace import load_trace_requests, select_requests_for_threshold  # noqa: E402
+
+
+ROLES = ("burnin", "low1", "high1", "low2", "high2", "low3", "high3")
+CELLS = {
+    "tp1_mns8": {"tp": 1, "mns": 8, "frontier_req_s_gpu": 2.3833333333333333},
+    "tp1_mns64": {"tp": 1, "mns": 64, "frontier_req_s_gpu": 2.3833333333333333},
+    "tp2_mns8": {"tp": 2, "mns": 8, "frontier_req_s_gpu": 2.2416666666666667},
+    "tp2_mns64": {"tp": 2, "mns": 64, "frontier_req_s_gpu": 2.3},
+    "tp4_mns16": {"tp": 4, "mns": 16, "frontier_req_s_gpu": 2.5},
+    "tp4_mns64": {"tp": 4, "mns": 64, "frontier_req_s_gpu": 2.5},
+}
+TARGET_MULTIPLIERS = {"low": 0.85, "high": 1.25}
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_suffix(path.suffix + ".tmp")
+    tmp.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    os.replace(tmp, path)
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def order_hash(values: list[str]) -> str:
+    return hashlib.sha256("\n".join(values).encode()).hexdigest()
+
+
+def resolve_source_trace(windows_path: Path, window_id: str) -> tuple[dict[str, Any], Path]:
+    payload = json.loads(windows_path.read_text(encoding="utf-8"))
+    for window in payload["windows"]:
+        if window["window_id"] != window_id:
+            continue
+        trace = Path(window["trace_file"])
+        if not trace.is_absolute():
+            trace = (windows_path.parent / trace).resolve()
+        return window, trace
+    raise ValueError(f"window not found: {window_id}")
+
+
+def materialize_bands(
+    source_trace: Path,
+    source_window: dict[str, Any],
+    private_root: Path,
+) -> tuple[Path, dict[str, Any]]:
+    traces_root = private_root / "traces"
+    traces_root.mkdir(parents=True, exist_ok=True)
+    temporary = {role: traces_root / f".{role}.jsonl.tmp" for role in ROLES}
+    final = {role: traces_root / f"{role}.jsonl" for role in ROLES}
+    handles = {role: temporary[role].open("w", encoding="utf-8") for role in ROLES}
+    stats = {
+        role: {
+            "rows": 0,
+            "sum_input_tokens": 0,
+            "min_timestamp": None,
+            "max_timestamp": None,
+        }
+        for role in ROLES
+    }
+    try:
+        with source_trace.open(encoding="utf-8") as source:
+            for line_number, line in enumerate(source):
+                row = json.loads(line)
+                value = float(row["sampling_u"])
+                if not 0.0 <= value <= 1.0:
+                    raise ValueError(f"sampling_u outside [0,1] at line {line_number}")
+                band = min(len(ROLES) - 1, int(value * len(ROLES)))
+                role = ROLES[band]
+                remapped = value * len(ROLES) - band
+                row["sampling_u"] = min(remapped, math.nextafter(1.0, 0.0))
+                row["fidelity_pilot_band"] = role
+                handles[role].write(json.dumps(row, ensure_ascii=False) + "\n")
+                timestamp = float(row["timestamp"])
+                item = stats[role]
+                item["rows"] += 1
+                item["sum_input_tokens"] += int(row.get("input_length") or 0)
+                item["min_timestamp"] = (
+                    timestamp if item["min_timestamp"] is None
+                    else min(float(item["min_timestamp"]), timestamp)
+                )
+                item["max_timestamp"] = (
+                    timestamp if item["max_timestamp"] is None
+                    else max(float(item["max_timestamp"]), timestamp)
+                )
+    finally:
+        for handle in handles.values():
+            handle.close()
+    for role in ROLES:
+        os.replace(temporary[role], final[role])
+        stats[role]["sha256"] = sha256_file(final[role])
+        stats[role]["bytes"] = final[role].stat().st_size
+
+    windows = []
+    for role in ROLES:
+        window = dict(source_window)
+        window["window_id"] = f"fidelity_pilot_{role}"
+        window["trace_file"] = f"traces/{role}.jsonl"
+        window["num_requests"] = stats[role]["rows"]
+        window["sum_input_length"] = stats[role]["sum_input_tokens"]
+        window["sampling_strategy"] = "session_uniform_seven_disjoint_bands_remapped"
+        window["fidelity_pilot_role"] = role
+        windows.append(window)
+    private_windows = private_root / "windows.json"
+    atomic_json(
+        private_windows,
+        {
+            "schema": "fidelity-pilot-private-windows-v1",
+            "roles": list(ROLES),
+            "windows": windows,
+        },
+    )
+    return private_windows, stats
+
+
+def write_studies(
+    *,
+    base_primary: Path,
+    base_tp4: Path,
+    private_windows: Path,
+    private_root: Path,
+) -> dict[str, dict[str, Path]]:
+    bases = {
+        "primary": json.loads(base_primary.read_text(encoding="utf-8")),
+        "tp4": json.loads(base_tp4.read_text(encoding="utf-8")),
+    }
+    result: dict[str, dict[str, Path]] = {}
+    for role in ROLES:
+        result[role] = {}
+        for tier, base in bases.items():
+            payload = json.loads(json.dumps(base))
+            payload["study_id"] = f"fidelity-prefix-pilot-{role}-{tier}"
+            payload["hardware"]["host_candidates"] = ["dash0"]
+            payload["engine"]["engine_version"] = "0.24.1.dev3+opprof"
+            payload["trace"]["windows_path"] = str(private_windows)
+            payload["trace"]["window_id"] = f"fidelity_pilot_{role}"
+            path = private_root / "studies" / f"{role}-{tier}.json"
+            atomic_json(path, payload)
+            result[role][tier] = path
+    return result
+
+
+def attainable_anchor(requests: list[Any], target_count: int) -> tuple[float, list[Any]]:
+    ordered = sorted(float(request.sampling_u) for request in requests)
+    if not ordered:
+        raise ValueError("no requests after study filtering")
+    candidate_indices = sorted({
+        max(0, min(len(ordered) - 1, target_count - 1)),
+        max(0, min(len(ordered) - 1, target_count)),
+    })
+    candidates = []
+    for index in candidate_indices:
+        anchor = ordered[index]
+        selected = select_requests_for_threshold(requests, threshold=anchor)
+        candidates.append((abs(len(selected) - target_count), len(selected), anchor, selected))
+    _error, _count, anchor, selected = min(candidates, key=lambda item: (item[0], item[1]))
+    return anchor, selected
+
+
+def selected_record(selected: list[Any], *, tp: int, duration_s: float) -> dict[str, Any]:
+    return {
+        "anchor": max(float(request.sampling_u) for request in selected),
+        "selected_count": len(selected),
+        "offered_req_s": len(selected) / duration_s,
+        "offered_req_s_per_gpu": len(selected) / duration_s / tp,
+        "request_id_order_sha256": order_hash([request.row_id for request in selected]),
+        "arrival_order_sha256": order_hash([f"{request.arrival_s:.12f}" for request in selected]),
+        "input_length_order_sha256": order_hash(
+            [str(request.prompt_tokens_hint) for request in selected]
+        ),
+    }
+
+
+def build_manifest(
+    *,
+    studies: dict[str, dict[str, Path]],
+    private_windows: Path,
+    band_stats: dict[str, Any],
+    source_trace: Path,
+    source_windows: Path,
+    source_window_id: str,
+) -> dict[str, Any]:
+    loaded = {}
+    durations = {}
+    for role, tiers in studies.items():
+        loaded[role] = {}
+        for tier, path in tiers.items():
+            study = load_study_spec(path)
+            window, requests = load_trace_requests(study, study_spec_path=path)
+            loaded[role][tier] = requests
+            durations[role] = float(window.window_end - window.window_start)
+
+    cells = {}
+    all_hashes = []
+    for cell, config in CELLS.items():
+        tp = int(config["tp"])
+        tier = "tp4" if tp == 4 else "primary"
+        targets = {}
+        for level, multiplier in TARGET_MULTIPLIERS.items():
+            target_rate = float(config["frontier_req_s_gpu"]) * multiplier
+            target_count = round(target_rate * durations["low1"] * tp)
+            roles = [
+                role
+                for role in ROLES
+                if role.startswith(level) or (level == "low" and role == "burnin")
+            ]
+            selections = {}
+            for role in roles:
+                anchor, selected = attainable_anchor(loaded[role][tier], target_count)
+                record = selected_record(selected, tp=tp, duration_s=durations[role])
+                record["anchor"] = anchor
+                record["study"] = str(studies[role][tier])
+                selections[role] = record
+                all_hashes.append(record["request_id_order_sha256"])
+            targets[level] = {
+                "multiplier": multiplier,
+                "target_req_s_per_gpu": target_rate,
+                "target_count": target_count,
+                "selections": selections,
+            }
+        cells[cell] = {**config, "targets": targets}
+
+    red_flags = []
+    for cell, config in cells.items():
+        for level, target in config["targets"].items():
+            if not target["selections"]:
+                red_flags.append(f"missing_{cell}_{level}")
+            for selection in target["selections"].values():
+                if selection["selected_count"] <= 0:
+                    red_flags.append(f"empty_{cell}_{level}")
+    per_cell_distinct = {}
+    for cell, config in cells.items():
+        hashes = [
+            selection["request_id_order_sha256"]
+            for target in config["targets"].values()
+            for selection in target["selections"].values()
+        ]
+        per_cell_distinct[cell] = len(hashes) == len(set(hashes))
+        if not per_cell_distinct[cell]:
+            red_flags.append(f"session_bands_overlap_{cell}")
+    return {
+        "schema": "fidelity-prefix-pilot-manifest-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "source": {
+            "windows": str(source_windows),
+            "window_id": source_window_id,
+            "trace": str(source_trace),
+            "trace_sha256": sha256_file(source_trace),
+        },
+        "private": {
+            "windows": str(private_windows),
+            "windows_sha256": sha256_file(private_windows),
+            "band_stats": band_stats,
+            "studies": {
+                role: {tier: str(path) for tier, path in tiers.items()}
+                for role, tiers in studies.items()
+            },
+        },
+        "roles": list(ROLES),
+        "cells": cells,
+        "execution": {
+            "cutoff_s": 5.0,
+            "replicates_per_level": 3,
+            "label": "2-of-3 session-disjoint repetitions",
+            "even_cell_order": ["low1", "high1", "high2", "low2", "low3", "high3"],
+            "odd_cell_order": ["high1", "low1", "low2", "high2", "high3", "low3"],
+            "hard_cap_h20_hours": 3.5,
+        },
+        "sanity": {
+            "red_flags": red_flags,
+            "n_cells": len(cells),
+            "n_roles": len(ROLES),
+            "selected_sets": len(all_hashes),
+            "distinct_selected_sets": len(set(all_hashes)),
+            "per_cell_selected_sets_distinct": per_cell_distinct,
+            "invariants": {
+                "cells_6": len(cells) == 6,
+                "roles_7": len(ROLES) == 7,
+                "band_rows_nonzero": all(stats["rows"] > 0 for stats in band_stats.values()),
+                "session_bands_disjoint_per_cell": all(per_cell_distinct.values()),
+            },
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--source-windows", type=Path, required=True)
+    parser.add_argument("--source-window-id", default="chat_w20260312_1000")
+    parser.add_argument("--base-primary-study", type=Path, required=True)
+    parser.add_argument("--base-tp4-study", type=Path, required=True)
+    parser.add_argument("--private-root", type=Path, required=True)
+    parser.add_argument("--public-manifest", type=Path, required=True)
+    args = parser.parse_args()
+
+    source_window, source_trace = resolve_source_trace(
+        args.source_windows, args.source_window_id
+    )
+    private_windows, band_stats = materialize_bands(
+        source_trace, source_window, args.private_root
+    )
+    studies = write_studies(
+        base_primary=args.base_primary_study,
+        base_tp4=args.base_tp4_study,
+        private_windows=private_windows,
+        private_root=args.private_root,
+    )
+    manifest = build_manifest(
+        studies=studies,
+        private_windows=private_windows,
+        band_stats=band_stats,
+        source_trace=source_trace,
+        source_windows=args.source_windows,
+        source_window_id=args.source_window_id,
+    )
+    atomic_json(args.public_manifest, manifest)
+    print(json.dumps({
+        "status": manifest["status"],
+        "manifest": str(args.public_manifest),
+        "sanity": manifest["sanity"],
+    }, sort_keys=True))
+    if manifest["status"] != "PASS":
+        raise RuntimeError(f"pilot preflight failed: {manifest['sanity']['red_flags']}")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/prepare_pilot_simulator.py

@@ -0,0 +1,334 @@
+#!/usr/bin/env python3
+"""Prepare exact Frontier fixtures for the P1 primary low/high probes.
+
+Prompt-bearing band traces remain under ``--private-root``.  The emitted
+fixtures and public manifest contain token IDs, block IDs, hashes, and
+aggregate metadata, but no prompt text.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import math
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+from transformers import AutoTokenizer
+
+
+HERE = Path(__file__).resolve().parent
+AITUNER_ROOT = HERE.parents[1]
+sys.path.insert(0, str(HERE))
+
+import prepare_pilot as pilot  # noqa: E402
+
+
+PRIMARY_ROLES = ("low1", "high1")
+
+
+def load_module(path: Path):
+    module_root = str(path.parent.resolve())
+    if module_root not in sys.path:
+        sys.path.insert(0, module_root)
+    spec = importlib.util.spec_from_file_location("simfid_s2rb_prepare", path)
+    if spec is None or spec.loader is None:
+        raise ImportError(path)
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def order_hash(values: list[str]) -> str:
+    return hashlib.sha256("\n".join(values).encode()).hexdigest()
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def git_capture(root: Path, *arguments: str) -> str:
+    return subprocess.run(
+        ["git", "-C", str(root), *arguments],
+        check=True,
+        text=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    ).stdout
+
+
+def raw_rows(path: Path) -> dict[int, dict[str, Any]]:
+    result = {}
+    with path.open(encoding="utf-8") as source:
+        for index, line in enumerate(source):
+            if line.strip():
+                result[index] = json.loads(line)
+    return result
+
+
+def selected_hashes(
+    selected: list[Any], rows: dict[int, dict[str, Any]]
+) -> dict[str, str]:
+    identifiers = []
+    arrivals = []
+    lengths = []
+    for item in selected:
+        row = rows[item.row_index]
+        identifiers.append(str(row.get("request_id") or row.get("id") or item.row_index))
+        arrivals.append(f"{float(item.timestamp) * 0.1:.12f}")
+        lengths.append(str(int(item.input_length)))
+    return {
+        "request_id_order_sha256": order_hash(identifiers),
+        "arrival_order_sha256": order_hash(arrivals),
+        "input_length_order_sha256": order_hash(lengths),
+    }
+
+
+def kv_blocks(raw_root: Path, cell: str) -> int:
+    stream = next((raw_root / cell / "opprof").glob("*.jsonl"))
+    with stream.open(encoding="utf-8") as source:
+        for line in source:
+            record = json.loads(line)
+            if "step_index" in record:
+                return int(record["kv"]["total_blocks"])
+    raise ValueError(f"no Layer-1 record for {cell}")
+
+
+def source_window(windows_path: Path, window_id: str) -> tuple[dict[str, Any], Path]:
+    return pilot.resolve_source_trace(windows_path, window_id)
+
+
+def prepare(args: argparse.Namespace) -> dict[str, Any]:
+    simulator = load_module(args.replayserve_root / "tools/simfid_s2rb_prepare.py")
+    manifest = json.loads(args.pilot_manifest.read_text(encoding="utf-8"))
+    window, trace = source_window(args.source_windows, args.source_window_id)
+    if args.band_root is not None:
+        role_paths = {
+            role: (args.band_root / f"{role}.jsonl").resolve()
+            for role in PRIMARY_ROLES
+        }
+        band_stats = {
+            role: manifest["private"]["band_stats"][role]
+            for role in PRIMARY_ROLES
+        }
+        for role, path in role_paths.items():
+            if sha256_file(path) != band_stats[role]["sha256"]:
+                raise ValueError(f"pre-materialized band hash mismatch: {role}")
+        private_windows = None
+    else:
+        private_windows, all_band_stats = pilot.materialize_bands(
+            trace, window, args.private_root
+        )
+        private_payload = json.loads(private_windows.read_text(encoding="utf-8"))
+        role_paths = {
+            item["fidelity_pilot_role"]: (
+                private_windows.parent / item["trace_file"]
+            ).resolve()
+            for item in private_payload["windows"]
+        }
+        band_stats = {
+            role: all_band_stats[role]
+            for role in PRIMARY_ROLES
+        }
+
+    tokenizer = AutoTokenizer.from_pretrained(
+        args.tokenizer, local_files_only=True, use_fast=True
+    )
+    fixture_root = args.output / "fixtures"
+    config_root = args.output / "configs"
+    fixture_root.mkdir(parents=True, exist_ok=True)
+    config_root.mkdir(parents=True, exist_ok=True)
+    entries = []
+    red_flags = []
+    for role in PRIMARY_ROLES:
+        trace_path = role_paths[role]
+        retained, trace_stats = simulator.scan_trace(trace_path)
+        rows = raw_rows(trace_path)
+        primary_pool = [retained[(index * len(retained)) // 512] for index in range(512)]
+        selections: dict[str, list[Any]] = {}
+        selected_union: set[int] = set()
+        for cell, cell_manifest in sorted(manifest["cells"].items()):
+            level = "low" if role.startswith("low") else "high"
+            expected = cell_manifest["targets"][level]["selections"][role]
+            pool = retained if int(cell_manifest["tp"]) == 4 else primary_pool
+            selected = [item for item in pool if item.sampling_u <= float(expected["anchor"])]
+            selections[cell] = selected
+            selected_union.update(item.row_index for item in selected)
+            hashes = selected_hashes(selected, rows)
+            if len(selected) != int(expected["selected_count"]):
+                red_flags.append(f"selection_count_{cell}_{role}")
+            for key, value in hashes.items():
+                if value != expected[key]:
+                    red_flags.append(f"selection_hash_{cell}_{role}_{key}")
+
+        token_gates, selected_records, block_stats = simulator.tokenize_and_hash(
+            trace=trace_path,
+            tokenizer=tokenizer,
+            retained=retained,
+            selected_union=selected_union,
+        )
+        if any(gate["status"] != "pass" for gate in token_gates.values()):
+            red_flags.append(f"token_gate_{role}")
+        for cell, selected in selections.items():
+            cell_manifest = manifest["cells"][cell]
+            level = "low" if role.startswith("low") else "high"
+            expected = cell_manifest["targets"][level]["selections"][role]
+            fixture_id = f"fidelity_p1_{cell}_{role}"
+            cell_record = {
+                "cell_id": cell,
+                "tensor_parallel_size": int(cell_manifest["tp"]),
+                "max_num_seqs": int(cell_manifest["mns"]),
+                "store_role": "companion" if int(cell_manifest["tp"]) == 4 else "primary",
+                "kv_capacity": {
+                    "block_size_tokens": 16,
+                    "num_blocks": kv_blocks(args.phase6_raw_root, cell),
+                },
+            }
+            probe = {
+                "probe_index": 0 if role == "low1" else 1,
+                "sampling_u": float(expected["anchor"]),
+            }
+            fixture = simulator.create_fixture(
+                fixture_root=fixture_root,
+                fixture_id=fixture_id,
+                cell=cell_record,
+                probe=probe,
+                row_indexes=[item.row_index for item in selected],
+                meta_by_index={item.row_index: item for item in retained},
+                selected_records=selected_records,
+            )
+            config_path = config_root / f"{fixture_id}.json"
+            config = simulator.build_config(
+                path=config_path,
+                cell=cell_record,
+                mode="frozen-calibrated",
+                fixture_ids=[fixture_id],
+                frontier_root=args.frontier_root,
+                cache_dir=args.cache_dir,
+            )
+            entries.append(
+                {
+                    "cell": cell,
+                    "role": role,
+                    "level": level,
+                    "anchor": expected["anchor"],
+                    "selected_count": len(selected),
+                    "fixture_id": fixture_id,
+                    "fixture_manifest": str(
+                        (fixture_root / fixture_id / "fixture_manifest.json").resolve()
+                    ),
+                    "frontier_csv": fixture["frontier_csv"]["path"],
+                    "sidecar": fixture["sidecar_jsonl"]["path"],
+                    "config": str(config_path.resolve()),
+                    "calibration_scale": config["calibration"]["a_tp"],
+                }
+            )
+        if block_stats["selected_union_records"] != len(selected_union):
+            red_flags.append(f"selected_union_{role}")
+        if trace_stats["retained_inclusive_0_8192"] < 512:
+            red_flags.append(f"retained_too_small_{role}")
+
+    selected_counts = [int(entry["selected_count"]) for entry in entries]
+    calibration = [float(entry["calibration_scale"]) for entry in entries]
+    result = {
+        "schema": "fidelity-p1-frontier-prepared-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "source": {
+            "pilot_manifest": str(args.pilot_manifest.resolve()),
+            "source_windows": str(args.source_windows.resolve()),
+            "source_window_id": args.source_window_id,
+            "source_trace": str(trace.resolve()),
+            "private_windows": (
+                str(private_windows.resolve()) if private_windows is not None else None
+            ),
+            "pre_materialized_band_root": (
+                str(args.band_root.resolve()) if args.band_root is not None else None
+            ),
+            "band_stats": band_stats,
+        },
+        "simulator": {
+            "replayserve_root": str(args.replayserve_root.resolve()),
+            "frontier_root": str(args.frontier_root.resolve()),
+            "tokenizer": str(args.tokenizer.resolve()),
+            "mode": "frozen-calibrated",
+        },
+        "generator": {
+            "script": str(Path(__file__).resolve()),
+            "script_sha256": sha256_file(Path(__file__).resolve()),
+            "aituner_git_head": git_capture(AITUNER_ROOT, "rev-parse", "HEAD").strip(),
+            "aituner_git_status_short": git_capture(AITUNER_ROOT, "status", "--short"),
+        },
+        "entries": entries,
+        "sanity": {
+            "red_flags": red_flags,
+            "n": len(entries),
+            "selected_count": {
+                "n": len(selected_counts),
+                "min": min(selected_counts),
+                "max": max(selected_counts),
+                "distinct_n": len(set(selected_counts)),
+            },
+            "calibration_scale": {
+                "n": len(calibration),
+                "min": min(calibration),
+                "max": max(calibration),
+                "distinct_n": len(set(calibration)),
+            },
+            "invariants": {
+                "entries_12": len(entries) == 12,
+                "roles_2": {entry["role"] for entry in entries} == set(PRIMARY_ROLES),
+                "cells_6": len({entry["cell"] for entry in entries}) == 6,
+                "selected_nonnegative": all(value > 0 for value in selected_counts),
+                "per_config_not_identical": len(set(selected_counts)) > 1,
+            },
+        },
+    }
+    args.public_manifest.parent.mkdir(parents=True, exist_ok=True)
+    args.public_manifest.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n")
+    return result
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("--pilot-manifest", type=Path, required=True)
+    result.add_argument("--source-windows", type=Path, required=True)
+    result.add_argument("--source-window-id", required=True)
+    result.add_argument("--private-root", type=Path, required=True)
+    result.add_argument("--band-root", type=Path)
+    result.add_argument("--output", type=Path, required=True)
+    result.add_argument("--public-manifest", type=Path, required=True)
+    result.add_argument("--phase6-raw-root", type=Path, required=True)
+    result.add_argument("--replayserve-root", type=Path, required=True)
+    result.add_argument("--frontier-root", type=Path, required=True)
+    result.add_argument("--cache-dir", type=Path, required=True)
+    result.add_argument("--tokenizer", type=Path, required=True)
+    return result
+
+
+def main() -> None:
+    result = prepare(parser().parse_args())
+    print(
+        json.dumps(
+            {
+                "status": result["status"],
+                "entries": len(result["entries"]),
+                "red_flags": result["sanity"]["red_flags"],
+            },
+            sort_keys=True,
+        )
+    )
+    if result["status"] != "PASS":
+        raise RuntimeError(result["sanity"]["red_flags"])
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/run_pilot_simulator.py

@@ -0,0 +1,295 @@
+#!/usr/bin/env python3
+"""Run and score the 12 frozen Frontier P1 primary probes, CPU only."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import os
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+
+def load_module(name: str, path: Path):
+    module_root = str(path.parent.resolve())
+    if module_root not in sys.path:
+        sys.path.insert(0, module_root)
+    spec = importlib.util.spec_from_file_location(name, path)
+    if spec is None or spec.loader is None:
+        raise ImportError(path)
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    temporary = path.with_suffix(path.suffix + ".tmp")
+    temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    os.replace(temporary, path)
+
+
+def git_capture(root: Path, *arguments: str) -> str:
+    return subprocess.run(
+        ["git", "-C", str(root), *arguments],
+        check=True,
+        text=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    ).stdout
+
+
+def execute(args: argparse.Namespace) -> dict[str, Any]:
+    prepared = json.loads(args.prepared_manifest.read_text(encoding="utf-8"))
+    if prepared["status"] != "PASS":
+        raise RuntimeError("prepared simulator manifest did not pass")
+    driver = load_module(
+        "simfid_execution_driver",
+        args.replayserve_root
+        / "runs/simfid_s2rb/results/execution_driver.py",
+    )
+    head = git_capture(args.frontier_root, "rev-parse", "HEAD").strip()
+    status_short = git_capture(args.frontier_root, "status", "--short")
+    aituner_root = Path(__file__).resolve().parents[2]
+    aituner_head = git_capture(aituner_root, "rev-parse", "HEAD").strip()
+    aituner_status_short = git_capture(aituner_root, "status", "--short")
+    results = []
+    failures = []
+    gpu_visibility_disabled = True
+    for sequence, entry in enumerate(prepared["entries"]):
+        run_root = args.output / f"{sequence:02d}_{entry['fixture_id']}"
+        scorer_path = run_root / "scorer_output.json"
+        if scorer_path.is_file() and args.resume:
+            scorer = json.loads(scorer_path.read_text(encoding="utf-8"))
+            results.append({**entry, "sequence": sequence, "scorer": scorer, "resumed": True})
+            continue
+        run_root.mkdir(parents=True, exist_ok=True)
+        config_path = Path(entry["config"])
+        config = json.loads(config_path.read_text(encoding="utf-8"))
+        fixture_manifest_path = Path(entry["fixture_manifest"])
+        fixture = json.loads(fixture_manifest_path.read_text(encoding="utf-8"))
+        trace_path = Path(entry["frontier_csv"])
+        sidecar_path = Path(entry["sidecar"])
+        metrics_root = run_root / "frontier_metrics"
+        run_id = f"fidelity_p1_frontier_{sequence:02d}_{entry['cell']}_{entry['role']}"
+        knobs = config["frontier"]["knobs"]
+        command = driver.build_command(
+            trace_path=trace_path,
+            metrics_root=metrics_root,
+            run_id=run_id,
+            knobs=knobs,
+        )
+        driver.audit_command(command, knobs)
+        row = {
+            "hook_path": config["calibration"]["hook_path"],
+            "applied_a_tp": config["calibration"]["a_tp"],
+            "sidecar_path": str(sidecar_path),
+            "request_count": int(fixture["request_count"]),
+            "tensor_parallel_size": int(fixture["tensor_parallel_size"]),
+        }
+        environment = driver.environment_for(row)
+        gpu_visibility_disabled = gpu_visibility_disabled and (
+            environment.get("CUDA_VISIBLE_DEVICES") == ""
+            and environment.get("NVIDIA_VISIBLE_DEVICES") == "void"
+        )
+        run_manifest = {
+            "schema": "fidelity-p1-frontier-run-v1",
+            "sequence": sequence,
+            "cell": entry["cell"],
+            "role": entry["role"],
+            "anchor": entry["anchor"],
+            "request_count": entry["selected_count"],
+            "frontier": {
+                "root": str(args.frontier_root.resolve()),
+                "git_head": head,
+                "git_status_short": status_short,
+            },
+            "runner": {
+                "script": str(Path(__file__).resolve()),
+                "script_sha256": sha256_file(Path(__file__).resolve()),
+                "aituner_git_head": aituner_head,
+                "aituner_git_status_short": aituner_status_short,
+            },
+            "inputs": {
+                "config": str(config_path),
+                "config_sha256": sha256_file(config_path),
+                "fixture_manifest": str(fixture_manifest_path),
+                "fixture_manifest_sha256": sha256_file(fixture_manifest_path),
+                "frontier_csv": str(trace_path),
+                "frontier_csv_sha256": sha256_file(trace_path),
+                "sidecar": str(sidecar_path),
+                "sidecar_sha256": sha256_file(sidecar_path),
+            },
+            "environment": {
+                key: environment[key]
+                for key in (
+                    "PYTHONPATH",
+                    "FRONTIER_EXECUTION_TIME_SCALE",
+                    "CUDA_VISIBLE_DEVICES",
+                    "NVIDIA_VISIBLE_DEVICES",
+                    "FRONTIER_LOG_LEVEL",
+                )
+            },
+            "command": command,
+            "contains_prompt_text": False,
+        }
+        atomic_json(run_root / "run_manifest.json", run_manifest)
+        start = time.time()
+        with (run_root / "stdout.log").open("w", encoding="utf-8") as stdout, (
+            run_root / "stderr.log"
+        ).open("w", encoding="utf-8") as stderr:
+            try:
+                process = subprocess.run(
+                    command,
+                    cwd=args.frontier_root,
+                    env=environment,
+                    stdout=stdout,
+                    stderr=stderr,
+                    timeout=args.timeout_s,
+                )
+                return_code = int(process.returncode)
+            except subprocess.TimeoutExpired:
+                return_code = 124
+        runtime = time.time() - start
+        if return_code != 0:
+            failure = {
+                "sequence": sequence,
+                "cell": entry["cell"],
+                "role": entry["role"],
+                "return_code": return_code,
+                "runtime_s": runtime,
+            }
+            failures.append(failure)
+            atomic_json(run_root / "failure.json", failure)
+            break
+        system_path, request_path = driver.find_metrics(run_root)
+        scorer = driver.score_trial(row, system_path, request_path)
+        scorer["runtime_s"] = runtime
+        atomic_json(scorer_path, scorer)
+        results.append({**entry, "sequence": sequence, "scorer": scorer, "resumed": False})
+        print(
+            json.dumps(
+                {
+                    "sequence": sequence,
+                    "cell": entry["cell"],
+                    "role": entry["role"],
+                    "runtime_s": runtime,
+                    "sim_pass_rate": scorer["slo"]["pass_rate"],
+                    "sim_feasible": scorer["slo"]["feasible"],
+                },
+                sort_keys=True,
+            ),
+            flush=True,
+        )
+
+    pass_rates = [float(item["scorer"]["slo"]["pass_rate"]) for item in results]
+    throughputs = [
+        float(item["scorer"]["throughput_requests_per_second_per_gpu"])
+        for item in results
+    ]
+    runtimes = [float(item["scorer"]["runtime_s"]) for item in results]
+    red_flags = []
+    if failures:
+        red_flags.append("frontier_run_failure")
+    if len(results) != 12:
+        red_flags.append("runs_not_12")
+    if any(not 0.0 <= value <= 1.0 for value in pass_rates):
+        red_flags.append("pass_rate_out_of_range")
+    if any(value <= 0 for value in throughputs):
+        red_flags.append("nonpositive_throughput")
+    result = {
+        "schema": "fidelity-p1-frontier-result-v1",
+        "status": "PASS" if not red_flags else "STOP",
+        "prepared_manifest": str(args.prepared_manifest.resolve()),
+        "prepared_manifest_sha256": sha256_file(args.prepared_manifest),
+        "frontier": {
+            "root": str(args.frontier_root.resolve()),
+            "git_head": head,
+            "git_status_short": status_short,
+        },
+        "runner": {
+            "script": str(Path(__file__).resolve()),
+            "script_sha256": sha256_file(Path(__file__).resolve()),
+            "aituner_git_head": aituner_head,
+            "aituner_git_status_short": aituner_status_short,
+        },
+        "results": results,
+        "failures": failures,
+        "sanity": {
+            "red_flags": red_flags,
+            "n": len(results),
+            "pass_rate": {
+                "n": len(pass_rates),
+                "min": min(pass_rates) if pass_rates else None,
+                "max": max(pass_rates) if pass_rates else None,
+                "distinct_n": len(set(pass_rates)),
+            },
+            "throughput_per_gpu": {
+                "n": len(throughputs),
+                "min": min(throughputs) if throughputs else None,
+                "max": max(throughputs) if throughputs else None,
+                "distinct_n": len(set(throughputs)),
+            },
+            "runtime_s": {
+                "n": len(runtimes),
+                "min": min(runtimes) if runtimes else None,
+                "max": max(runtimes) if runtimes else None,
+                "distinct_n": len(set(runtimes)),
+            },
+            "invariants": {
+                "runs_12": len(results) == 12,
+                "zero_failures": not failures,
+                "ratios_bounded": all(0.0 <= value <= 1.0 for value in pass_rates),
+                "nonnegative_metrics": all(value > 0 for value in throughputs),
+                "per_config_not_identical": len(set(pass_rates)) > 1,
+                "gpu_visibility_disabled": gpu_visibility_disabled,
+            },
+        },
+    }
+    atomic_json(args.output / "metrics.json", result)
+    return result
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("--prepared-manifest", type=Path, required=True)
+    result.add_argument("--output", type=Path, required=True)
+    result.add_argument("--replayserve-root", type=Path, required=True)
+    result.add_argument("--frontier-root", type=Path, required=True)
+    result.add_argument("--timeout-s", type=float, default=900.0)
+    result.add_argument("--resume", action="store_true")
+    return result
+
+
+def main() -> None:
+    result = execute(parser().parse_args())
+    print(
+        json.dumps(
+            {
+                "status": result["status"],
+                "runs": len(result["results"]),
+                "red_flags": result["sanity"]["red_flags"],
+            },
+            sort_keys=True,
+        )
+    )
+    if result["status"] != "PASS":
+        raise RuntimeError(result["sanity"]["red_flags"])
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/strong-baseline-metrics.json

@@ -0,0 +1,477 @@
+{
+  "comparison": "same 5-second prefix, folds, logistic family, regularization, and frozen Frontier outputs; the only nested difference is real Layer-1 engine state",
+  "decision": {
+    "contribution_established": false,
+    "prospective_requirement": "repeat sim+outcome versus sim+outcome+instrumentation on complete held-out tasks"
+  },
+  "features": {
+    "instrumentation_only": [
+      "model_steps_per_second",
+      "waiting_mean",
+      "waiting_max",
+      "waiting_nonzero_share",
+      "running_mean",
+      "running_max",
+      "decode_batch_mean",
+      "decode_batch_max",
+      "decode_batch_cv",
+      "kv_usage_mean",
+      "kv_usage_max",
+      "kv_usage_end_minus_start",
+      "graph_none_share",
+      "graph_full_share",
+      "padding_fraction",
+      "prefill_token_fraction",
+      "preemptions"
+    ],
+    "shared_outcome": [
+      "log_offered_rate_per_gpu",
+      "log2_tp",
+      "log2_max_num_seqs",
+      "admitted_fraction",
+      "completed_over_admitted",
+      "completed_pass_rate",
+      "completed_fail_fraction_of_total",
+      "outstanding_over_admitted",
+      "ttft_max_over_slo_max",
+      "ttft_mean_over_slo_max",
+      "tpot_max_over_slo",
+      "tpot_mean_over_slo",
+      "admitted_input_tokens_mean_over_limit"
+    ],
+    "shared_simulator": [
+      "log_sim_completed_throughput_per_gpu",
+      "sim_slo_pass_rate",
+      "sim_slo_feasible"
+    ]
+  },
+  "headline": {
+    "group_bootstrap": {
+      "accuracy_delta_instrumentation_minus_outcome": {
+        "ci95": [
+          0.0,
+          0.18181818181818188
+        ],
+        "point": 0.08108108108108103
+      },
+      "brier_delta_instrumentation_minus_outcome": {
+        "ci95": [
+          -0.04292727744470806,
+          0.019924730979981074
+        ],
+        "point": -0.010145365131402809
+      },
+      "replicates": 10000,
+      "seed": 20260714,
+      "semantics": "group bootstrap over cells; diagnostic confidence interval"
+    },
+    "paired_correctness": {
+      "both_correct": 30,
+      "both_wrong": 4,
+      "instrumentation_only_correct": 3,
+      "mcnemar_exact_two_sided_p": 0.25,
+      "sim_outcome_only_correct": 0
+    },
+    "sim_plus_outcome": {
+      "classification": {
+        "accuracy": 0.8108108108108109,
+        "balanced_accuracy": 0.7242063492063493,
+        "brier": 0.1058226346682949,
+        "confusion": {
+          "false_negative": 3,
+          "false_positive": 4,
+          "true_negative": 5,
+          "true_positive": 25
+        },
+        "log_loss": 0.3011048455679668
+      },
+      "policy_0p95": {
+        "abstain_continue_full": 17,
+        "correctly_saved_h20_hours": 0.5429431818208333,
+        "decision_coverage": 0.5405405405405406,
+        "early_accept": 16,
+        "early_reject": 4,
+        "false_accept": 0,
+        "false_accept_examples": [],
+        "false_reject": 0,
+        "false_reject_examples": [],
+        "full_trial_h20_hours": 1.0669595034675,
+        "invalidly_saved_h20_hours": 0.0,
+        "remaining_h20_hours_at_cutoff": 0.957237281245278,
+        "saved_h20_hours_if_decisions_used": 0.5429431818208333,
+        "threshold": 0.95,
+        "valid_cost_reduction_fraction": 0.5088695307144538,
+        "valid_zero_error_policy": true
+      }
+    },
+    "sim_plus_outcome_plus_instrumentation": {
+      "classification": {
+        "accuracy": 0.8918918918918919,
+        "balanced_accuracy": 0.8154761904761905,
+        "brier": 0.0956772695368921,
+        "confusion": {
+          "false_negative": 1,
+          "false_positive": 3,
+          "true_negative": 6,
+          "true_positive": 27
+        },
+        "log_loss": 0.288823031828762
+      },
+      "policy_0p95": {
+        "abstain_continue_full": 12,
+        "correctly_saved_h20_hours": 0.7360063646722222,
+        "decision_coverage": 0.6756756756756757,
+        "early_accept": 20,
+        "early_reject": 5,
+        "false_accept": 0,
+        "false_accept_examples": [],
+        "false_reject": 0,
+        "false_reject_examples": [],
+        "full_trial_h20_hours": 1.0669595034675,
+        "invalidly_saved_h20_hours": 0.0,
+        "remaining_h20_hours_at_cutoff": 0.957237281245278,
+        "saved_h20_hours_if_decisions_used": 0.7360063646722222,
+        "threshold": 0.95,
+        "valid_cost_reduction_fraction": 0.6898165884274738,
+        "valid_zero_error_policy": true
+      }
+    }
+  },
+  "headline_regularization": 1.0,
+  "provenance": {
+    "frozen_simulator_manifest_scorer_set_sha256": "833842d96ecaa0b059ef99852621752f7989e63d100118b6025425fb119b7a55",
+    "phase6_metrics": "/home/gahow/phd/aituner/runs/opprof-phase6/phase6/metrics.json",
+    "phase6_metrics_sha256": "290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16",
+    "phase6_raw_root": "/home/gahow/phd/aituner/runs/opprof-phase6/phase6/solo-authoritative/cells",
+    "simulator_metrics": "/home/gahow/phd/replayserve/runs/simfid_s2rb/results/metrics.json",
+    "simulator_metrics_sha256": "55edb37d5692e979ab6f6dc6c65913a9db0aa0a836c350e4c05d9c38eee78206",
+    "simulator_raw_root": "/home/gahow/phd/replayserve/runs/simfid_s2rb/results/raw"
+  },
+  "regularization_sensitivity": {
+    "0.1": {
+      "group_bootstrap": {
+        "accuracy_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            -0.17500000000000004,
+            0.0
+          ],
+          "point": -0.08108108108108103
+        },
+        "brier_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            -0.026383192545085435,
+            0.0607951286646285
+          ],
+          "point": 0.019228316404518567
+        },
+        "replicates": 10000,
+        "seed": 20260714,
+        "semantics": "group bootstrap over cells; diagnostic confidence interval"
+      },
+      "paired_correctness": {
+        "both_correct": 30,
+        "both_wrong": 4,
+        "instrumentation_only_correct": 0,
+        "mcnemar_exact_two_sided_p": 0.25,
+        "sim_outcome_only_correct": 3
+      },
+      "sim_plus_outcome": {
+        "classification": {
+          "accuracy": 0.8918918918918919,
+          "balanced_accuracy": 0.8154761904761905,
+          "brier": 0.10990776306815446,
+          "confusion": {
+            "false_negative": 1,
+            "false_positive": 3,
+            "true_negative": 6,
+            "true_positive": 27
+          },
+          "log_loss": 0.328357763455984
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 12,
+          "correctly_saved_h20_hours": 0.7402314096841667,
+          "decision_coverage": 0.6756756756756757,
+          "early_accept": 20,
+          "early_reject": 5,
+          "false_accept": 0,
+          "false_accept_examples": [],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.0,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.7402314096841667,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": 0.6937764809990414,
+          "valid_zero_error_policy": true
+        }
+      },
+      "sim_plus_outcome_plus_instrumentation": {
+        "classification": {
+          "accuracy": 0.8108108108108109,
+          "balanced_accuracy": 0.7619047619047619,
+          "brier": 0.12913607947267303,
+          "confusion": {
+            "false_negative": 4,
+            "false_positive": 3,
+            "true_negative": 6,
+            "true_positive": 24
+          },
+          "log_loss": 0.4373556318820343
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 9,
+          "correctly_saved_h20_hours": 0.7469523484622221,
+          "decision_coverage": 0.7567567567567568,
+          "early_accept": 22,
+          "early_reject": 6,
+          "false_accept": 2,
+          "false_accept_examples": [
+            {
+              "anchor": 0.49609375,
+              "cell": "tp2_mns8",
+              "label_feasible": false,
+              "probability_feasible": 0.9869795738005246,
+              "remaining_h20_hours": 0.010117910306111111
+            },
+            {
+              "anchor": 0.033717411016,
+              "cell": "tp4_mns16",
+              "label_feasible": false,
+              "probability_feasible": 0.9855364057197005,
+              "remaining_h20_hours": 0.023106262014444445
+            }
+          ],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.03322417232055556,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.7801765207827777,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": null,
+          "valid_zero_error_policy": false
+        }
+      }
+    },
+    "1.0": {
+      "group_bootstrap": {
+        "accuracy_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            0.0,
+            0.18181818181818188
+          ],
+          "point": 0.08108108108108103
+        },
+        "brier_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            -0.04292727744470806,
+            0.019924730979981074
+          ],
+          "point": -0.010145365131402809
+        },
+        "replicates": 10000,
+        "seed": 20260714,
+        "semantics": "group bootstrap over cells; diagnostic confidence interval"
+      },
+      "paired_correctness": {
+        "both_correct": 30,
+        "both_wrong": 4,
+        "instrumentation_only_correct": 3,
+        "mcnemar_exact_two_sided_p": 0.25,
+        "sim_outcome_only_correct": 0
+      },
+      "sim_plus_outcome": {
+        "classification": {
+          "accuracy": 0.8108108108108109,
+          "balanced_accuracy": 0.7242063492063493,
+          "brier": 0.1058226346682949,
+          "confusion": {
+            "false_negative": 3,
+            "false_positive": 4,
+            "true_negative": 5,
+            "true_positive": 25
+          },
+          "log_loss": 0.3011048455679668
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 17,
+          "correctly_saved_h20_hours": 0.5429431818208333,
+          "decision_coverage": 0.5405405405405406,
+          "early_accept": 16,
+          "early_reject": 4,
+          "false_accept": 0,
+          "false_accept_examples": [],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.0,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.5429431818208333,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": 0.5088695307144538,
+          "valid_zero_error_policy": true
+        }
+      },
+      "sim_plus_outcome_plus_instrumentation": {
+        "classification": {
+          "accuracy": 0.8918918918918919,
+          "balanced_accuracy": 0.8154761904761905,
+          "brier": 0.0956772695368921,
+          "confusion": {
+            "false_negative": 1,
+            "false_positive": 3,
+            "true_negative": 6,
+            "true_positive": 27
+          },
+          "log_loss": 0.288823031828762
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 12,
+          "correctly_saved_h20_hours": 0.7360063646722222,
+          "decision_coverage": 0.6756756756756757,
+          "early_accept": 20,
+          "early_reject": 5,
+          "false_accept": 0,
+          "false_accept_examples": [],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.0,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.7360063646722222,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": 0.6898165884274738,
+          "valid_zero_error_policy": true
+        }
+      }
+    },
+    "10.0": {
+      "group_bootstrap": {
+        "accuracy_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            -0.13333333333333341,
+            0.05555555555555558
+          ],
+          "point": -0.027027027027027084
+        },
+        "brier_delta_instrumentation_minus_outcome": {
+          "ci95": [
+            -0.03091105649870874,
+            0.01684192005239855
+          ],
+          "point": -0.007318433328714388
+        },
+        "replicates": 10000,
+        "seed": 20260714,
+        "semantics": "group bootstrap over cells; diagnostic confidence interval"
+      },
+      "paired_correctness": {
+        "both_correct": 30,
+        "both_wrong": 4,
+        "instrumentation_only_correct": 1,
+        "mcnemar_exact_two_sided_p": 1.0,
+        "sim_outcome_only_correct": 2
+      },
+      "sim_plus_outcome": {
+        "classification": {
+          "accuracy": 0.8648648648648649,
+          "balanced_accuracy": 0.7222222222222222,
+          "brier": 0.10613344425735322,
+          "confusion": {
+            "false_negative": 0,
+            "false_positive": 5,
+            "true_negative": 4,
+            "true_positive": 28
+          },
+          "log_loss": 0.3404203142465075
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 32,
+          "correctly_saved_h20_hours": 0.21727432337249997,
+          "decision_coverage": 0.13513513513513514,
+          "early_accept": 5,
+          "early_reject": 0,
+          "false_accept": 0,
+          "false_accept_examples": [],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.0,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.21727432337249997,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": 0.20363877229302757,
+          "valid_zero_error_policy": true
+        }
+      },
+      "sim_plus_outcome_plus_instrumentation": {
+        "classification": {
+          "accuracy": 0.8378378378378378,
+          "balanced_accuracy": 0.7420634920634921,
+          "brier": 0.09881501092863883,
+          "confusion": {
+            "false_negative": 2,
+            "false_positive": 4,
+            "true_negative": 5,
+            "true_positive": 26
+          },
+          "log_loss": 0.312914193285738
+        },
+        "policy_0p95": {
+          "abstain_continue_full": 30,
+          "correctly_saved_h20_hours": 0.2384080185036111,
+          "decision_coverage": 0.1891891891891892,
+          "early_accept": 6,
+          "early_reject": 1,
+          "false_accept": 0,
+          "false_accept_examples": [],
+          "false_reject": 0,
+          "false_reject_examples": [],
+          "full_trial_h20_hours": 1.0669595034675,
+          "invalidly_saved_h20_hours": 0.0,
+          "remaining_h20_hours_at_cutoff": 0.957237281245278,
+          "saved_h20_hours_if_decisions_used": 0.2384080185036111,
+          "threshold": 0.95,
+          "valid_cost_reduction_fraction": 0.22344617366339725,
+          "valid_zero_error_policy": true
+        }
+      }
+    }
+  },
+  "sanity": {
+    "examples": {
+      "distinct_n": 1,
+      "max": 1.0,
+      "min": 1.0,
+      "n": 37
+    },
+    "frozen_simulator_runs": 92,
+    "invariants": {
+      "all_examples_matched_once": true,
+      "labels_not_identical": true,
+      "per_config_results_not_all_identical": true,
+      "same_nested_folds": true,
+      "simulator_ratios_bounded": true
+    },
+    "labels": {
+      "distinct_n": 2,
+      "max": 1.0,
+      "min": 0.0,
+      "n": 37,
+      "negative": 9,
+      "positive": 28
+    },
+    "matched_simulator_pass_rate": {
+      "distinct_n": 12,
+      "max": 1.0,
+      "min": 0.06884057971014493,
+      "n": 37
+    },
+    "red_flags": []
+  },
+  "schema": "fidelity-strong-baseline-v1",
+  "scope": "retrospective one-task headroom audit; not contribution evidence",
+  "status": "PASS"
+}

runs/fidelity-headroom/test_analysis.py

@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import importlib.util
+import math
+import sys
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def load_analysis():
+    spec = importlib.util.spec_from_file_location("fidelity_headroom", HERE / "analyze_existing.py")
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def main() -> None:
+    analysis = load_analysis()
+    curve = analysis.topk_curve(
+        {"a": 3.0, "b": 2.0, "c": 1.0},
+        {"a": 1.0, "b": 2.0, "c": 2.0},
+        2e-6,
+    )
+    assert curve["points"][0]["expanded_k"] == 2
+    assert curve["points"][0]["candidates"] == ["b", "c"]
+    assert math.isclose(curve["points"][0]["real_regret"], 1.0 / 3.0)
+    assert curve["points"][2]["real_regret"] == 0.0
+    assert curve["minimum_k"]["five_percent"] == {"nominal_k": 3, "expanded_k": 3}
+    assert analysis._mcnemar_exact_p(0, 1) == 1.0
+    assert analysis._mcnemar_exact_p(0, 5) == 0.0625
+    print("fidelity headroom analysis: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_pilot_e2e.py

@@ -0,0 +1,50 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+from analyze_pilot_e2e import expanded_top_k, replay
+
+
+def candidate(
+    cell: str,
+    sim_score: float,
+    real_feasible: bool,
+    probability: float,
+) -> dict[str, object]:
+    return {
+        "cell": cell,
+        "level": "high",
+        "sim_throughput_req_s_per_gpu": sim_score,
+        "real_goodput_req_s_per_gpu": sim_score,
+        "real_feasible": real_feasible,
+        "setup_h20_hours": 0.1,
+        "full_trial_h20_hours": 0.05,
+        "prefix_h20_hours": 0.01,
+        "instrument_probability": probability,
+    }
+
+
+def main() -> None:
+    candidates = [
+        candidate("a", 3.0, True, 0.5),
+        candidate("b", 2.0, False, 0.01),
+        candidate("c", 2.0, True, 0.99),
+    ]
+    shortlist = expanded_top_k(candidates, 2)
+    assert [item["cell"] for item in shortlist] == ["a", "b", "c"]
+    result = replay(
+        shortlist,
+        probability_key="instrument_probability",
+        oracle_goodput=3.0,
+        common_failure_h20_hours=0.02,
+    )
+    assert result["selected_cell"] == "a"
+    assert result["false_accept"] == 0
+    assert result["false_reject"] == 0
+    assert result["early_accept"] == 1
+    assert result["early_reject"] == 1
+    assert result["online_h20_hours"] > 0
+    print("fidelity pilot e2e: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_pilot_tools.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import json
+import math
+import sys
+import tempfile
+from dataclasses import dataclass
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+
+import pilot_controller as controller  # noqa: E402
+import prepare_pilot as prepare  # noqa: E402
+from analyze_pilot import campaign_gpu_accounting  # noqa: E402
+
+
+@dataclass
+class Request:
+    row_id: str
+    sampling_u: float
+    arrival_s: float = 0.0
+    prompt_tokens_hint: int = 1
+
+
+def main() -> None:
+    requests = [
+        Request("a", 0.1),
+        Request("b", 0.2),
+        Request("c", 0.2),
+        Request("d", 0.9),
+    ]
+    anchor, selected = prepare.attainable_anchor(requests, target_count=2)
+    assert anchor == 0.2
+    assert [request.row_id for request in selected] == ["a", "b", "c"]
+
+    with tempfile.TemporaryDirectory() as temporary:
+        root = Path(temporary)
+        source = root / "source.jsonl"
+        rows = []
+        for index, role in enumerate(prepare.ROLES):
+            rows.append(
+                {
+                    "request_id": role,
+                    "timestamp": float(index),
+                    "sampling_u": (index + 0.5) / len(prepare.ROLES),
+                    "input_length": 16 + index,
+                    "messages": [{"role": "user", "content": role}],
+                }
+            )
+        source.write_text(
+            "".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8"
+        )
+        windows, stats = prepare.materialize_bands(
+            source,
+            {
+                "window_id": "source",
+                "trace_type": "chat",
+                "window_start": 0.0,
+                "window_end": 600.0,
+            },
+            root / "private",
+        )
+        assert windows.is_file()
+        assert all(stats[role]["rows"] == 1 for role in prepare.ROLES)
+        for role in prepare.ROLES:
+            row = json.loads((root / "private" / "traces" / f"{role}.jsonl").read_text())
+            assert row["fidelity_pilot_band"] == role
+            assert abs(float(row["sampling_u"]) - 0.5) < 1e-12
+
+        prior = root / "prior-state.json"
+        primary = root / "primary-state.json"
+        prior.write_text(
+            json.dumps(
+                {
+                    "status": "failed",
+                    "gpu_hours_total": 0.02,
+                    "hard_cap_h20_hours": 3.5,
+                }
+            ),
+            encoding="utf-8",
+        )
+        primary.write_text(
+            json.dumps(
+                {
+                    "status": "complete",
+                    "gpu_hours_total": 1.5,
+                    "hard_cap_h20_hours": 3.5,
+                }
+            ),
+            encoding="utf-8",
+        )
+        accounting = campaign_gpu_accounting(primary, (prior,))
+        assert math.isclose(accounting["aggregate_h20_hours"], 1.52)
+        assert all(accounting["invariants"].values())
+
+    assert len(controller.ORDER) == 6
+    assert set(controller.ORDER) == set(prepare.CELLS)
+    assert math.isclose(
+        sum(
+            controller.CELL_ESTIMATE_H20_HOURS[int(config["tp"])]
+            for config in prepare.CELLS.values()
+        ) + controller.SAFETY_H20_HOURS,
+        3.0,
+    )
+    selection = {
+        "selected_count": 122,
+        "request_id_order_sha256": "request-hash",
+        "arrival_order_sha256": "arrival-hash",
+        "input_length_order_sha256": "length-hash",
+    }
+    warmup = {
+        "kind": "warmup",
+        "selection": {"count": 16},
+        "invariants": {
+            "warmup_16": True,
+            "warmup_exact_16": True,
+            "warmup_long": True,
+        },
+    }
+    controller.validate_result_selection(
+        result=warmup,
+        selection=selection,
+        cell="tp1_mns8",
+        role="burnin",
+        warmup=True,
+    )
+    measured = {
+        "kind": "anchor",
+        "selection": {
+            "count": 122,
+            "request_id_order_sha256": "request-hash",
+            "arrival_order_sha256": "arrival-hash",
+            "raw_length_order_sha256": "length-hash",
+        },
+        "invariants": {},
+    }
+    controller.validate_result_selection(
+        result=measured,
+        selection=selection,
+        cell="tp1_mns8",
+        role="low1",
+        warmup=False,
+    )
+    try:
+        controller.validate_result_selection(
+            result=warmup,
+            selection=selection,
+            cell="tp1_mns8",
+            role="low1",
+            warmup=False,
+        )
+    except RuntimeError as error:
+        assert "selection count mismatch" in str(error)
+    else:
+        raise AssertionError("measured selection accepted a warmup subset")
+    print("fidelity pilot tools: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_prefix_analysis.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import math
+import sys
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+
+import analyze_prefixes as analysis  # noqa: E402
+
+
+def main() -> None:
+    exact, exact_source = analysis.completion_elapsed_s(
+        {"completed_elapsed_s": 7.25}
+    )
+    assert exact == 7.25 and exact_source == "exact_monotonic"
+
+    reconstructed, reconstructed_source = analysis.completion_elapsed_s(
+        {
+            "success": True,
+            "arrival_s": 2.0,
+            "ttft_ms": 100.0,
+            "tpot_ms": 10.0,
+            "completion_tokens": 11,
+        }
+    )
+    assert math.isclose(reconstructed or 0.0, 2.2)
+    assert reconstructed_source == "reconstructed_from_latency"
+    missing, missing_source = analysis.completion_elapsed_s({"success": False})
+    assert missing is None and missing_source == "unobserved_failure"
+
+    examples = [
+        analysis.PrefixExample(
+            cell=f"c{index}",
+            anchor=float(index),
+            cutoff_s=5.0,
+            tp=1,
+            full_elapsed_s=65.0,
+            feasible=label,
+            primary_feasible=label,
+            outcome=(float(index),),
+            instrumentation=(float(index % 2),),
+            completion_time_source="exact_monotonic",
+        )
+        for index, label in enumerate((0, 1, 1))
+    ]
+    labels = analysis.np.asarray([0, 1, 1])
+    probabilities = analysis.np.asarray([0.01, 0.99, 0.60])
+    policy = analysis.policy_metrics(examples, labels, probabilities, 0.95)
+    assert policy["early_accept"] == 1
+    assert policy["early_reject"] == 1
+    assert policy["abstain_continue_full"] == 1
+    assert policy["false_accept"] == 0 and policy["false_reject"] == 0
+    assert policy["valid_zero_error_policy"]
+    assert policy["valid_cost_reduction_fraction"] is not None
+    model = analysis.fit_frozen_model(
+        examples,
+        instrumentation_aware=True,
+        regularization=1.0,
+    )
+    frozen_probability = analysis.predict_frozen_model(model, examples)
+    assert len(frozen_probability) == len(examples)
+    assert analysis.np.all(frozen_probability >= 0.0)
+    assert analysis.np.all(frozen_probability <= 1.0)
+    print("fidelity prefix analysis: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_strong_baseline.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from analyze_strong_baseline import analyze
+
+
+ROOT = Path(__file__).resolve().parents[2]
+REPLAYSERVE = ROOT.parent / "replayserve"
+
+
+def main() -> None:
+    result = analyze(
+        ROOT / "runs/opprof-phase6/phase6/metrics.json",
+        ROOT / "runs/opprof-phase6/phase6/solo-authoritative/cells",
+        REPLAYSERVE / "runs/simfid_s2rb/results/raw",
+        REPLAYSERVE / "runs/simfid_s2rb/results/metrics.json",
+    )
+    assert result["status"] == "PASS", json.dumps(result["sanity"], indent=2)
+    assert result["sanity"]["frozen_simulator_runs"] == 92
+    assert result["sanity"]["labels"]["n"] == 37
+    headline = result["headline"]
+    assert headline["sim_plus_outcome"]["policy_0p95"]["false_accept"] == 0
+    assert headline["sim_plus_outcome"]["policy_0p95"]["false_reject"] == 0
+    assert (
+        headline["sim_plus_outcome_plus_instrumentation"]["policy_0p95"][
+            "false_accept"
+        ]
+        == 0
+    )
+    print("fidelity strong baseline: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/fidelity-headroom/test_strong_pilot.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import json
+import tempfile
+from pathlib import Path
+
+import numpy as np
+
+from analyze_prefixes import PrefixExample
+from prepare_pilot_simulator import load_module as load_prepare_module
+from run_pilot_simulator import load_module as load_run_module
+from analyze_strong_pilot import (
+    covariate_shift,
+    fit_model,
+    load_pilot_simulator,
+    predict_model,
+)
+
+
+def example(index: int) -> PrefixExample:
+    label = int(index >= 4)
+    return PrefixExample(
+        cell=f"cell-{index // 2}",
+        anchor=float(index),
+        cutoff_s=5.0,
+        tp=1,
+        full_elapsed_s=10.0,
+        feasible=label,
+        primary_feasible=label,
+        outcome=tuple(float(index + offset) for offset in range(13)),
+        instrumentation=tuple(float(index * offset + 1) for offset in range(17)),
+        completion_time_source="exact_monotonic",
+    )
+
+
+def main() -> None:
+    examples = [example(index) for index in range(8)]
+    simulator = [(float(index), index / 10.0, float(index >= 4)) for index in range(8)]
+    for instrumentation_aware in (False, True):
+        model = fit_model(
+            examples,
+            simulator,
+            instrumentation_aware=instrumentation_aware,
+            regularization=1.0,
+        )
+        probability = predict_model(model, examples, simulator)
+        assert probability.shape == (8,)
+        assert np.all((probability >= 0.0) & (probability <= 1.0))
+        shift = covariate_shift(
+            examples,
+            simulator,
+            examples,
+            simulator,
+            instrumentation_aware=instrumentation_aware,
+        )
+        assert shift["values"]["min"] >= 0.0
+        assert shift["count_gt_3"] == 0
+
+    payload = {
+        "status": "PASS",
+        "results": [
+            {
+                "cell": f"cell-{index // 2}",
+                "role": "low1" if index % 2 == 0 else "high1",
+                "scorer": {
+                    "throughput_requests_per_second_per_gpu": 1.0 + index,
+                    "slo": {
+                        "pass_rate": index / 12.0,
+                        "feasible": index % 2 == 0,
+                    },
+                },
+            }
+            for index in range(12)
+        ],
+    }
+    with tempfile.TemporaryDirectory() as temporary:
+        path = Path(temporary) / "metrics.json"
+        path.write_text(json.dumps(payload), encoding="utf-8")
+        features, red_flags = load_pilot_simulator(path)
+    assert len(features) == 12
+    assert red_flags == []
+
+    with tempfile.TemporaryDirectory() as temporary:
+        root = Path(temporary)
+        (root / "prepare_dependency.py").write_text("VALUE = 17\n", encoding="utf-8")
+        (root / "prepare_target.py").write_text(
+            "from prepare_dependency import VALUE\n", encoding="utf-8"
+        )
+        assert load_prepare_module(root / "prepare_target.py").VALUE == 17
+        (root / "run_dependency.py").write_text("VALUE = 23\n", encoding="utf-8")
+        (root / "run_target.py").write_text(
+            "from run_dependency import VALUE\n", encoding="utf-8"
+        )
+        assert load_run_module("run_target", root / "run_target.py").VALUE == 23
+    print("fidelity strong pilot: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/intervention-response-v0/analyze_p1.py

@@ -0,0 +1,691 @@
+#!/usr/bin/env python3
+"""Prospective-repeat confirmation of the intervention-response hypothesis.
+
+P1 contains three pre-arranged, disjoint request bands per cell/load.  TP1 and
+TP4 use matched offered loads and request sequences across their MNS endpoints.
+This script asks both whether the MNS response exceeds prospective repeat noise
+and whether an early telemetry delta predicts full-run action efficacy beyond
+the corresponding external-outcome delta.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import math
+import re
+import sys
+from collections import defaultdict
+from pathlib import Path
+from statistics import fmean
+from typing import Any, Iterable, Mapping
+
+
+HERE = Path(__file__).resolve().parent
+COMMON_STATE_DIR = HERE.parent / "telemetry-residual"
+sys.path.insert(0, str(COMMON_STATE_DIR))
+
+from common_state import load_jsonl, summarize_engine  # noqa: E402
+
+
+def _load_v0():
+    spec = importlib.util.spec_from_file_location(
+        "intervention_response_phase6_v0", HERE / "analyze_phase6.py"
+    )
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+V0 = _load_v0()
+SCHEMA = "intervention-response-p1-confirmation-v1"
+HORIZONS_S = V0.HORIZONS_S
+EXPECTED_ACTION_PAIRS = 12
+EXPECTED_REPEAT_PAIRS = 24
+MIN_EFFICACY_CLASS = 4
+MIN_EFFICACY_BALANCED_ACCURACY = 0.75
+MIN_EFFICACY_DELTA_OVER_OUTCOME = 0.15
+OUTCOME_FEATURES = (
+    "admitted_fraction",
+    "completed_over_admitted",
+    "completed_pass_rate",
+    "completed_fail_fraction_of_total",
+    "outstanding_over_admitted",
+    "ttft_max_over_slo_max",
+    "ttft_mean_over_slo_max",
+    "tpot_max_over_slo",
+    "tpot_mean_over_slo",
+    "admitted_input_tokens_mean_over_limit",
+)
+RUN_PATTERN = re.compile(r"^(low|high)-rep([123])$")
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def _prefix_outcome(
+    result: Mapping[str, Any],
+    requests: list[dict[str, Any]],
+    horizon_s: float,
+) -> dict[str, float]:
+    admitted = [request for request in requests if float(request["arrival_s"]) <= horizon_s]
+    completed = [
+        request
+        for request in requests
+        if request.get("completed_elapsed_s") is not None
+        and float(request["completed_elapsed_s"]) <= horizon_s
+    ]
+    if not admitted:
+        raise ValueError("prefix contains no admitted request")
+    admitted_ids = {str(request["request_id"]) for request in admitted}
+    if any(str(request["request_id"]) not in admitted_ids for request in completed):
+        raise ValueError("completed request was not admitted in the prefix")
+    passed = sum(bool(request["slo_pass"]) for request in completed)
+    ttft = [float(request["ttft_ms"]) for request in completed]
+    tpot = [float(request["tpot_ms"]) for request in completed]
+    total = int(result["selection"]["count"])
+    if total != len(requests):
+        raise ValueError("request JSONL count does not match the result")
+    return {
+        "admitted_fraction": len(admitted) / total,
+        "completed_over_admitted": len(completed) / len(admitted),
+        "completed_pass_rate": passed / max(1, len(completed)),
+        "completed_fail_fraction_of_total": (len(completed) - passed) / total,
+        "outstanding_over_admitted": (len(admitted) - len(completed)) / len(admitted),
+        "ttft_max_over_slo_max": max(ttft, default=0.0) / 6000.0,
+        "ttft_mean_over_slo_max": fmean(ttft) / 6000.0 if ttft else 0.0,
+        "tpot_max_over_slo": max(tpot, default=0.0) / 50.0,
+        "tpot_mean_over_slo": fmean(tpot) / 50.0 if tpot else 0.0,
+        "admitted_input_tokens_mean_over_limit": fmean(
+            float(request["raw_input_tokens"]) for request in admitted
+        )
+        / 8192.0,
+    }
+
+
+def load_trials(
+    run_root: Path,
+    *,
+    horizons_s: tuple[float, ...] = HORIZONS_S,
+) -> tuple[dict[float, list[dict[str, Any]]], list[dict[str, Any]]]:
+    by_horizon = {horizon: [] for horizon in horizons_s}
+    streams = []
+    for cell_dir in sorted((run_root / "cells").iterdir()):
+        if not cell_dir.is_dir():
+            continue
+        stream_paths = sorted((cell_dir / "opprof").glob("*.jsonl"))
+        if len(stream_paths) != 1:
+            raise ValueError(f"{cell_dir}: expected one Layer-1 stream")
+        stream_path = stream_paths[0]
+        stream = load_jsonl(stream_path)
+        streams.append(
+            {
+                "path": str(stream_path.resolve()),
+                "sha256": sha256_file(stream_path),
+                "bytes": stream_path.stat().st_size,
+            }
+        )
+        for run_dir in sorted(cell_dir.iterdir()):
+            match = RUN_PATTERN.match(run_dir.name)
+            if match is None:
+                continue
+            level, replicate_text = match.groups()
+            replicate = int(replicate_text)
+            result_path = run_dir / "result.json"
+            requests_path = run_dir / "requests.jsonl"
+            result = json.loads(result_path.read_text(encoding="utf-8"))
+            requests = load_jsonl(requests_path)
+            elapsed_s = float(result["interval"]["elapsed_s"])
+            start_ns = int(result["interval"]["start_mono_ns"])
+            for horizon_s in horizons_s:
+                if elapsed_s < horizon_s:
+                    raise ValueError(
+                        f"{result_path}: elapsed {elapsed_s} shorter than {horizon_s}s"
+                    )
+                state = V0.flatten_state(
+                    summarize_engine(
+                        stream,
+                        start_ns=start_ns,
+                        end_ns=start_ns + int(horizon_s * 1e9),
+                        request_count=int(result["selection"]["count"]),
+                    )
+                )
+                by_horizon[horizon_s].append(
+                    {
+                        "trial_id": str(result_path.relative_to(run_root)),
+                        "cell": str(result["cell"]),
+                        "tp": int(result["tp"]),
+                        "mns": int(result["mns"]),
+                        "level": level,
+                        "replicate": replicate,
+                        "offered_rate_per_gpu": float(
+                            result["selection"]["offered_req_s_per_gpu"]
+                        ),
+                        "request_hash": str(
+                            result["selection"]["request_id_order_sha256"]
+                        ),
+                        "request_count": int(result["selection"]["count"]),
+                        "result_sha256": sha256_file(result_path),
+                        "requests_sha256": sha256_file(requests_path),
+                        "full_pass_rate": float(result["pass_rate"]),
+                        "full_feasible": bool(result["feasible"]),
+                        "early_stopped": bool(result["early_stopped"]),
+                        "state": state,
+                        "outcome": _prefix_outcome(result, requests, horizon_s),
+                    }
+                )
+    return by_horizon, streams
+
+
+def validate_manifest(
+    trials: list[dict[str, Any]], manifest_path: Path
+) -> dict[str, Any]:
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    if manifest.get("schema") != "fidelity-prefix-pilot-manifest-v1":
+        raise ValueError("unexpected P1 manifest schema")
+    cells = manifest.get("cells")
+    if not isinstance(cells, dict):
+        raise ValueError("P1 manifest has no cell mapping")
+    seen = set()
+    for trial in trials:
+        key = (trial["cell"], trial["level"], trial["replicate"])
+        if key in seen:
+            raise ValueError(f"duplicate P1 trial identity: {key}")
+        seen.add(key)
+        try:
+            cell = cells[trial["cell"]]
+            selection = cell["targets"][trial["level"]]["selections"][
+                f"{trial['level']}{trial['replicate']}"
+            ]
+        except (KeyError, TypeError) as error:
+            raise ValueError(f"trial is absent from P1 manifest: {key}") from error
+        if int(cell["tp"]) != trial["tp"] or int(cell["mns"]) != trial["mns"]:
+            raise ValueError(f"trial config disagrees with P1 manifest: {key}")
+        if str(selection["request_id_order_sha256"]) != trial["request_hash"]:
+            raise ValueError(f"trial request hash disagrees with P1 manifest: {key}")
+        if int(selection["selected_count"]) != trial["request_count"]:
+            raise ValueError(f"trial request count disagrees with P1 manifest: {key}")
+        if not math.isclose(
+            float(selection["offered_req_s_per_gpu"]),
+            trial["offered_rate_per_gpu"],
+            rel_tol=0.0,
+            abs_tol=1e-12,
+        ):
+            raise ValueError(f"trial offered load disagrees with P1 manifest: {key}")
+    expected = {
+        (cell_name, level, replicate)
+        for cell_name in cells
+        for level in ("low", "high")
+        for replicate in (1, 2, 3)
+    }
+    if seen != expected:
+        missing = sorted(expected - seen)
+        unexpected = sorted(seen - expected)
+        raise ValueError(
+            f"P1 trial/manifest coverage mismatch: missing={missing}, "
+            f"unexpected={unexpected}"
+        )
+    return {
+        "schema": str(manifest["schema"]),
+        "expected_trials": len(expected),
+        "matched_trials": len(seen),
+    }
+
+
+def _delta(
+    source: Mapping[str, Any],
+    target: Mapping[str, Any],
+    features: Iterable[str],
+) -> dict[str, float]:
+    return {
+        feature: float(target[feature]) - float(source[feature])
+        for feature in features
+    }
+
+
+def _action_pair(source: Mapping[str, Any], target: Mapping[str, Any]) -> dict[str, Any]:
+    if source["tp"] != target["tp"]:
+        raise ValueError("action endpoints changed TP")
+    if source["level"] != target["level"] or source["replicate"] != target["replicate"]:
+        raise ValueError("action endpoints changed load role or repeat")
+    if source["request_hash"] != target["request_hash"]:
+        raise ValueError("action endpoints changed request sequence")
+    if not math.isclose(
+        source["offered_rate_per_gpu"],
+        target["offered_rate_per_gpu"],
+        rel_tol=0.0,
+        abs_tol=1e-12,
+    ):
+        raise ValueError("action endpoints changed offered load")
+    if source["mns"] >= target["mns"]:
+        raise ValueError("action must increase MNS")
+    beneficial = target["full_feasible"] and not source["full_feasible"]
+    return {
+        "kind": "matched_mns_increase",
+        "group": {
+            "tp": source["tp"],
+            "level": source["level"],
+            "replicate": source["replicate"],
+            "request_hash": source["request_hash"],
+            "offered_rate_per_gpu": source["offered_rate_per_gpu"],
+        },
+        "source": {
+            key: source[key]
+            for key in (
+                "trial_id",
+                "result_sha256",
+                "requests_sha256",
+                "cell",
+                "mns",
+                "full_pass_rate",
+                "full_feasible",
+                "early_stopped",
+            )
+        },
+        "target": {
+            key: target[key]
+            for key in (
+                "trial_id",
+                "result_sha256",
+                "requests_sha256",
+                "cell",
+                "mns",
+                "full_pass_rate",
+                "full_feasible",
+                "early_stopped",
+            )
+        },
+        "delta_state": _delta(source["state"], target["state"], V0.ALL_FEATURES),
+        "delta_outcome": _delta(source["outcome"], target["outcome"], OUTCOME_FEATURES),
+        "full_action_efficacy": int(beneficial),
+        "full_feasibility_transition": (
+            f"{str(source['full_feasible']).lower()}->"
+            f"{str(target['full_feasible']).lower()}"
+        ),
+    }
+
+
+def _repeat_pair(source: Mapping[str, Any], target: Mapping[str, Any]) -> dict[str, Any]:
+    if source["cell"] != target["cell"] or source["level"] != target["level"]:
+        raise ValueError("repeat endpoints changed config or load role")
+    if target["replicate"] != source["replicate"] + 1:
+        raise ValueError("repeat endpoints are not consecutive pre-arranged bands")
+    if not math.isclose(
+        source["offered_rate_per_gpu"],
+        target["offered_rate_per_gpu"],
+        rel_tol=0.0,
+        abs_tol=1e-12,
+    ):
+        raise ValueError("repeat endpoints changed offered load")
+    return {
+        "kind": "same_config_workload_repeat",
+        "group": {
+            "cell": source["cell"],
+            "tp": source["tp"],
+            "mns": source["mns"],
+            "level": source["level"],
+            "source_replicate": source["replicate"],
+            "target_replicate": target["replicate"],
+        },
+        "source": {
+            key: source[key]
+            for key in ("trial_id", "result_sha256", "requests_sha256")
+        },
+        "target": {
+            key: target[key]
+            for key in ("trial_id", "result_sha256", "requests_sha256")
+        },
+        "delta_state": _delta(source["state"], target["state"], V0.ALL_FEATURES),
+        "delta_outcome": _delta(source["outcome"], target["outcome"], OUTCOME_FEATURES),
+    }
+
+
+def build_pairs(
+    trials: list[dict[str, Any]],
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+    action_groups: dict[tuple[Any, ...], list[dict[str, Any]]] = defaultdict(list)
+    repeat_groups: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
+    for trial in trials:
+        action_groups[
+            (
+                trial["tp"],
+                trial["level"],
+                trial["replicate"],
+                trial["request_hash"],
+                trial["offered_rate_per_gpu"],
+            )
+        ].append(trial)
+        repeat_groups[(trial["cell"], trial["level"])].append(trial)
+
+    actions = []
+    for group in action_groups.values():
+        if len(group) != 2:
+            continue
+        source, target = sorted(group, key=lambda trial: trial["mns"])
+        actions.append(_action_pair(source, target))
+
+    repeats = []
+    for group in repeat_groups.values():
+        ordered = sorted(group, key=lambda trial: trial["replicate"])
+        if len(ordered) != 3:
+            raise ValueError("each prospective repeat group must contain three runs")
+        repeats.extend(
+            _repeat_pair(source, target)
+            for source, target in zip(ordered, ordered[1:], strict=False)
+        )
+    return actions, repeats
+
+
+def _balanced_accuracy(labels: list[int], predictions: list[int]) -> float:
+    positive = [prediction for label, prediction in zip(labels, predictions) if label == 1]
+    negative = [prediction for label, prediction in zip(labels, predictions) if label == 0]
+    if not positive or not negative:
+        raise ValueError("balanced accuracy requires both classes")
+    sensitivity = sum(prediction == 1 for prediction in positive) / len(positive)
+    specificity = sum(prediction == 0 for prediction in negative) / len(negative)
+    return (sensitivity + specificity) / 2.0
+
+
+def _threshold_candidates(values: list[float]) -> list[float]:
+    unique = sorted(set(values))
+    if len(unique) == 1:
+        return [unique[0] - 1.0, unique[0], unique[0] + 1.0]
+    scale = max(1.0, max(abs(value) for value in unique))
+    candidates = [unique[0] - scale * 1e-6]
+    candidates.extend(
+        (left + right) / 2.0
+        for left, right in zip(unique, unique[1:], strict=False)
+    )
+    candidates.append(unique[-1] + scale * 1e-6)
+    return candidates
+
+
+def _fit_threshold(values: list[float], labels: list[int]) -> tuple[float, int, float]:
+    best: tuple[float, int, float, float] | None = None
+    for threshold in _threshold_candidates(values):
+        for direction in (-1, 1):
+            predictions = [int(direction * (value - threshold) >= 0.0) for value in values]
+            balanced = _balanced_accuracy(labels, predictions)
+            accuracy = sum(
+                prediction == label
+                for prediction, label in zip(predictions, labels, strict=True)
+            ) / len(labels)
+            candidate = (balanced, accuracy, -abs(threshold), float(direction))
+            if best is None or candidate > best:
+                best = candidate
+                selected_threshold = threshold
+                selected_direction = direction
+    assert best is not None
+    return selected_threshold, selected_direction, best[0]
+
+
+def one_feature_leave_repeat_out(
+    actions: list[dict[str, Any]],
+    *,
+    delta_key: str,
+    features: tuple[str, ...],
+) -> dict[str, Any]:
+    labels = [int(pair["full_action_efficacy"]) for pair in actions]
+    results = {}
+    for feature in features:
+        predictions = []
+        held_out_labels = []
+        folds = []
+        for held_out in (1, 2, 3):
+            train = [pair for pair in actions if pair["group"]["replicate"] != held_out]
+            test = [pair for pair in actions if pair["group"]["replicate"] == held_out]
+            train_values = [float(pair[delta_key][feature]) for pair in train]
+            train_labels = [int(pair["full_action_efficacy"]) for pair in train]
+            threshold, direction, train_balanced = _fit_threshold(
+                train_values, train_labels
+            )
+            test_values = [float(pair[delta_key][feature]) for pair in test]
+            test_predictions = [
+                int(direction * (value - threshold) >= 0.0) for value in test_values
+            ]
+            test_labels = [int(pair["full_action_efficacy"]) for pair in test]
+            predictions.extend(test_predictions)
+            held_out_labels.extend(test_labels)
+            folds.append(
+                {
+                    "held_out_replicate": held_out,
+                    "threshold": threshold,
+                    "direction": direction,
+                    "train_balanced_accuracy": train_balanced,
+                    "test_labels": test_labels,
+                    "test_predictions": test_predictions,
+                }
+            )
+        balanced = _balanced_accuracy(held_out_labels, predictions)
+        accuracy = sum(
+            prediction == label
+            for prediction, label in zip(predictions, held_out_labels, strict=True)
+        ) / len(held_out_labels)
+        results[feature] = {
+            "balanced_accuracy": balanced,
+            "accuracy": accuracy,
+            "folds": folds,
+        }
+    best_feature = max(
+        results,
+        key=lambda feature: (
+            results[feature]["balanced_accuracy"],
+            results[feature]["accuracy"],
+            feature,
+        ),
+    )
+    return {
+        "labels": V0.numeric(labels),
+        "positive": sum(labels),
+        "negative": len(labels) - sum(labels),
+        "features": results,
+        "best_feature": best_feature,
+        "best_balanced_accuracy": results[best_feature]["balanced_accuracy"],
+        "best_accuracy": results[best_feature]["accuracy"],
+    }
+
+
+def analyze_horizon(trials: list[dict[str, Any]], horizon_s: float) -> dict[str, Any]:
+    actions, repeats = build_pairs(trials)
+    response = V0.response_statistics(actions, repeats)
+    qualifying_response = sorted(
+        feature for feature, item in response.items() if item["qualifies"]
+    )
+    outcome_cv = one_feature_leave_repeat_out(
+        actions,
+        delta_key="delta_outcome",
+        features=OUTCOME_FEATURES,
+    )
+    telemetry_cv = one_feature_leave_repeat_out(
+        actions,
+        delta_key="delta_state",
+        features=V0.GATE_FEATURES,
+    )
+    outcome_best = float(outcome_cv["best_balanced_accuracy"])
+    efficacy_qualifying = sorted(
+        feature
+        for feature, item in telemetry_cv["features"].items()
+        if item["balanced_accuracy"] >= MIN_EFFICACY_BALANCED_ACCURACY
+        and item["balanced_accuracy"]
+        >= outcome_best + MIN_EFFICACY_DELTA_OVER_OUTCOME
+    )
+    action_hashes_match = all(
+        pair["group"]["request_hash"] for pair in actions
+    )
+    labels = [int(pair["full_action_efficacy"]) for pair in actions]
+    invariants = {
+        "expected_action_pair_count": len(actions) == EXPECTED_ACTION_PAIRS,
+        "expected_repeat_pair_count": len(repeats) == EXPECTED_REPEAT_PAIRS,
+        "matched_action_request_hashes": action_hashes_match,
+        "efficacy_label_balance": (
+            sum(labels) >= MIN_EFFICACY_CLASS
+            and len(labels) - sum(labels) >= MIN_EFFICACY_CLASS
+        ),
+        "finite_deltas": all(
+            math.isfinite(value)
+            for pair in [*actions, *repeats]
+            for values in (pair["delta_state"], pair["delta_outcome"])
+            for value in values.values()
+        ),
+        "probabilities_bounded": all(
+            0.0 <= trial["outcome"][feature] <= 1.0
+            for trial in trials
+            for feature in (
+                "admitted_fraction",
+                "completed_over_admitted",
+                "completed_pass_rate",
+                "completed_fail_fraction_of_total",
+                "outstanding_over_admitted",
+                "admitted_input_tokens_mean_over_limit",
+            )
+        ),
+    }
+    red_flags = [name for name, passed in invariants.items() if not passed]
+    transitions = defaultdict(int)
+    for pair in actions:
+        transitions[pair["full_feasibility_transition"]] += 1
+    return {
+        "horizon_s": horizon_s,
+        "actions": actions,
+        "repeats": repeats,
+        "response_statistics": response,
+        "qualifying_response_features": qualifying_response,
+        "efficacy": {
+            "outcome_delta": outcome_cv,
+            "telemetry_delta": telemetry_cv,
+            "telemetry_qualifying_features": efficacy_qualifying,
+            "minimum_balanced_accuracy": MIN_EFFICACY_BALANCED_ACCURACY,
+            "minimum_delta_over_best_outcome": MIN_EFFICACY_DELTA_OVER_OUTCOME,
+            "feasibility_transitions": dict(sorted(transitions.items())),
+        },
+        "sanity": {
+            "trials": len(trials),
+            "action_pairs": len(actions),
+            "repeat_pairs": len(repeats),
+            "invariants": invariants,
+            "red_flags": red_flags,
+        },
+    }
+
+
+def audit(*, run_root: Path, manifest_path: Path, output_path: Path) -> dict[str, Any]:
+    trials_by_horizon, streams = load_trials(run_root)
+    manifest_validation = validate_manifest(
+        trials_by_horizon[min(trials_by_horizon)], manifest_path
+    )
+    horizons = {
+        str(int(horizon)): analyze_horizon(trials, horizon)
+        for horizon, trials in sorted(trials_by_horizon.items())
+    }
+    red_flags = sorted(
+        {
+            flag
+            for horizon in horizons.values()
+            for flag in horizon["sanity"]["red_flags"]
+        }
+    )
+    stable_response = sorted(
+        set.intersection(
+            *(
+                set(horizon["qualifying_response_features"])
+                for horizon in horizons.values()
+            )
+        )
+    )
+    stable_efficacy = sorted(
+        set.intersection(
+            *(
+                set(horizon["efficacy"]["telemetry_qualifying_features"])
+                for horizon in horizons.values()
+            )
+        )
+    )
+    if red_flags:
+        decision = "STOP_DATA_INVALID"
+    elif len(stable_response) < V0.MIN_STABLE_FEATURES:
+        decision = "STOP_NO_PROSPECTIVE_RESPONSE"
+    elif not stable_efficacy:
+        decision = "STOP_NO_INCREMENTAL_TUNING_SIGNAL"
+    else:
+        decision = "OPEN_MATCHED_GPU_PILOT"
+    payload = {
+        "schema": SCHEMA,
+        "status": "COMPLETE",
+        "decision": decision,
+        "claim_boundary": (
+            "Development-only confirmation on an already-consumed P1 task. "
+            "Passing can open a newly registered matched pilot but cannot be "
+            "reported as held-out tuning evidence."
+        ),
+        "frozen_gate": {
+            "response_thresholds_identical_to_phase6_v0": True,
+            "expected_action_pairs": EXPECTED_ACTION_PAIRS,
+            "expected_repeat_pairs": EXPECTED_REPEAT_PAIRS,
+            "minimum_stable_response_features": V0.MIN_STABLE_FEATURES,
+            "minimum_efficacy_class": MIN_EFFICACY_CLASS,
+            "minimum_efficacy_balanced_accuracy": MIN_EFFICACY_BALANCED_ACCURACY,
+            "minimum_efficacy_delta_over_best_outcome": (
+                MIN_EFFICACY_DELTA_OVER_OUTCOME
+            ),
+        },
+        "stable_response_features": stable_response,
+        "stable_incremental_efficacy_features": stable_efficacy,
+        "horizons": horizons,
+        "provenance": {
+            "analysis_script": str(Path(__file__).resolve()),
+            "analysis_script_sha256": sha256_file(Path(__file__).resolve()),
+            "phase6_v0_script_sha256": sha256_file(HERE / "analyze_phase6.py"),
+            "run_root": str(run_root.resolve()),
+            "manifest": str(manifest_path.resolve()),
+            "manifest_sha256": sha256_file(manifest_path),
+            "manifest_validation": manifest_validation,
+            "streams": streams,
+        },
+        "sanity": {
+            "stream_count": len(streams),
+            "stream_bytes": V0.numeric(item["bytes"] for item in streams),
+            "red_flags": red_flags,
+        },
+    }
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    return payload
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    payload = audit(
+        run_root=args.run_root,
+        manifest_path=args.manifest,
+        output_path=args.output,
+    )
+    print(
+        json.dumps(
+            {
+                "decision": payload["decision"],
+                "stable_response_features": payload["stable_response_features"],
+                "stable_incremental_efficacy_features": payload[
+                    "stable_incremental_efficacy_features"
+                ],
+                "sanity": payload["sanity"],
+            },
+            indent=2,
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/intervention-response-v0/analyze_phase6.py

@@ -0,0 +1,520 @@
+#!/usr/bin/env python3
+"""Audit whether a controlled knob change produces identifiable telemetry deltas.
+
+This is a development-only feasibility audit.  It compares adjacent MNS
+interventions at an identical TP, offered-load anchor, and request sequence
+against same-config primary/confirmation repeat noise.  It does not claim that
+the observed response is causal or that it improves an end-to-end tuner.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import math
+import sys
+from collections import defaultdict
+from pathlib import Path
+from statistics import median
+from typing import Any, Iterable, Mapping
+
+
+HERE = Path(__file__).resolve().parent
+COMMON_STATE_DIR = HERE.parent / "telemetry-residual"
+sys.path.insert(0, str(COMMON_STATE_DIR))
+
+from common_state import load_jsonl, summarize_engine  # noqa: E402
+
+
+SCHEMA = "intervention-response-audit-v0"
+HORIZONS_S = (5.0, 10.0)
+GATE_FEATURES = (
+    "scheduler_steps_per_s",
+    "decode_batch_size.mean",
+    "prefill_token_fraction",
+    "queue_waiting_mean",
+    "queue_running_mean",
+    "kv_usage_mean",
+    "graph_padding_fraction",
+)
+ALL_FEATURES = (
+    "scheduler_steps_per_s",
+    "batch_size.mean",
+    "batch_tokens.mean",
+    "decode_batch_size.mean",
+    "prefill_token_fraction",
+    "queue_waiting_mean",
+    "queue_running_mean",
+    "preemptions",
+    "kv_usage_mean",
+    "kv_usage_max",
+    "kv_usage_end_minus_start",
+    "graph_none_share",
+    "graph_full_share",
+    "graph_padding_fraction",
+)
+EXPECTED_ACTION_PAIRS = 17
+MIN_REPEAT_PAIRS = 20
+MIN_STABLE_FEATURES = 2
+MIN_SIGN_CONSISTENCY = 0.75
+MIN_EFFECT_TO_NOISE = 2.0
+MIN_ABOVE_NOISE_P95_FRACTION = 0.5
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def numeric(values: Iterable[float]) -> dict[str, Any]:
+    finite = [float(value) for value in values]
+    if not finite:
+        raise ValueError("numeric summary requires at least one value")
+    if any(not math.isfinite(value) for value in finite):
+        raise ValueError("numeric summary received a non-finite value")
+    return {
+        "n": len(finite),
+        "min": min(finite),
+        "max": max(finite),
+        "distinct_n": len(set(finite)),
+    }
+
+
+def quantile(values: Iterable[float], probability: float) -> float:
+    ordered = sorted(float(value) for value in values)
+    if not ordered:
+        raise ValueError("quantile requires at least one value")
+    if not 0.0 <= probability <= 1.0:
+        raise ValueError("quantile probability must be in [0, 1]")
+    position = probability * (len(ordered) - 1)
+    lower = math.floor(position)
+    upper = math.ceil(position)
+    if lower == upper:
+        return ordered[lower]
+    weight = position - lower
+    return ordered[lower] * (1.0 - weight) + ordered[upper] * weight
+
+
+def flatten_state(summary: Mapping[str, Any]) -> dict[str, float]:
+    common = summary["common"]
+    engine = summary["engine_only"]
+    state = {
+        "scheduler_steps_per_s": float(common["scheduler_steps_per_s"]),
+        "batch_size.mean": float(common["batch_size"]["mean"]),
+        "batch_tokens.mean": float(common["batch_tokens"]["mean"]),
+        "decode_batch_size.mean": float(common["decode_batch_size"]["mean"]),
+        "prefill_token_fraction": float(common["prefill_token_fraction"]),
+        "queue_waiting_mean": float(common["queue_waiting_mean"]),
+        "queue_running_mean": float(common["queue_running_mean"]),
+        "preemptions": float(common["preemptions"]),
+        "kv_usage_mean": float(engine["kv_usage_mean"]),
+        "kv_usage_max": float(engine["kv_usage_max"]),
+        "kv_usage_end_minus_start": float(engine["kv_usage_end_minus_start"]),
+        "graph_none_share": float(engine["graph_none_share"]),
+        "graph_full_share": float(engine["graph_full_share"]),
+        "graph_padding_fraction": float(engine["graph_padding_fraction"]),
+    }
+    if set(state) != set(ALL_FEATURES):
+        raise ValueError("flattened state does not match the frozen feature set")
+    if any(not math.isfinite(value) for value in state.values()):
+        raise ValueError("flattened state contains a non-finite value")
+    return state
+
+
+def _trial_role(path: Path) -> str:
+    return "confirmation" if path.parent.name.startswith("confirm-") else "primary"
+
+
+def load_trials(
+    raw_root: Path,
+    *,
+    horizons_s: tuple[float, ...] = HORIZONS_S,
+) -> tuple[dict[float, list[dict[str, Any]]], list[dict[str, Any]]]:
+    by_horizon = {horizon: [] for horizon in horizons_s}
+    stream_provenance = []
+    for cell_dir in sorted(path for path in raw_root.iterdir() if path.is_dir()):
+        streams = sorted((cell_dir / "opprof").glob("*.jsonl"))
+        if len(streams) != 1:
+            raise ValueError(f"{cell_dir}: expected exactly one Layer-1 stream")
+        stream = streams[0]
+        records = load_jsonl(stream)
+        stream_provenance.append(
+            {
+                "path": str(stream),
+                "sha256": sha256_file(stream),
+                "bytes": stream.stat().st_size,
+            }
+        )
+        result_paths = sorted(cell_dir.glob("anchor-*/result.json"))
+        result_paths.extend(sorted(cell_dir.glob("confirm-*-anchor-*/result.json")))
+        for result_path in result_paths:
+            result = json.loads(result_path.read_text(encoding="utf-8"))
+            start_ns = int(result["interval"]["start_mono_ns"])
+            elapsed_s = float(result["interval"]["elapsed_s"])
+            for horizon_s in horizons_s:
+                if elapsed_s < horizon_s:
+                    raise ValueError(
+                        f"{result_path}: elapsed {elapsed_s} is shorter than {horizon_s}s"
+                    )
+                state = flatten_state(
+                    summarize_engine(
+                        records,
+                        start_ns=start_ns,
+                        end_ns=start_ns + int(horizon_s * 1e9),
+                        request_count=int(result["selection"]["count"]),
+                    )
+                )
+                by_horizon[horizon_s].append(
+                    {
+                        "trial_id": str(result_path.relative_to(raw_root)),
+                        "result_sha256": sha256_file(result_path),
+                        "role": _trial_role(result_path),
+                        "cell": str(result["cell"]),
+                        "study_sha256": str(result["study_sha256"]),
+                        "tp": int(result["tp"]),
+                        "mns": int(result["mns"]),
+                        "anchor": float(result["anchor"]),
+                        "request_hash": str(
+                            result["selection"]["request_id_order_sha256"]
+                        ),
+                        "request_count": int(result["selection"]["count"]),
+                        "early_stopped": bool(result["early_stopped"]),
+                        "full_pass_rate": float(result["pass_rate"]),
+                        "full_feasible": bool(result["feasible"]),
+                        "state": state,
+                    }
+                )
+    return by_horizon, stream_provenance
+
+
+def _group_key(trial: Mapping[str, Any]) -> tuple[Any, ...]:
+    return (
+        trial["study_sha256"],
+        trial["tp"],
+        trial["anchor"],
+        trial["request_hash"],
+    )
+
+
+def _delta(source: Mapping[str, Any], target: Mapping[str, Any]) -> dict[str, float]:
+    return {
+        feature: float(target["state"][feature]) - float(source["state"][feature])
+        for feature in ALL_FEATURES
+    }
+
+
+def _pair(source: Mapping[str, Any], target: Mapping[str, Any], kind: str) -> dict[str, Any]:
+    if _group_key(source) != _group_key(target):
+        raise ValueError("pair endpoints do not share workload identity")
+    return {
+        "kind": kind,
+        "group": {
+            "study_sha256": source["study_sha256"],
+            "tp": source["tp"],
+            "anchor": source["anchor"],
+            "request_hash": source["request_hash"],
+        },
+        "source": {
+            "trial_id": source["trial_id"],
+            "cell": source["cell"],
+            "mns": source["mns"],
+            "early_stopped": source["early_stopped"],
+            "full_pass_rate": source["full_pass_rate"],
+            "full_feasible": source["full_feasible"],
+        },
+        "target": {
+            "trial_id": target["trial_id"],
+            "cell": target["cell"],
+            "mns": target["mns"],
+            "early_stopped": target["early_stopped"],
+            "full_pass_rate": target["full_pass_rate"],
+            "full_feasible": target["full_feasible"],
+        },
+        "delta_state": _delta(source, target),
+        "descriptive_full_outcome": {
+            "delta_pass_rate": target["full_pass_rate"] - source["full_pass_rate"],
+            "feasibility_transition": (
+                f"{str(source['full_feasible']).lower()}->"
+                f"{str(target['full_feasible']).lower()}"
+            ),
+        },
+    }
+
+
+def build_pairs(
+    trials: list[dict[str, Any]],
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+    primary_groups: dict[tuple[Any, ...], list[dict[str, Any]]] = defaultdict(list)
+    primary_by_cell_anchor: dict[tuple[Any, ...], dict[str, Any]] = {}
+    confirmations = []
+    for trial in trials:
+        if trial["role"] == "primary":
+            primary_groups[_group_key(trial)].append(trial)
+            primary_by_cell_anchor[
+                (trial["cell"], trial["anchor"], trial["request_hash"])
+            ] = trial
+        else:
+            confirmations.append(trial)
+
+    actions = []
+    for group in primary_groups.values():
+        ordered = sorted(group, key=lambda item: item["mns"])
+        for source, target in zip(ordered, ordered[1:], strict=False):
+            if target["mns"] == source["mns"] * 2:
+                actions.append(_pair(source, target, "mns_increase"))
+
+    repeats = []
+    for confirmation in confirmations:
+        key = (
+            confirmation["cell"],
+            confirmation["anchor"],
+            confirmation["request_hash"],
+        )
+        primary = primary_by_cell_anchor.get(key)
+        if primary is None:
+            raise ValueError(f"{confirmation['trial_id']}: missing matched primary")
+        if primary["mns"] != confirmation["mns"]:
+            raise ValueError("repeat endpoints changed MNS")
+        repeats.append(_pair(primary, confirmation, "same_config_repeat"))
+    return actions, repeats
+
+
+def response_statistics(
+    actions: list[dict[str, Any]],
+    repeats: list[dict[str, Any]],
+) -> dict[str, Any]:
+    statistics = {}
+    for feature in ALL_FEATURES:
+        action = [float(pair["delta_state"][feature]) for pair in actions]
+        noise = [float(pair["delta_state"][feature]) for pair in repeats]
+        action_abs = [abs(value) for value in action]
+        noise_abs = [abs(value) for value in noise]
+        positive = sum(value > 1e-12 for value in action)
+        negative = sum(value < -1e-12 for value in action)
+        zero = len(action) - positive - negative
+        nonzero = positive + negative
+        sign_consistency = max(positive, negative) / nonzero if nonzero else 0.0
+        action_median = median(action_abs)
+        noise_median = median(noise_abs)
+        noise_p95 = quantile(noise_abs, 0.95)
+        effect_to_noise = (
+            action_median / noise_median
+            if noise_median > 0
+            else (math.inf if action_median > 0 else 0.0)
+        )
+        above_noise = sum(value > noise_p95 for value in action_abs) / len(action_abs)
+        qualifies = (
+            feature in GATE_FEATURES
+            and sign_consistency >= MIN_SIGN_CONSISTENCY
+            and effect_to_noise >= MIN_EFFECT_TO_NOISE
+            and above_noise >= MIN_ABOVE_NOISE_P95_FRACTION
+        )
+        statistics[feature] = {
+            "action_delta": numeric(action),
+            "repeat_delta": numeric(noise),
+            "action_abs_median": action_median,
+            "repeat_abs_median": noise_median,
+            "repeat_abs_p95": noise_p95,
+            "effect_to_repeat_median": (
+                effect_to_noise if math.isfinite(effect_to_noise) else None
+            ),
+            "effect_to_repeat_median_is_infinite": math.isinf(effect_to_noise),
+            "action_signs": {
+                "positive": positive,
+                "negative": negative,
+                "zero": zero,
+                "consistency": sign_consistency,
+            },
+            "action_above_repeat_p95_fraction": above_noise,
+            "gate_feature": feature in GATE_FEATURES,
+            "qualifies": qualifies,
+        }
+    return statistics
+
+
+def analyze_horizon(trials: list[dict[str, Any]], horizon_s: float) -> dict[str, Any]:
+    actions, repeats = build_pairs(trials)
+    feature_statistics = response_statistics(actions, repeats)
+    qualifying = sorted(
+        feature for feature, item in feature_statistics.items() if item["qualifies"]
+    )
+    all_values = [
+        value
+        for trial in trials
+        for value in trial["state"].values()
+    ]
+    action_vectors = {
+        tuple(round(float(pair["delta_state"][feature]), 12) for feature in ALL_FEATURES)
+        for pair in actions
+    }
+    pair_invariants = {
+        "expected_action_pair_count": len(actions) == EXPECTED_ACTION_PAIRS,
+        "sufficient_repeat_pair_count": len(repeats) >= MIN_REPEAT_PAIRS,
+        "all_pair_hashes_match": all(
+            pair["group"]["request_hash"] for pair in [*actions, *repeats]
+        ),
+        "all_values_finite": all(math.isfinite(value) for value in all_values),
+        "state_vectors_not_all_identical": len(action_vectors) > 1,
+        "ratios_bounded": all(
+            0.0 <= trial["state"][feature] <= 1.0
+            for trial in trials
+            for feature in (
+                "prefill_token_fraction",
+                "kv_usage_mean",
+                "kv_usage_max",
+                "graph_none_share",
+                "graph_full_share",
+                "graph_padding_fraction",
+            )
+        ),
+        "nonnegative_counters": all(
+            trial["state"][feature] >= 0.0
+            for trial in trials
+            for feature in (
+                "scheduler_steps_per_s",
+                "batch_size.mean",
+                "batch_tokens.mean",
+                "decode_batch_size.mean",
+                "queue_waiting_mean",
+                "queue_running_mean",
+                "preemptions",
+            )
+        ),
+    }
+    red_flags = [name for name, passed in pair_invariants.items() if not passed]
+    pass_deltas = [
+        pair["descriptive_full_outcome"]["delta_pass_rate"] for pair in actions
+    ]
+    transitions = defaultdict(int)
+    for pair in actions:
+        transitions[pair["descriptive_full_outcome"]["feasibility_transition"]] += 1
+    return {
+        "horizon_s": horizon_s,
+        "actions": actions,
+        "repeats": repeats,
+        "feature_statistics": feature_statistics,
+        "qualifying_features": qualifying,
+        "descriptive_full_outcome": {
+            "delta_pass_rate": numeric(pass_deltas),
+            "positive": sum(value > 1e-12 for value in pass_deltas),
+            "negative": sum(value < -1e-12 for value in pass_deltas),
+            "zero": sum(abs(value) <= 1e-12 for value in pass_deltas),
+            "feasibility_transitions": dict(sorted(transitions.items())),
+            "limitation": (
+                "Full outcomes may use different elapsed durations when a trial "
+                "early-stopped; they are descriptive and are not a gate input."
+            ),
+        },
+        "sanity": {
+            "trials": len(trials),
+            "action_pairs": len(actions),
+            "repeat_pairs": len(repeats),
+            "distinct_action_vectors": len(action_vectors),
+            "invariants": pair_invariants,
+            "red_flags": red_flags,
+        },
+    }
+
+
+def audit(
+    *,
+    metrics_path: Path,
+    raw_root: Path,
+    output_path: Path,
+) -> dict[str, Any]:
+    trials_by_horizon, streams = load_trials(raw_root)
+    horizons = {
+        str(int(horizon)): analyze_horizon(trials, horizon)
+        for horizon, trials in sorted(trials_by_horizon.items())
+    }
+    red_flags = sorted(
+        {
+            red_flag
+            for horizon in horizons.values()
+            for red_flag in horizon["sanity"]["red_flags"]
+        }
+    )
+    stable_features = sorted(
+        set.intersection(
+            *(set(horizon["qualifying_features"]) for horizon in horizons.values())
+        )
+    )
+    if red_flags:
+        decision = "STOP_DATA_INVALID"
+    elif len(stable_features) < MIN_STABLE_FEATURES:
+        decision = "STOP_NO_IDENTIFIABLE_RESPONSE"
+    else:
+        decision = "OPEN_MATCHED_PILOT"
+    payload = {
+        "schema": SCHEMA,
+        "status": "COMPLETE",
+        "decision": decision,
+        "claim_boundary": (
+            "Development-only identifiability gate. Passing opens a controlled "
+            "real-GPU pilot; it does not establish tuning benefit or causality."
+        ),
+        "frozen_gate": {
+            "horizons_s": list(HORIZONS_S),
+            "expected_action_pairs": EXPECTED_ACTION_PAIRS,
+            "minimum_repeat_pairs": MIN_REPEAT_PAIRS,
+            "minimum_stable_features": MIN_STABLE_FEATURES,
+            "minimum_sign_consistency": MIN_SIGN_CONSISTENCY,
+            "minimum_effect_to_repeat_median": MIN_EFFECT_TO_NOISE,
+            "minimum_action_above_repeat_p95_fraction": (
+                MIN_ABOVE_NOISE_P95_FRACTION
+            ),
+            "gate_features": list(GATE_FEATURES),
+        },
+        "stable_qualifying_features": stable_features,
+        "horizons": horizons,
+        "provenance": {
+            "analysis_script": str(Path(__file__).resolve()),
+            "analysis_script_sha256": sha256_file(Path(__file__).resolve()),
+            "phase6_metrics": str(metrics_path.resolve()),
+            "phase6_metrics_sha256": sha256_file(metrics_path),
+            "raw_root": str(raw_root.resolve()),
+            "streams": streams,
+        },
+        "sanity": {
+            "stream_count": len(streams),
+            "stream_bytes": numeric(item["bytes"] for item in streams),
+            "red_flags": red_flags,
+        },
+    }
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    return payload
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--metrics", type=Path, required=True)
+    parser.add_argument("--raw-root", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    payload = audit(
+        metrics_path=args.metrics,
+        raw_root=args.raw_root,
+        output_path=args.output,
+    )
+    print(
+        json.dumps(
+            {
+                "decision": payload["decision"],
+                "stable_qualifying_features": payload[
+                    "stable_qualifying_features"
+                ],
+                "sanity": payload["sanity"],
+            },
+            indent=2,
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/intervention-response-v0/test_analysis.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import importlib.util
+import math
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def load_module():
+    spec = importlib.util.spec_from_file_location(
+        "intervention_response_v0", HERE / "analyze_phase6.py"
+    )
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+def pair(module, delta: dict[str, float]) -> dict[str, object]:
+    state = {feature: 0.0 for feature in module.ALL_FEATURES}
+    state.update(delta)
+    return {"delta_state": state}
+
+
+def main() -> None:
+    module = load_module()
+    assert module.numeric([0.0, 1.0, 1.0]) == {
+        "n": 3,
+        "min": 0.0,
+        "max": 1.0,
+        "distinct_n": 2,
+    }
+    assert math.isclose(module.quantile([0.0, 10.0], 0.95), 9.5)
+
+    actions = [
+        pair(module, {"queue_waiting_mean": -1.0 - 0.1 * index})
+        for index in range(8)
+    ]
+    repeats = [
+        pair(module, {"queue_waiting_mean": 0.01 * ((index % 3) - 1)})
+        for index in range(20)
+    ]
+    stats = module.response_statistics(actions, repeats)
+    waiting = stats["queue_waiting_mean"]
+    assert waiting["qualifies"]
+    assert waiting["action_signs"]["negative"] == 8
+    assert waiting["action_signs"]["consistency"] == 1.0
+    assert waiting["effect_to_repeat_median"] > 2.0
+    assert not stats["kv_usage_mean"]["qualifies"]
+    print("intervention response v0 analysis: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/intervention-response-v0/test_p1_analysis.py

@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def load_module():
+    spec = importlib.util.spec_from_file_location(
+        "intervention_response_p1", HERE / "analyze_p1.py"
+    )
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+def main() -> None:
+    module = load_module()
+    values = [-2.0, -1.0, 1.0, 2.0]
+    labels = [0, 0, 1, 1]
+    threshold, direction, balanced = module._fit_threshold(values, labels)
+    assert direction == 1
+    assert -1.0 < threshold < 1.0
+    assert balanced == 1.0
+    assert module._balanced_accuracy(labels, labels) == 1.0
+    assert module._balanced_accuracy(labels, [1, 1, 0, 0]) == 0.0
+    print("intervention response P1 confirmation analysis: PASS")
+
+
+if __name__ == "__main__":
+    main()

runs/intervention-response-v2/analyze_existing.py

@@ -0,0 +1,467 @@
+#!/usr/bin/env python3
+"""Audit telemetry responses over every uncensored replay decile.
+
+This corrective analysis keeps the frozen P1 pairs and thresholds, but replaces
+the absolute 5/10-second cutoff with cumulative and non-overlapping 10%-of-trace
+windows.  It deliberately reports every common decile instead of selecting the
+best-looking horizon.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import math
+from pathlib import Path
+from statistics import median
+from typing import Any, Iterable, Mapping
+
+
+HERE = Path(__file__).resolve().parent
+P1_PATH = HERE.parent / "intervention-response-v0" / "analyze_p1.py"
+SCHEMA = "intervention-response-phase-aware-existing-v2"
+DECILE_FRACTION = 0.1
+MAX_DECILES = 10
+
+
+def _load_p1():
+    spec = importlib.util.spec_from_file_location("intervention_response_p1", P1_PATH)
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+P1 = _load_p1()
+
+
+def numeric(values: Iterable[float | int]) -> dict[str, Any]:
+    finite = [float(value) for value in values]
+    result = P1.V0.numeric(finite)
+    result["median"] = median(finite)
+    return result
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def trial_directories(run_root: Path) -> list[Path]:
+    result = []
+    for cell in sorted((run_root / "cells").iterdir()):
+        if not cell.is_dir():
+            continue
+        for candidate in sorted(cell.iterdir()):
+            if candidate.is_dir() and P1.RUN_PATTERN.match(candidate.name):
+                result.append(candidate)
+    if not result:
+        raise ValueError("P1 run root contains no measured trial directories")
+    return result
+
+
+def load_metadata(run_root: Path) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+    metadata = []
+    streams = []
+    for cell in sorted((run_root / "cells").iterdir()):
+        if not cell.is_dir():
+            continue
+        stream_paths = sorted((cell / "opprof").glob("*.jsonl"))
+        if len(stream_paths) != 1:
+            raise ValueError(f"{cell}: expected one Layer-1 stream")
+        stream_path = stream_paths[0]
+        streams.append(
+            {
+                "cell": cell.name,
+                "path": str(stream_path.resolve()),
+                "sha256": sha256_file(stream_path),
+                "bytes": stream_path.stat().st_size,
+            }
+        )
+    for run_dir in trial_directories(run_root):
+        match = P1.RUN_PATTERN.match(run_dir.name)
+        assert match is not None
+        level, replicate_text = match.groups()
+        result_path = run_dir / "result.json"
+        requests_path = run_dir / "requests.jsonl"
+        result = json.loads(result_path.read_text(encoding="utf-8"))
+        selected = int(result["selection"]["count"])
+        offered = float(result["selection"]["offered_req_s"])
+        if selected <= 0 or offered <= 0.0:
+            raise ValueError(f"{result_path}: invalid selected count or offered rate")
+        metadata.append(
+            {
+                "trial_id": str(result_path.relative_to(run_root)),
+                "cell": str(result["cell"]),
+                "tp": int(result["tp"]),
+                "mns": int(result["mns"]),
+                "level": level,
+                "replicate": int(replicate_text),
+                "elapsed_s": float(result["interval"]["elapsed_s"]),
+                "trace_duration_s": round(selected / offered, 9),
+                "early_stopped": bool(result["early_stopped"]),
+                "request_count": selected,
+                "result_sha256": sha256_file(result_path),
+                "requests_sha256": sha256_file(requests_path),
+            }
+        )
+    return metadata, streams
+
+
+def common_decile_fractions(
+    *, trace_duration_s: float, minimum_elapsed_s: float
+) -> tuple[float, ...]:
+    if trace_duration_s <= 0.0 or minimum_elapsed_s <= 0.0:
+        raise ValueError("trace duration and elapsed time must be positive")
+    supported = min(
+        MAX_DECILES,
+        int(math.floor((minimum_elapsed_s / trace_duration_s) * 10.0 + 1e-12)),
+    )
+    return tuple(
+        round(index * DECILE_FRACTION, 10) for index in range(1, supported + 1)
+    )
+
+
+def _trial_record(
+    *,
+    run_root: Path,
+    run_dir: Path,
+    result: Mapping[str, Any],
+    state: dict[str, float],
+    outcome: dict[str, float],
+) -> dict[str, Any]:
+    match = P1.RUN_PATTERN.match(run_dir.name)
+    assert match is not None
+    level, replicate_text = match.groups()
+    result_path = run_dir / "result.json"
+    requests_path = run_dir / "requests.jsonl"
+    return {
+        "trial_id": str(result_path.relative_to(run_root)),
+        "cell": str(result["cell"]),
+        "tp": int(result["tp"]),
+        "mns": int(result["mns"]),
+        "level": level,
+        "replicate": int(replicate_text),
+        "offered_rate_per_gpu": float(
+            result["selection"]["offered_req_s_per_gpu"]
+        ),
+        "request_hash": str(result["selection"]["request_id_order_sha256"]),
+        "request_count": int(result["selection"]["count"]),
+        "result_sha256": sha256_file(result_path),
+        "requests_sha256": sha256_file(requests_path),
+        "full_pass_rate": float(result["pass_rate"]),
+        "full_feasible": bool(result["feasible"]),
+        "early_stopped": bool(result["early_stopped"]),
+        "state": state,
+        "outcome": outcome,
+    }
+
+
+def load_interval_trials(
+    run_root: Path,
+    intervals_s: tuple[tuple[float, float], ...],
+) -> tuple[dict[tuple[float, float], list[dict[str, Any]]], list[dict[str, Any]]]:
+    by_interval = {interval: [] for interval in intervals_s}
+    stream_provenance = []
+    for cell in sorted((run_root / "cells").iterdir()):
+        if not cell.is_dir():
+            continue
+        stream_paths = sorted((cell / "opprof").glob("*.jsonl"))
+        if len(stream_paths) != 1:
+            raise ValueError(f"{cell}: expected one Layer-1 stream")
+        stream_path = stream_paths[0]
+        stream = P1.load_jsonl(stream_path)
+        stream_provenance.append(
+            {
+                "cell": cell.name,
+                "path": str(stream_path.resolve()),
+                "sha256": sha256_file(stream_path),
+                "bytes": stream_path.stat().st_size,
+            }
+        )
+        for run_dir in sorted(cell.iterdir()):
+            if not run_dir.is_dir() or P1.RUN_PATTERN.match(run_dir.name) is None:
+                continue
+            result_path = run_dir / "result.json"
+            requests_path = run_dir / "requests.jsonl"
+            result = json.loads(result_path.read_text(encoding="utf-8"))
+            requests = P1.load_jsonl(requests_path)
+            start_ns = int(result["interval"]["start_mono_ns"])
+            elapsed_s = float(result["interval"]["elapsed_s"])
+            for interval in intervals_s:
+                start_s, end_s = interval
+                if start_s < 0.0 or end_s <= start_s:
+                    raise ValueError(f"invalid analysis interval: {interval}")
+                if elapsed_s + 1e-9 < end_s:
+                    raise ValueError(
+                        f"{result_path}: elapsed {elapsed_s} shorter than {end_s}s"
+                    )
+                state = P1.V0.flatten_state(
+                    P1.summarize_engine(
+                        stream,
+                        start_ns=start_ns + int(start_s * 1e9),
+                        end_ns=start_ns + int(end_s * 1e9),
+                        request_count=int(result["selection"]["count"]),
+                    )
+                )
+                outcome = P1._prefix_outcome(result, requests, end_s)
+                by_interval[interval].append(
+                    _trial_record(
+                        run_root=run_root,
+                        run_dir=run_dir,
+                        result=result,
+                        state=state,
+                        outcome=outcome,
+                    )
+                )
+    return by_interval, stream_provenance
+
+
+def coverage(trials: list[dict[str, Any]]) -> dict[str, Any]:
+    admitted = [float(trial["outcome"]["admitted_fraction"]) for trial in trials]
+    completed = [
+        float(trial["outcome"]["admitted_fraction"])
+        * float(trial["outcome"]["completed_over_admitted"])
+        for trial in trials
+    ]
+    return {
+        "admitted_fraction_of_total": numeric(admitted),
+        "completed_fraction_of_total": numeric(completed),
+    }
+
+
+def slim_window_analysis(
+    trials: list[dict[str, Any]], *, start_s: float, end_s: float, fraction: float
+) -> dict[str, Any]:
+    analysis = P1.analyze_horizon(trials, end_s)
+    return {
+        "start_s": start_s,
+        "end_s": end_s,
+        "end_fraction": fraction,
+        "coverage_at_end": coverage(trials),
+        "action_pairs": len(analysis["actions"]),
+        "repeat_pairs": len(analysis["repeats"]),
+        "response_statistics": analysis["response_statistics"],
+        "qualifying_response_features": analysis["qualifying_response_features"],
+        "efficacy": analysis["efficacy"],
+        "sanity": analysis["sanity"],
+    }
+
+
+def _pearson(left: list[float], right: list[float]) -> float | None:
+    if len(left) != len(right) or not left:
+        raise ValueError("Pearson inputs must be non-empty and have equal length")
+    left_mean = sum(left) / len(left)
+    right_mean = sum(right) / len(right)
+    numerator = sum(
+        (x - left_mean) * (y - right_mean)
+        for x, y in zip(left, right, strict=True)
+    )
+    left_ss = sum((x - left_mean) ** 2 for x in left)
+    right_ss = sum((y - right_mean) ** 2 for y in right)
+    if left_ss == 0.0 or right_ss == 0.0:
+        return None
+    return numerator / math.sqrt(left_ss * right_ss)
+
+
+def trajectory_summary(
+    block_trials: list[tuple[tuple[float, float], list[dict[str, Any]]]]
+) -> dict[str, Any]:
+    if not block_trials:
+        raise ValueError("trajectory requires at least one block")
+    identities = []
+    states_by_block = []
+    for interval, trials in block_trials:
+        ordered = sorted(
+            trials,
+            key=lambda trial: (trial["cell"], trial["level"], trial["replicate"]),
+        )
+        current_identities = [
+            (trial["cell"], trial["level"], trial["replicate"]) for trial in ordered
+        ]
+        if identities and current_identities != identities:
+            raise ValueError("trajectory blocks do not contain identical trials")
+        identities = current_identities
+        states_by_block.append((interval, [trial["state"] for trial in ordered]))
+
+    features = {}
+    for feature in P1.V0.ALL_FEATURES:
+        block_values = [
+            [float(state[feature]) for state in states]
+            for _interval, states in states_by_block
+        ]
+        first = block_values[0]
+        last = block_values[-1]
+        delta = [right - left for left, right in zip(first, last, strict=True)]
+        features[feature] = {
+            "block_medians": [median(values) for values in block_values],
+            "first_to_last_delta": numeric(delta),
+            "first_to_last_abs_delta": numeric(abs(value) for value in delta),
+            "first_to_last_pearson": _pearson(first, last),
+            "changed_trials": sum(abs(value) > 1e-12 for value in delta),
+        }
+    return {
+        "trial_count": len(identities),
+        "blocks": [
+            {"start_s": interval[0], "end_s": interval[1]}
+            for interval, _states in states_by_block
+        ],
+        "features": features,
+    }
+
+
+def audit(*, run_root: Path, manifest_path: Path, output_path: Path) -> dict[str, Any]:
+    metadata, metadata_streams = load_metadata(run_root)
+    durations = [float(item["trace_duration_s"]) for item in metadata]
+    elapsed = [float(item["elapsed_s"]) for item in metadata]
+    duration = median(durations)
+    deciles = common_decile_fractions(
+        trace_duration_s=duration, minimum_elapsed_s=min(elapsed)
+    )
+    if not deciles:
+        raise ValueError("no complete replay decile is shared by all trials")
+    cumulative_intervals = tuple(
+        (0.0, round(duration * fraction, 9)) for fraction in deciles
+    )
+    block_intervals = tuple(
+        (
+            round(duration * (fraction - DECILE_FRACTION), 9),
+            round(duration * fraction, 9),
+        )
+        for fraction in deciles
+    )
+    all_intervals = tuple(dict.fromkeys([*cumulative_intervals, *block_intervals]))
+    trials_by_interval, streams = load_interval_trials(run_root, all_intervals)
+    manifest_validation = P1.validate_manifest(
+        trials_by_interval[cumulative_intervals[0]], manifest_path
+    )
+
+    cumulative = []
+    blocks = []
+    for fraction, cumulative_interval, block_interval in zip(
+        deciles, cumulative_intervals, block_intervals, strict=True
+    ):
+        cumulative.append(
+            slim_window_analysis(
+                trials_by_interval[cumulative_interval],
+                start_s=cumulative_interval[0],
+                end_s=cumulative_interval[1],
+                fraction=fraction,
+            )
+        )
+        blocks.append(
+            slim_window_analysis(
+                trials_by_interval[block_interval],
+                start_s=block_interval[0],
+                end_s=block_interval[1],
+                fraction=fraction,
+            )
+        )
+
+    invariants = {
+        "expected_trial_count": len(metadata) == 36,
+        "trace_duration_consistent": max(durations) - min(durations) <= 1e-9,
+        "all_intervals_uncensored": all(
+            item["elapsed_s"] + 1e-9 >= cumulative_intervals[-1][1]
+            for item in metadata
+        ),
+        "stream_provenance_consistent": metadata_streams == streams,
+        "manifest_trials_match": (
+            manifest_validation["expected_trials"]
+            == manifest_validation["matched_trials"]
+            == len(metadata)
+        ),
+        "all_window_sanity_pass": all(
+            not item["sanity"]["red_flags"] for item in [*cumulative, *blocks]
+        ),
+    }
+    red_flags = [name for name, passed in invariants.items() if not passed]
+    complete_full_trajectory = min(elapsed) + 1e-9 >= duration
+    if red_flags:
+        decision = "STOP_DATA_INVALID"
+    elif not complete_full_trajectory:
+        decision = "REQUIRES_UNCENSORED_PHASE_AWARE_PILOT"
+    else:
+        decision = "FULL_TRAJECTORY_AVAILABLE"
+
+    payload = {
+        "schema": SCHEMA,
+        "status": "COMPLETE",
+        "decision": decision,
+        "claim_boundary": (
+            "Post-hoc corrective audit over every common replay decile. It can "
+            "diagnose horizon sensitivity but cannot establish a held-out tuning claim."
+        ),
+        "design": {
+            "decile_fraction": DECILE_FRACTION,
+            "available_deciles": list(deciles),
+            "trace_duration_s": duration,
+            "maximum_common_end_s": cumulative_intervals[-1][1],
+            "maximum_common_fraction": deciles[-1],
+            "select_best_horizon": False,
+            "cumulative_and_nonoverlapping_blocks": True,
+        },
+        "cumulative": cumulative,
+        "blocks": blocks,
+        "trajectory": trajectory_summary(
+            [(interval, trials_by_interval[interval]) for interval in block_intervals]
+        ),
+        "provenance": {
+            "analysis_script": str(Path(__file__).resolve()),
+            "analysis_script_sha256": sha256_file(Path(__file__).resolve()),
+            "p1_analysis_script": str(P1_PATH.resolve()),
+            "p1_analysis_script_sha256": sha256_file(P1_PATH),
+            "run_root": str(run_root.resolve()),
+            "manifest": str(manifest_path.resolve()),
+            "manifest_sha256": sha256_file(manifest_path),
+            "manifest_validation": manifest_validation,
+            "streams": streams,
+            "trial_inputs": metadata,
+        },
+        "sanity": {
+            "trials": len(metadata),
+            "elapsed_s": numeric(elapsed),
+            "trace_duration_s": numeric(durations),
+            "early_stopped": sum(bool(item["early_stopped"]) for item in metadata),
+            "request_count": numeric(item["request_count"] for item in metadata),
+            "stream_bytes": numeric(item["bytes"] for item in streams),
+            "invariants": invariants,
+            "red_flags": red_flags,
+        },
+    }
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    return payload
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    payload = audit(
+        run_root=args.run_root,
+        manifest_path=args.manifest,
+        output_path=args.output,
+    )
+    print(
+        json.dumps(
+            {
+                "decision": payload["decision"],
+                "design": payload["design"],
+                "sanity": payload["sanity"],
+            },
+            indent=2,
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/intervention-response-v2/analyze_pilot.py

@@ -0,0 +1,651 @@
+#!/usr/bin/env python3
+"""Analyze the uncensored 300-second phase-aware matched pilot."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.util
+import json
+import math
+from collections import defaultdict
+from pathlib import Path
+from typing import Any, Iterable, Mapping
+
+
+HERE = Path(__file__).resolve().parent
+P1_PATH = HERE.parent / "intervention-response-v0" / "analyze_p1.py"
+SCHEMA = "intervention-response-phase-aware-pilot-analysis-v3"
+EXPECTED_ACTION_PAIRS = 6
+EXPECTED_REPEAT_PAIRS = 8
+MIN_EFFICACY_CLASS = 2
+MAX_LAYER1_GAP_S = 1.0
+
+
+def _load_p1():
+    spec = importlib.util.spec_from_file_location("intervention_response_p1", P1_PATH)
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+P1 = _load_p1()
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as source:
+        for chunk in iter(lambda: source.read(1 << 20), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def numeric(values: Iterable[float | int]) -> dict[str, Any]:
+    return P1.V0.numeric(values)
+
+
+def _trial_record(
+    *,
+    run_root: Path,
+    session: Mapping[str, Any],
+    level: str,
+    result: Mapping[str, Any],
+    result_path: Path,
+    requests_path: Path,
+    state: dict[str, float],
+    outcome: dict[str, float],
+    telemetry_coverage: dict[str, float],
+) -> dict[str, Any]:
+    return {
+        "trial_id": str(result_path.relative_to(run_root)),
+        "cell": str(result["cell"]),
+        "tp": int(result["tp"]),
+        "mns": int(result["mns"]),
+        "level": level,
+        "replicate": int(session["replicate"]),
+        "offered_rate_per_gpu": float(
+            result["selection"]["offered_req_s_per_gpu"]
+        ),
+        "request_hash": str(result["selection"]["request_id_order_sha256"]),
+        "request_count": int(result["selection"]["count"]),
+        "result_sha256": sha256_file(result_path),
+        "requests_sha256": sha256_file(requests_path),
+        "full_pass_rate": float(result["pass_rate"]),
+        "full_feasible": bool(result["feasible"]),
+        "early_stopped": bool(result["early_stopped"]),
+        "state": state,
+        "outcome": outcome,
+        "telemetry_coverage": telemetry_coverage,
+    }
+
+
+def telemetry_coverage(
+    records: list[dict[str, Any]], *, start_ns: int, end_ns: int
+) -> dict[str, float]:
+    layer1 = [record for record in records if "step_index" in record]
+    timestamps = [int(record["submit_mono_ns"]) for record in layer1]
+    if timestamps != sorted(timestamps):
+        raise ValueError("Layer-1 timestamps are not monotonic")
+    selected = [timestamp for timestamp in timestamps if start_ns <= timestamp <= end_ns]
+    if not selected:
+        raise ValueError("Layer-1 coverage interval contains no records")
+    internal_gaps = [
+        (right - left) / 1e9
+        for left, right in zip(selected, selected[1:], strict=False)
+    ]
+    coverage = {
+        "start_gap_s": (selected[0] - start_ns) / 1e9,
+        "end_gap_s": (end_ns - selected[-1]) / 1e9,
+        "max_internal_gap_s": max(internal_gaps, default=0.0),
+    }
+    if any(value > MAX_LAYER1_GAP_S for value in coverage.values()):
+        raise ValueError(f"Layer-1 coverage gap exceeds {MAX_LAYER1_GAP_S}s: {coverage}")
+    return coverage
+
+
+def validate_result_against_manifest(
+    *,
+    result: Mapping[str, Any],
+    selection: Mapping[str, Any],
+    session: Mapping[str, Any],
+    level: str,
+    expected_duration_s: float,
+) -> None:
+    identity = f"{session['session']}:{level}"
+    if int(result["mns"]) != int(session["mns"]) or int(result["tp"]) != 4:
+        raise ValueError(f"config mismatch: {identity}")
+    if bool(result["early_stopped"]):
+        raise ValueError(f"early-stopped measured result: {identity}")
+    if result.get("slo_early_stop_disabled") is not True:
+        raise ValueError(f"SLO early stop was enabled: {identity}")
+    if float(result["interval"]["elapsed_s"]) + 1e-9 < expected_duration_s:
+        raise ValueError(f"result does not cover full arrival window: {identity}")
+    if int(result["selection"]["count"]) != int(selection["selected_count"]):
+        raise ValueError(f"selection count mismatch: {identity}")
+    for result_key, manifest_key in (
+        ("request_id_order_sha256", "request_id_order_sha256"),
+        ("arrival_order_sha256", "arrival_order_sha256"),
+        ("raw_length_order_sha256", "input_length_order_sha256"),
+    ):
+        if result["selection"][result_key] != selection[manifest_key]:
+            raise ValueError(f"selection hash mismatch {result_key}: {identity}")
+    if int(result["observed_count"]) != int(selection["selected_count"]):
+        raise ValueError(f"request accounting mismatch: {identity}")
+
+
+def load_interval_trials(
+    *,
+    run_root: Path,
+    manifest: Mapping[str, Any],
+    intervals_s: tuple[tuple[float, float], ...],
+) -> tuple[dict[tuple[float, float], list[dict[str, Any]]], list[dict[str, Any]]]:
+    by_interval = {interval: [] for interval in intervals_s}
+    streams = []
+    duration_s = float(manifest["engine"]["duration_s"])
+    for session in manifest["sessions"]:
+        session_root = run_root / "sessions" / str(session["session"])
+        stream_paths = sorted((session_root / "opprof").glob("*.jsonl"))
+        if len(stream_paths) != 1:
+            raise ValueError(f"{session_root}: expected one Layer-1 stream")
+        stream_path = stream_paths[0]
+        stream = P1.load_jsonl(stream_path)
+        streams.append(
+            {
+                "session": str(session["session"]),
+                "path": str(stream_path.resolve()),
+                "sha256": sha256_file(stream_path),
+                "bytes": stream_path.stat().st_size,
+            }
+        )
+        repetition = manifest["repetitions"][str(session["replicate"])]
+        for level, selection in repetition["selections"].items():
+            result_path = session_root / level / "result.json"
+            requests_path = session_root / level / "requests.jsonl"
+            result = json.loads(result_path.read_text(encoding="utf-8"))
+            requests = P1.load_jsonl(requests_path)
+            validate_result_against_manifest(
+                result=result,
+                selection=selection,
+                session=session,
+                level=level,
+                expected_duration_s=duration_s,
+            )
+            start_ns = int(result["interval"]["start_mono_ns"])
+            for interval in intervals_s:
+                start_s, end_s = interval
+                interval_start_ns = start_ns + int(start_s * 1e9)
+                interval_end_ns = start_ns + int(end_s * 1e9)
+                coverage = telemetry_coverage(
+                    stream, start_ns=interval_start_ns, end_ns=interval_end_ns
+                )
+                state = P1.V0.flatten_state(
+                    P1.summarize_engine(
+                        stream,
+                        start_ns=interval_start_ns,
+                        end_ns=interval_end_ns,
+                        request_count=int(result["selection"]["count"]),
+                    )
+                )
+                outcome = P1._prefix_outcome(result, requests, end_s)
+                by_interval[interval].append(
+                    _trial_record(
+                        run_root=run_root,
+                        session=session,
+                        level=level,
+                        result=result,
+                        result_path=result_path,
+                        requests_path=requests_path,
+                        state=state,
+                        outcome=outcome,
+                        telemetry_coverage=coverage,
+                    )
+                )
+    return by_interval, streams
+
+
+def analyze_window(
+    trials: list[dict[str, Any]], *, start_s: float, end_s: float, fraction: float
+) -> dict[str, Any]:
+    actions, repeats = P1.build_pairs(trials)
+    response = P1.V0.response_statistics(actions, repeats)
+    response_qualifying = sorted(
+        feature for feature, item in response.items() if item["qualifies"]
+    )
+    labels = [int(pair["full_action_efficacy"]) for pair in actions]
+    cross_validation_possible = all(
+        set(
+            int(pair["full_action_efficacy"])
+            for pair in actions
+            if pair["group"]["replicate"] != held_out
+        )
+        == {0, 1}
+        for held_out in (1, 2, 3)
+    )
+    if cross_validation_possible:
+        outcome_cv = P1.one_feature_leave_repeat_out(
+            actions, delta_key="delta_outcome", features=P1.OUTCOME_FEATURES
+        )
+        telemetry_cv = P1.one_feature_leave_repeat_out(
+            actions, delta_key="delta_state", features=P1.V0.GATE_FEATURES
+        )
+        outcome_best = float(outcome_cv["best_balanced_accuracy"])
+        efficacy_qualifying = sorted(
+            feature
+            for feature, item in telemetry_cv["features"].items()
+            if item["balanced_accuracy"] >= P1.MIN_EFFICACY_BALANCED_ACCURACY
+            and item["balanced_accuracy"]
+            >= outcome_best + P1.MIN_EFFICACY_DELTA_OVER_OUTCOME
+        )
+    else:
+        unavailable = {
+            "status": "UNAVAILABLE",
+            "reason": "each leave-one-repetition-out train fold needs both classes",
+        }
+        outcome_cv = unavailable
+        telemetry_cv = unavailable
+        efficacy_qualifying = []
+    transitions = defaultdict(int)
+    for pair in actions:
+        transitions[pair["full_feasibility_transition"]] += 1
+    admitted = [float(trial["outcome"]["admitted_fraction"]) for trial in trials]
+    completed = [
+        float(trial["outcome"]["admitted_fraction"])
+        * float(trial["outcome"]["completed_over_admitted"])
+        for trial in trials
+    ]
+    state_vectors = {
+        tuple(round(float(trial["state"][feature]), 12) for feature in P1.V0.ALL_FEATURES)
+        for trial in trials
+    }
+    per_cell_vectors: dict[str, set[tuple[float, ...]]] = defaultdict(set)
+    for trial in trials:
+        per_cell_vectors[str(trial["cell"])].add(
+            tuple(
+                round(float(trial["state"][feature]), 12)
+                for feature in P1.V0.ALL_FEATURES
+            )
+        )
+    ratio_features = (
+        "prefill_token_fraction",
+        "kv_usage_mean",
+        "kv_usage_max",
+        "graph_none_share",
+        "graph_full_share",
+        "graph_padding_fraction",
+    )
+    nonnegative_features = tuple(
+        feature
+        for feature in P1.V0.ALL_FEATURES
+        if feature != "kv_usage_end_minus_start"
+    )
+    action_metadata = [
+        {
+            "group": pair["group"],
+            "source": pair["source"],
+            "target": pair["target"],
+            "full_action_efficacy": pair["full_action_efficacy"],
+            "full_feasibility_transition": pair["full_feasibility_transition"],
+            "delta_state": pair["delta_state"],
+            "delta_outcome": pair["delta_outcome"],
+        }
+        for pair in actions
+    ]
+    invariants = {
+        "expected_action_pair_count": len(actions) == EXPECTED_ACTION_PAIRS,
+        "expected_repeat_pair_count": len(repeats) == EXPECTED_REPEAT_PAIRS,
+        "finite_deltas": all(
+            math.isfinite(value)
+            for pair in [*actions, *repeats]
+            for key in ("delta_state", "delta_outcome")
+            for value in pair[key].values()
+        ),
+        "all_results_uncensored": all(not trial["early_stopped"] for trial in trials),
+        "state_vectors_not_all_identical": len(state_vectors) > 1,
+        "per_cell_state_vectors_not_all_identical": all(
+            len(vectors) > 1 for vectors in per_cell_vectors.values()
+        ),
+        "ratios_bounded": all(
+            0.0 <= float(trial["state"][feature]) <= 1.0
+            for trial in trials
+            for feature in ratio_features
+        ),
+        "nonnegative_counters": all(
+            float(trial["state"][feature]) >= 0.0
+            for trial in trials
+            for feature in nonnegative_features
+        ),
+        "layer1_boundary_and_internal_gaps_bounded": all(
+            value <= MAX_LAYER1_GAP_S
+            for trial in trials
+            for value in trial["telemetry_coverage"].values()
+        ),
+    }
+    return {
+        "start_s": start_s,
+        "end_s": end_s,
+        "end_fraction": fraction,
+        "coverage_at_end": {
+            "admitted_fraction_of_total": numeric(admitted),
+            "completed_fraction_of_total": numeric(completed),
+        },
+        "action_pairs": len(actions),
+        "repeat_pairs": len(repeats),
+        "actions": action_metadata,
+        "trial_sanity": [
+            {
+                "trial_id": trial["trial_id"],
+                "cell": trial["cell"],
+                "level": trial["level"],
+                "replicate": trial["replicate"],
+                "admitted_fraction": trial["outcome"]["admitted_fraction"],
+                "completed_fraction": (
+                    trial["outcome"]["admitted_fraction"]
+                    * trial["outcome"]["completed_over_admitted"]
+                ),
+                "telemetry_coverage": trial["telemetry_coverage"],
+            }
+            for trial in trials
+        ],
+        "response_statistics": response,
+        "qualifying_response_features": response_qualifying,
+        "efficacy": {
+            "labels": numeric(labels),
+            "positive": sum(labels),
+            "negative": len(labels) - sum(labels),
+            "label_balance_sufficient": (
+                sum(labels) >= MIN_EFFICACY_CLASS
+                and len(labels) - sum(labels) >= MIN_EFFICACY_CLASS
+            ),
+            "cross_validation_possible": cross_validation_possible,
+            "transitions": dict(sorted(transitions.items())),
+            "outcome_delta": outcome_cv,
+            "telemetry_delta": telemetry_cv,
+            "telemetry_qualifying_features": efficacy_qualifying,
+        },
+        "sanity": {
+            "trials": len(trials),
+            "invariants": invariants,
+            "red_flags": [name for name, passed in invariants.items() if not passed],
+        },
+    }
+
+
+def stable_adjacent_features(windows: list[dict[str, Any]]) -> dict[str, list[str]]:
+    result = {}
+    for left, right in zip(windows, windows[1:], strict=False):
+        key = f"{left['end_fraction']:.2f}->{right['end_fraction']:.2f}"
+        result[key] = sorted(
+            set(left["qualifying_response_features"])
+            & set(right["qualifying_response_features"])
+        )
+    return result
+
+
+def stable_adjacent_efficacy_features(
+    windows: list[dict[str, Any]],
+) -> dict[str, list[str]]:
+    eligible = [window for window in windows if window["end_fraction"] >= 0.25]
+    result = {}
+    for left, right in zip(eligible, eligible[1:], strict=False):
+        key = f"{left['end_fraction']:.2f}->{right['end_fraction']:.2f}"
+        result[key] = sorted(
+            set(left["efficacy"]["telemetry_qualifying_features"])
+            & set(right["efficacy"]["telemetry_qualifying_features"])
+        )
+    return result
+
+
+def consistent_load_regimes(
+    windows: list[dict[str, Any]], stable: dict[str, list[str]]
+) -> dict[str, Any]:
+    by_end = {float(window["end_fraction"]): window for window in windows}
+    result = {}
+    for transition, features in stable.items():
+        _left_text, right_text = transition.split("->")
+        window = by_end[float(right_text)]
+        for feature in features:
+            deltas_by_level: dict[str, list[float]] = defaultdict(list)
+            all_deltas = []
+            for action in window["actions"]:
+                value = float(action["delta_state"][feature])
+                deltas_by_level[str(action["group"]["level"])].append(value)
+                all_deltas.append(value)
+            positive = sum(value > 1e-12 for value in all_deltas)
+            negative = sum(value < -1e-12 for value in all_deltas)
+            direction = 1 if positive >= negative else -1
+            consistent = []
+            for level, values in sorted(deltas_by_level.items()):
+                matching = sum(direction * value > 1e-12 for value in values)
+                nonzero = sum(abs(value) > 1e-12 for value in values)
+                if nonzero and matching / nonzero >= 2.0 / 3.0:
+                    consistent.append(level)
+            result[f"{transition}:{feature}"] = {
+                "direction": direction,
+                "consistent_load_regimes": consistent,
+                "passes_two_regimes": len(consistent) >= 2,
+            }
+    return result
+
+
+def mechanism_gate(
+    stable: Mapping[str, list[str]], load_consistency: Mapping[str, Mapping[str, Any]]
+) -> dict[str, Any]:
+    by_transition = {}
+    for transition, features in stable.items():
+        qualifying = sorted(
+            feature
+            for feature in features
+            if load_consistency[f"{transition}:{feature}"]["passes_two_regimes"]
+        )
+        by_transition[transition] = qualifying
+    passing_transitions = sorted(
+        transition
+        for transition, features in by_transition.items()
+        if len(features) >= 2
+    )
+    return {
+        "minimum_features": 2,
+        "by_transition": by_transition,
+        "passing_transitions": passing_transitions,
+        "passes": bool(passing_transitions),
+    }
+
+
+def controller_gate(
+    run_root: Path, manifest: Mapping[str, Any]
+) -> dict[str, Any]:
+    path = run_root / "controller-state.json"
+    state = json.loads(path.read_text(encoding="utf-8"))
+    expected_sessions = {str(session["session"]) for session in manifest["sessions"]}
+    actual_sessions = set(state.get("sessions", {}))
+    session_invariants = [
+        passed
+        for session in state.get("sessions", {}).values()
+        for passed in session.get("validation", {}).get("invariants", {}).values()
+    ]
+    invariants = {
+        "controller_complete": state.get("status") == "complete",
+        "completed_session_count": int(state.get("completed_sessions", -1))
+        == len(expected_sessions),
+        "exact_session_set": actual_sessions == expected_sessions,
+        "all_sessions_complete": all(
+            session.get("status") == "complete"
+            for session in state.get("sessions", {}).values()
+        ),
+        "no_controller_failures": not state.get("failures"),
+        "under_h20_hour_cap": float(state.get("gpu_hours_total", math.inf))
+        <= float(manifest["budget"]["hard_cap_h20_hours"]),
+        "all_stream_validation_invariants_pass": bool(session_invariants)
+        and all(session_invariants),
+    }
+    return {
+        "path": str(path.resolve()),
+        "sha256": sha256_file(path),
+        "gpu_hours_total": float(state.get("gpu_hours_total", math.nan)),
+        "invariants": invariants,
+        "red_flags": [name for name, passed in invariants.items() if not passed],
+    }
+
+
+def cumulative_coverage_gate(windows: list[dict[str, Any]]) -> dict[str, Any]:
+    trajectories: dict[str, list[tuple[float, float]]] = defaultdict(list)
+    trial_sets = []
+    for window in windows:
+        trial_sets.append({str(trial["trial_id"]) for trial in window["trial_sanity"]})
+        for trial in window["trial_sanity"]:
+            trajectories[str(trial["trial_id"])].append(
+                (
+                    float(trial["admitted_fraction"]),
+                    float(trial["completed_fraction"]),
+                )
+            )
+    invariants = {
+        "same_trials_at_every_checkpoint": bool(trial_sets)
+        and all(trial_set == trial_sets[0] for trial_set in trial_sets[1:]),
+        "admitted_fraction_monotonic": all(
+            all(right[0] + 1e-12 >= left[0] for left, right in zip(values, values[1:]))
+            for values in trajectories.values()
+        ),
+        "completed_fraction_monotonic": all(
+            all(right[1] + 1e-12 >= left[1] for left, right in zip(values, values[1:]))
+            for values in trajectories.values()
+        ),
+    }
+    return {
+        "invariants": invariants,
+        "red_flags": [name for name, passed in invariants.items() if not passed],
+    }
+
+
+def audit(*, run_root: Path, manifest_path: Path, output_path: Path) -> dict[str, Any]:
+    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    if manifest.get("schema") != "intervention-response-phase-aware-pilot-manifest-v3":
+        raise ValueError("unexpected phase-aware pilot manifest schema")
+    fractions = [float(value) for value in manifest["checkpoints"]["fractions"]]
+    seconds = [float(value) for value in manifest["checkpoints"]["seconds"]]
+    cumulative_intervals = tuple((0.0, end_s) for end_s in seconds)
+    quarter_intervals = ((0.0, 75.0), (75.0, 150.0), (150.0, 225.0), (225.0, 300.0))
+    intervals = tuple(dict.fromkeys([*cumulative_intervals, *quarter_intervals]))
+    trials_by_interval, streams = load_interval_trials(
+        run_root=run_root, manifest=manifest, intervals_s=intervals
+    )
+    cumulative = [
+        analyze_window(
+            trials_by_interval[interval],
+            start_s=interval[0],
+            end_s=interval[1],
+            fraction=fraction,
+        )
+        for fraction, interval in zip(fractions, cumulative_intervals, strict=True)
+    ]
+    quarter_blocks = [
+        analyze_window(
+            trials_by_interval[interval],
+            start_s=interval[0],
+            end_s=interval[1],
+            fraction=interval[1] / 300.0,
+        )
+        for interval in quarter_intervals
+    ]
+    stable = stable_adjacent_features(cumulative)
+    load_consistency = consistent_load_regimes(cumulative, stable)
+    mechanism = mechanism_gate(stable, load_consistency)
+    mechanism_features = sorted(
+        {
+            feature
+            for transition in mechanism["passing_transitions"]
+            for feature in mechanism["by_transition"][transition]
+        }
+    )
+    full = cumulative[-1]
+    efficacy_stable = stable_adjacent_efficacy_features(cumulative)
+    efficacy_candidates = sorted(
+        {feature for features in efficacy_stable.values() for feature in features}
+    )
+    efficacy_features = sorted(set(efficacy_candidates) & set(mechanism_features))
+    controller = controller_gate(run_root, manifest)
+    coverage = cumulative_coverage_gate(cumulative)
+    red_flags = sorted(
+        {
+            flag
+            for window in [*cumulative, *quarter_blocks]
+            for flag in window["sanity"]["red_flags"]
+        }
+        | set(controller["red_flags"])
+        | set(coverage["red_flags"])
+    )
+    if red_flags:
+        decision = "STOP_DATA_INVALID"
+    elif not mechanism["passes"]:
+        decision = "STOP_NO_PHASE_STABLE_RESPONSE"
+    elif not full["efficacy"]["label_balance_sufficient"]:
+        decision = "MECHANISM_ONLY_NO_LABEL_BALANCE"
+    elif not efficacy_features:
+        decision = "STOP_NO_INCREMENTAL_TUNING_SIGNAL"
+    else:
+        decision = "OPEN_E2E_POLICY_TEST"
+    payload = {
+        "schema": SCHEMA,
+        "status": "COMPLETE",
+        "decision": decision,
+        "claim_boundary": "Development mechanism pilot; not a held-out paper claim.",
+        "mechanism_features": mechanism_features,
+        "mechanism_gate": mechanism,
+        "stable_adjacent_features": stable,
+        "load_consistency": load_consistency,
+        "stable_incremental_efficacy_features": efficacy_features,
+        "stable_incremental_efficacy_candidates": efficacy_candidates,
+        "stable_adjacent_efficacy_features": efficacy_stable,
+        "cumulative": cumulative,
+        "quarter_blocks": quarter_blocks,
+        "provenance": {
+            "analysis_script": str(Path(__file__).resolve()),
+            "analysis_script_sha256": sha256_file(Path(__file__).resolve()),
+            "manifest": str(manifest_path.resolve()),
+            "manifest_sha256": sha256_file(manifest_path),
+            "run_root": str(run_root.resolve()),
+            "streams": streams,
+        },
+        "sanity": {
+            "streams": len(streams),
+            "stream_bytes": numeric(item["bytes"] for item in streams),
+            "controller": controller,
+            "cumulative_coverage": coverage,
+            "red_flags": red_flags,
+        },
+    }
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
+    return payload
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("--manifest", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    payload = audit(
+        run_root=args.run_root,
+        manifest_path=args.manifest,
+        output_path=args.output,
+    )
+    print(
+        json.dumps(
+            {
+                "decision": payload["decision"],
+                "mechanism_features": payload["mechanism_features"],
+                "stable_incremental_efficacy_features": payload[
+                    "stable_incremental_efficacy_features"
+                ],
+                "sanity": payload["sanity"],
+            },
+            indent=2,
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

runs/intervention-response-v2/failed-three-load-controller-state-v2.json

@@ -0,0 +1,49 @@
+{
+  "completed_sessions": 0,
+  "failures": [
+    {
+      "failure": "TimeoutError('client timeout: tp4_mns16 high')",
+      "session": "rep1-mns16"
+    }
+  ],
+  "gpu_hours_total": 1.3580546813540988,
+  "hard_cap_h20_hours": 8.0,
+  "schema": "intervention-response-phase-aware-pilot-state-v2",
+  "sessions": {
+    "rep1-mns16": {
+      "burnin": {
+        "elapsed_s": 64.469448031,
+        "feasible": false,
+        "pass_rate": 0.5137254901960784
+      },
+      "failure": "TimeoutError('client timeout: tp4_mns16 high')",
+      "gpu_hours": 1.3580546813540988,
+      "mns": 16,
+      "replicate": 1,
+      "runs": [
+        {
+          "early_stopped": false,
+          "elapsed_s": 301.114789132,
+          "feasible": true,
+          "level": "low",
+          "offered_req_s_per_gpu": 1.5,
+          "pass_rate": 1.0,
+          "selected_count": 1800
+        },
+        {
+          "early_stopped": false,
+          "elapsed_s": 313.040655133,
+          "feasible": false,
+          "level": "mid",
+          "offered_req_s_per_gpu": 2.125,
+          "pass_rate": 0.5631372549019608,
+          "selected_count": 2550
+        }
+      ],
+      "started_at": 1784021277.1191542,
+      "status": "failed"
+    }
+  },
+  "started_at": 1784021276.8224819,
+  "status": "failed"
+}

runs/intervention-response-v2/pilot-controller-state-v3.json

@@ -0,0 +1,342 @@
+{
+  "completed_at": 1784027453.4005148,
+  "completed_sessions": 6,
+  "failures": [],
+  "gpu_hours_total": 5.092428478929731,
+  "hard_cap_h20_hours": 6.641945318645901,
+  "schema": "intervention-response-phase-aware-pilot-state-v3",
+  "sessions": {
+    "rep1-mns16": {
+      "burnin": {
+        "elapsed_s": 64.194648239,
+        "feasible": false,
+        "pass_rate": 0.5235294117647059
+      },
+      "completed_at": 1784023620.6911042,
+      "gpu_hours": 0.856446467505561,
+      "mns": 16,
+      "replicate": 1,
+      "runs": [
+        {
+          "early_stopped": false,
+          "elapsed_s": 301.068878512,
+          "feasible": true,
+          "level": "low",
+          "offered_req_s_per_gpu": 1.5,
+          "pass_rate": 1.0,
+          "selected_count": 1800
+        },
+        {
+          "early_stopped": false,
+          "elapsed_s": 313.268231477,
+          "feasible": false,
+          "level": "mid",
+          "offered_req_s_per_gpu": 2.125,
+          "pass_rate": 0.5603921568627451,
+          "selected_count": 2550
+        }
+      ],
+      "started_at": 1784022846.257515,
+      "status": "complete",
+      "validation": {
+        "accounting_mode": "graceful-footer",
+        "cell": "tp4_mns16",
+        "invariants": {
+          "anchor_intervals_present": true,
+          "compile_capture_pre_ready": true,
+          "encoded_balanced": true,
+          "footer_sidecar_agrees": true,
+          "last_step_matches": true,
+          "layer1_contiguous": true,
+          "layer1_zero_drops": true,
+          "one_footer_last": true,
+          "one_ready_marker": true,
+          "sidecar_final": true,
+          "warmup_exact_16": true,
+          "warmup_long": true,
+          "written_matches_records": true
+        },
+        "layer1_records": 60017,
+        "post_ready_capture_events": [],
+        "stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep1-mns16/opprof/opprof-v1-dp0-pid458427-1784022917661752826.jsonl"
+      }
+    },
+    "rep1-mns64": {
+      "burnin": {
+        "elapsed_s": 60.79926316,
+        "feasible": true,
+        "pass_rate": 1.0
+      },
+      "completed_at": 1784024382.2164423,
+      "gpu_hours": 0.8415956518385146,
+      "mns": 64,
+      "replicate": 1,
+      "runs": [
+        {
+          "early_stopped": false,
+          "elapsed_s": 301.022700013,
+          "feasible": true,
+          "level": "low",
+          "offered_req_s_per_gpu": 1.5,
+          "pass_rate": 1.0,
+          "selected_count": 1800
+        },
+        {
+          "early_stopped": false,
+          "elapsed_s": 301.100993458,
+          "feasible": true,
+          "level": "mid",
+          "offered_req_s_per_gpu": 2.125,
+          "pass_rate": 1.0,
+          "selected_count": 2550
+        }
+      ],
+      "started_at": 1784023620.9865165,
+      "status": "complete",
+      "validation": {
+        "accounting_mode": "graceful-footer",
+        "cell": "tp4_mns64",
+        "invariants": {
+          "anchor_intervals_present": true,
+          "compile_capture_pre_ready": true,
+          "encoded_balanced": true,
+          "footer_sidecar_agrees": true,
+          "last_step_matches": true,
+          "layer1_contiguous": true,
+          "layer1_zero_drops": true,
+          "one_footer_last": true,
+          "one_ready_marker": true,
+          "sidecar_final": true,
+          "warmup_exact_16": true,
+          "warmup_long": true,
+          "written_matches_records": true
+        },
+        "layer1_records": 62456,
+        "post_ready_capture_events": [],
+        "stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep1-mns64/opprof/opprof-v1-dp0-pid471212-1784023695315485142.jsonl"
+      }
+    },
+    "rep2-mns16": {
+      "burnin": {
+        "elapsed_s": 63.518994307,
+        "feasible": false,
+        "pass_rate": 0.6647058823529411
+      },
+      "completed_at": 1784025910.710227,
+      "gpu_hours": 0.8481734045346578,
+      "mns": 16,
+      "replicate": 2,
+      "runs": [
+        {
+          "early_stopped": false,
+          "elapsed_s": 315.780805484,
+          "feasible": false,
+          "level": "mid",
+          "offered_req_s_per_gpu": 2.125,
+          "pass_rate": 0.31450980392156863,
+          "selected_count": 2550
+        },
+        {
+          "early_stopped": false,
+          "elapsed_s": 300.346368314,
+          "feasible": true,
+          "level": "low",
+          "offered_req_s_per_gpu": 1.5,
+          "pass_rate": 1.0,
+          "selected_count": 1800
+        }
+      ],
+      "started_at": 1784025143.731658,
+      "status": "complete",
+      "validation": {
+        "accounting_mode": "graceful-footer",
+        "cell": "tp4_mns16",
+        "invariants": {
+          "anchor_intervals_present": true,
+          "compile_capture_pre_ready": true,
+          "encoded_balanced": true,
+          "footer_sidecar_agrees": true,
+          "last_step_matches": true,
+          "layer1_contiguous": true,
+          "layer1_zero_drops": true,
+          "one_footer_last": true,
+          "one_ready_marker": true,
+          "sidecar_final": true,
+          "warmup_exact_16": true,
+          "warmup_long": true,
+          "written_matches_records": true
+        },
+        "layer1_records": 58465,
+        "post_ready_capture_events": [],
+        "stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep2-mns16/opprof/opprof-v1-dp0-pid496100-1784025206790752505.jsonl"
+      }
+    },
+    "rep2-mns64": {
+      "burnin": {
+        "elapsed_s": 60.803771161,
+        "feasible": true,
+        "pass_rate": 1.0
+      },
+      "completed_at": 1784025143.4362338,
+      "gpu_hours": 0.8412917502721151,
+      "mns": 64,
+      "replicate": 2,
+      "runs": [
+        {
+          "early_stopped": false,
+          "elapsed_s": 300.605515019,
+          "feasible": true,
+          "level": "mid",
+          "offered_req_s_per_gpu": 2.125,
+          "pass_rate": 1.0,
+          "selected_count": 2550
+        },
+        {
+          "early_stopped": false,
+          "elapsed_s": 300.354084826,
+          "feasible": true,
+          "level": "low",
+          "offered_req_s_per_gpu": 1.5,
+          "pass_rate": 1.0,
+          "selected_count": 1800
+        }
+      ],
+      "started_at": 1784024382.5080929,
+      "status": "complete",
+      "validation": {
+        "accounting_mode": "graceful-footer",
+        "cell": "tp4_mns64",
+        "invariants": {
+          "anchor_intervals_present": true,
+          "compile_capture_pre_ready": true,
+          "encoded_balanced": true,
+          "footer_sidecar_agrees": true,
+          "last_step_matches": true,
+          "layer1_contiguous": true,
+          "layer1_zero_drops": true,
+          "one_footer_last": true,
+          "one_ready_marker": true,
+          "sidecar_final": true,
+          "warmup_exact_16": true,
+          "warmup_long": true,
+          "written_matches_records": true
+        },
+        "layer1_records": 61696,
+        "post_ready_capture_events": [],
+        "stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep2-mns64/opprof/opprof-v1-dp0-pid483758-1784024456752184540.jsonl"
+      }
+    },
+    "rep3-mns16": {
+      "burnin": {
+        "elapsed_s": 64.324514503,
+        "feasible": false,
+        "pass_rate": 0.5588235294117647
+      },
+      "completed_at": 1784026691.6293187,
+      "gpu_hours": 0.8631463612450494,
+      "mns": 16,
+      "replicate": 3,
+      "runs": [
+        {
+          "early_stopped": false,
+          "elapsed_s": 300.711109876,
+          "feasible": true,
+          "level": "low",
+          "offered_req_s_per_gpu": 1.5,
+          "pass_rate": 1.0,
+          "selected_count": 1800
+        },
+        {
+          "early_stopped": false,
+          "elapsed_s": 317.011840655,
+          "feasible": false,
+          "level": "mid",
+          "offered_req_s_per_gpu": 2.125,
+          "pass_rate": 0.2635294117647059,
+          "selected_count": 2550
+        }
+      ],
+      "started_at": 1784025911.00421,
+      "status": "complete",
+      "validation": {
+        "accounting_mode": "graceful-footer",
+        "cell": "tp4_mns16",
+        "invariants": {
+          "anchor_intervals_present": true,
+          "compile_capture_pre_ready": true,
+          "encoded_balanced": true,
+          "footer_sidecar_agrees": true,
+          "last_step_matches": true,
+          "layer1_contiguous": true,
+          "layer1_zero_drops": true,
+          "one_footer_last": true,
+          "one_ready_marker": true,
+          "sidecar_final": true,
+          "warmup_exact_16": true,
+          "warmup_long": true,
+          "written_matches_records": true
+        },
+        "layer1_records": 61438,
+        "post_ready_capture_events": [],
+        "stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep3-mns16/opprof/opprof-v1-dp0-pid508913-1784025983046041167.jsonl"
+      }
+    },
+    "rep3-mns64": {
+      "burnin": {
+        "elapsed_s": 60.850091109,
+        "feasible": true,
+        "pass_rate": 1.0
+      },
+      "completed_at": 1784027453.3790896,
+      "gpu_hours": 0.8417748435338338,
+      "mns": 64,
+      "replicate": 3,
+      "runs": [
+        {
+          "early_stopped": false,
+          "elapsed_s": 300.710505069,
+          "feasible": true,
+          "level": "low",
+          "offered_req_s_per_gpu": 1.5,
+          "pass_rate": 1.0,
+          "selected_count": 1800
+        },
+        {
+          "early_stopped": false,
+          "elapsed_s": 300.869326418,
+          "feasible": true,
+          "level": "mid",
+          "offered_req_s_per_gpu": 2.125,
+          "pass_rate": 1.0,
+          "selected_count": 2550
+        }
+      ],
+      "started_at": 1784026691.96421,
+      "status": "complete",
+      "validation": {
+        "accounting_mode": "graceful-footer",
+        "cell": "tp4_mns64",
+        "invariants": {
+          "anchor_intervals_present": true,
+          "compile_capture_pre_ready": true,
+          "encoded_balanced": true,
+          "footer_sidecar_agrees": true,
+          "last_step_matches": true,
+          "layer1_contiguous": true,
+          "layer1_zero_drops": true,
+          "one_footer_last": true,
+          "one_ready_marker": true,
+          "sidecar_final": true,
+          "warmup_exact_16": true,
+          "warmup_long": true,
+          "written_matches_records": true
+        },
+        "layer1_records": 64776,
+        "post_ready_capture_events": [],
+        "stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep3-mns64/opprof/opprof-v1-dp0-pid521797-1784026767359409666.jsonl"
+      }
+    }
+  },
+  "started_at": 1784022845.964628,
+  "status": "complete"
+}

runs/intervention-response-v2/pilot-manifest-v3.json

@@ -0,0 +1,324 @@
+{
+  "budget": {
+    "expected_h20_hours": [
+      4.8,
+      6.641945318645901
+    ],
+    "expected_wall_minutes": [
+      75,
+      95
+    ],
+    "global_hard_cap_h20_hours": 8.0,
+    "hard_cap_h20_hours": 6.641945318645901,
+    "prior_attempt_h20_hours": 1.3580546813540988,
+    "session_estimate_h20_hours": 1.0
+  },
+  "burnin": {
+    "anchor": 0.18919793755240089,
+    "arrival_order_sha256": "6c0ac4cb9a30ef501eeeacc8e6cc631c345e976db5ccf530ea5a1ec706d62a24",
+    "input_length_order_sha256": "7939cc20e1a00d1031d27d71508789f38decbbbb6ea59a1df18b2ec342fd2ef8",
+    "offered_req_s": 8.5,
+    "offered_req_s_per_gpu": 2.125,
+    "request_id_order_sha256": "84f4809acbc8acd3b1d14dfa357134a1dc0b9287341624b33f598dafeef54dc7",
+    "selected_count": 510,
+    "study": "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/studies/burnin-tp4.json",
+    "study_sha256": "5d6c2098042909a863efd3112818fbee9bafe96f22898ac98b66846dbe1fef0f"
+  },
+  "checkpoints": {
+    "fractions": [
+      0.1,
+      0.25,
+      0.5,
+      0.75,
+      1.0
+    ],
+    "seconds": [
+      30.0,
+      75.0,
+      150.0,
+      225.0,
+      300.0
+    ]
+  },
+  "engine": {
+    "disable_slo_early_stop": true,
+    "duration_s": 300.0,
+    "mns_endpoints": [
+      16,
+      64
+    ],
+    "replay_time_scale": 0.5,
+    "safety_deadline_s": 360.0,
+    "tp": 4
+  },
+  "private": {
+    "merged_traces": {
+      "1": {
+        "bytes": 337429767,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep1.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9420,
+        "sha256": "68983266aa0e66aa589562f7c08edbd966f9ba4405e20c105adb43777d2dfbf5",
+        "source_sha256": [
+          "b242d1d9086df3accab57b4c92445d5edd581e12f47e12cea227aa63964c6930",
+          "d23b549f7b69af3647308677bbf76f818a3c226a1c98f9a9f93f09ceee46be87"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low1.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high1.jsonl"
+        ]
+      },
+      "2": {
+        "bytes": 337509330,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep2.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9457,
+        "sha256": "f38e8938f6a481fc6725b71b21aa04ff7eaf79783cdfd6e41aa2f074156f00c2",
+        "source_sha256": [
+          "4cbb0baac082bd54af562ce2f39104c5c23b4671672da365a67b1e8c146adf9f",
+          "bb0bcd2564a88000f435f12feb21c7c902eafc9ea5fe916adfe9d1eae47f3f9a"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low2.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high2.jsonl"
+        ]
+      },
+      "3": {
+        "bytes": 337450256,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep3.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9431,
+        "sha256": "3094084b0bb20cc02eecf465091a5c919b4e5b112f704cdc36a563d1efdcee46",
+        "source_sha256": [
+          "1f7ececb142f9a363d2d1ca25eb7b8488b2cc319a51b55faa384f2a3d51f2142",
+          "6f326234791e1cff4ff866bface0d097d0d6e3844eebb1c97653d8e9c35e9397"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low3.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high3.jsonl"
+        ]
+      }
+    }
+  },
+  "repetitions": {
+    "1": {
+      "duration_s": 300.0,
+      "load_order": [
+        "low",
+        "mid"
+      ],
+      "merged_trace": {
+        "bytes": 337429767,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep1.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9420,
+        "sha256": "68983266aa0e66aa589562f7c08edbd966f9ba4405e20c105adb43777d2dfbf5",
+        "source_sha256": [
+          "b242d1d9086df3accab57b4c92445d5edd581e12f47e12cea227aa63964c6930",
+          "d23b549f7b69af3647308677bbf76f818a3c226a1c98f9a9f93f09ceee46be87"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low1.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high1.jsonl"
+        ]
+      },
+      "selections": {
+        "low": {
+          "anchor": 0.34815061842962747,
+          "arrival_order_sha256": "0b9473052a28b2eb98bb4bda59c726d6c74cd42656c00046f0559cefd38e0c42",
+          "input_length_order_sha256": "0172aa1fd7d994e9d95857c32293e25f6c537616b40e01034308a0d57cd94139",
+          "offered_req_s": 6.0,
+          "offered_req_s_per_gpu": 1.5,
+          "request_id_order_sha256": "af50f0c3edfc6f9fb74734b2501378fc8ba6fdd3dd3fe84f2bb1d68540eee69b",
+          "selected_count": 1800,
+          "target_count": 1800,
+          "target_req_s_per_gpu": 1.5
+        },
+        "mid": {
+          "anchor": 0.48686986110831465,
+          "arrival_order_sha256": "c2ad99986ce558da5901a9c5ec0a00bd69f198c981d8779235f2773a5c87f1c0",
+          "input_length_order_sha256": "9442bfebdc3fab5062dc1f4d688dc28c02afe3fd806c56dd8159f0ac7e6d0b94",
+          "offered_req_s": 8.5,
+          "offered_req_s_per_gpu": 2.125,
+          "request_id_order_sha256": "0bb61dbc9c26875e991d0d4f984134910d37463e5063f86ee960cf4f8aafb771",
+          "selected_count": 2550,
+          "target_count": 2550,
+          "target_req_s_per_gpu": 2.125
+        }
+      },
+      "source_roles": [
+        "low1",
+        "high1"
+      ],
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep1-tp4.json",
+      "study_sha256": "ecfff96e33d458eb1e3b9a6d24386f00cc6f1b19ff926e2ec6320b3f671a7ae3"
+    },
+    "2": {
+      "duration_s": 300.0,
+      "load_order": [
+        "mid",
+        "low"
+      ],
+      "merged_trace": {
+        "bytes": 337509330,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep2.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9457,
+        "sha256": "f38e8938f6a481fc6725b71b21aa04ff7eaf79783cdfd6e41aa2f074156f00c2",
+        "source_sha256": [
+          "4cbb0baac082bd54af562ce2f39104c5c23b4671672da365a67b1e8c146adf9f",
+          "bb0bcd2564a88000f435f12feb21c7c902eafc9ea5fe916adfe9d1eae47f3f9a"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low2.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high2.jsonl"
+        ]
+      },
+      "selections": {
+        "low": {
+          "anchor": 0.3411190868318572,
+          "arrival_order_sha256": "f4eafbdf7420e8a97e38d46fff1aec5fdbc2a049133862a793b4635b51061b7f",
+          "input_length_order_sha256": "cdf82cfc546651f1859c53348eace72348572f09e6e1879ad77d51b489a998fe",
+          "offered_req_s": 6.0,
+          "offered_req_s_per_gpu": 1.5,
+          "request_id_order_sha256": "adc728f189a1ce0301609cdbae7d1dc2318609b13eafb94bcb90903bb7998eb1",
+          "selected_count": 1800,
+          "target_count": 1800,
+          "target_req_s_per_gpu": 1.5
+        },
+        "mid": {
+          "anchor": 0.4825698948735577,
+          "arrival_order_sha256": "b9fc12cf3f86bc8a79bee65296e65aa2b8bf2aeca46b2887094c669adcbb9a00",
+          "input_length_order_sha256": "d8d4bd6fc8ba852a45605b673b6b3e4f33b58f459e69f2a032d226ee175b074e",
+          "offered_req_s": 8.5,
+          "offered_req_s_per_gpu": 2.125,
+          "request_id_order_sha256": "56a0616b6b54abafd37875c7cb25f8639afef2706ccc55dfbe568f45859ea382",
+          "selected_count": 2550,
+          "target_count": 2550,
+          "target_req_s_per_gpu": 2.125
+        }
+      },
+      "source_roles": [
+        "low2",
+        "high2"
+      ],
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep2-tp4.json",
+      "study_sha256": "d92a576db031db24bb58f354ea725d7f7567cb76699d387117ac5a6c9317bbb9"
+    },
+    "3": {
+      "duration_s": 300.0,
+      "load_order": [
+        "low",
+        "mid"
+      ],
+      "merged_trace": {
+        "bytes": 337450256,
+        "path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep3.jsonl",
+        "request_id_scheme": "sha256(source_sha256:line_number:original_id)",
+        "rows": 9431,
+        "sha256": "3094084b0bb20cc02eecf465091a5c919b4e5b112f704cdc36a563d1efdcee46",
+        "source_sha256": [
+          "1f7ececb142f9a363d2d1ca25eb7b8488b2cc319a51b55faa384f2a3d51f2142",
+          "6f326234791e1cff4ff866bface0d097d0d6e3844eebb1c97653d8e9c35e9397"
+        ],
+        "sources": [
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low3.jsonl",
+          "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high3.jsonl"
+        ]
+      },
+      "selections": {
+        "low": {
+          "anchor": 0.3459572828521402,
+          "arrival_order_sha256": "68de977e2ea7396ffe9406675a3d2c5ebecd1b9ea5afc8116c836cc53cfca188",
+          "input_length_order_sha256": "dffa79545e738b899bec2e9d282764a3388fe38b162982fb2d26928e17ef6016",
+          "offered_req_s": 6.0,
+          "offered_req_s_per_gpu": 1.5,
+          "request_id_order_sha256": "4993c26442fc6b5a4c09b6f576fd8b3cb1be7ce83b7a771e4a57ee7600de688d",
+          "selected_count": 1800,
+          "target_count": 1800,
+          "target_req_s_per_gpu": 1.5
+        },
+        "mid": {
+          "anchor": 0.48664343020532463,
+          "arrival_order_sha256": "efce7339e22d3618cb4d55e6b55bfddb2c563c18faba2a992d5829c13e3f55e9",
+          "input_length_order_sha256": "0792b05fff6729fbd92ab2bb4cb6d31bea7799e232ad42772936bc06efbafb54",
+          "offered_req_s": 8.5,
+          "offered_req_s_per_gpu": 2.125,
+          "request_id_order_sha256": "2a2fabe2c4cf176aeb7e0d32fb8e7dbb1f27429a2e7a0cd18d7d186f23096f19",
+          "selected_count": 2550,
+          "target_count": 2550,
+          "target_req_s_per_gpu": 2.125
+        }
+      },
+      "source_roles": [
+        "low3",
+        "high3"
+      ],
+      "study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep3-tp4.json",
+      "study_sha256": "fb8ffe256dace32f4ca8a8d49b662d98c3b69b94ecc8fa826e43068b238884ab"
+    }
+  },
+  "sanity": {
+    "distinct_selected_sets": 6,
+    "invariants": {
+      "all_counts_positive": true,
+      "load_levels_two": true,
+      "selection_hashes_unique": true,
+      "selection_sets_disjoint_across_repetitions": true,
+      "six_sessions": true,
+      "three_repetitions": true
+    },
+    "red_flags": [],
+    "selected_sets": 6
+  },
+  "schema": "intervention-response-phase-aware-pilot-manifest-v3",
+  "sessions": [
+    {
+      "mns": 16,
+      "replicate": 1,
+      "session": "rep1-mns16"
+    },
+    {
+      "mns": 64,
+      "replicate": 1,
+      "session": "rep1-mns64"
+    },
+    {
+      "mns": 64,
+      "replicate": 2,
+      "session": "rep2-mns64"
+    },
+    {
+      "mns": 16,
+      "replicate": 2,
+      "session": "rep2-mns16"
+    },
+    {
+      "mns": 16,
+      "replicate": 3,
+      "session": "rep3-mns16"
+    },
+    {
+      "mns": 64,
+      "replicate": 3,
+      "session": "rep3-mns64"
+    }
+  ],
+  "source": {
+    "base_manifest": "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/runs/p1b/pilot-manifest.json",
+    "base_manifest_sha256": "86b249a087ab1bcd51cd8de3d1da6bf34afd41af977eb58c93229b941dbf5c8a",
+    "prior_attempt_failure": [
+      {
+        "failure": "TimeoutError('client timeout: tp4_mns16 high')",
+        "session": "rep1-mns16"
+      }
+    ],
+    "prior_attempt_h20_hours": 1.3580546813540988,
+    "prior_attempt_state": "/home/admin/cpfs/wjh/intervention-response-v2-20260714/runs/pilot/controller-state.json",
+    "prior_attempt_state_sha256": "c88030ae36862abfc643a855eb3630fb18cd74a58b388514aa31ec1ab485a431",
+    "source_trace": "/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260312_1000.jsonl",
+    "source_trace_sha256": "875ba869775deb78086477919f03b322da14e2673c7d070e26528c4190912757",
+    "window_id": "chat_w20260312_1000"
+  },
+  "status": "PASS"
+}

runs/intervention-response-v2/pilot_controller.py

@@ -0,0 +1,578 @@
+#!/usr/bin/env python3
+"""Serialized, resumable controller for the 300-second phase-aware pilot."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shlex
+import signal
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any, Mapping
+
+
+HERE = Path(__file__).resolve().parent
+PHASE6 = HERE.parent / "opprof-phase6"
+sys.path.insert(0, str(PHASE6))
+
+import opprof_phase6_controller as base  # noqa: E402
+
+
+SCHEMA = "intervention-response-phase-aware-pilot-state-v3"
+SESSION_ESTIMATE_H20_HOURS = 1.0
+SAFETY_H20_HOURS = 0.20
+CLIENT_TIMEOUT_S = 450.0
+
+
+def atomic_json(path: Path, payload: Any) -> None:
+    base.atomic_json(path, payload)
+
+
+def wait_all_idle(timeout_s: float = 30.0) -> None:
+    deadline = time.monotonic() + timeout_s
+    last_error: Exception | None = None
+    while time.monotonic() < deadline:
+        try:
+            base.assert_all_idle()
+            return
+        except RuntimeError as error:
+            last_error = error
+            time.sleep(1.0)
+    raise last_error or RuntimeError("GPU idle timeout")
+
+
+def configure(args: argparse.Namespace, manifest: dict[str, Any]) -> None:
+    base.WORKDIR = args.run_root.parent
+    base.RUN_ROOT = args.run_root
+    base.STATE = args.run_root / "controller-state.json"
+    base.SOURCE = args.vllm_source
+    base.VENV = args.venv
+    base.AITUNER = args.aituner_root
+    base.MODEL = args.model
+    base.CLIENT = args.client
+    base.GPU_LIMIT = float(manifest["budget"]["hard_cap_h20_hours"])
+    base.MARKER = "intervention-response-phase-aware-v3"
+    base.CELLS = {
+        f"tp4_mns{mns}": {"tp": 4, "mns": int(mns)}
+        for mns in manifest["engine"]["mns_endpoints"]
+    }
+
+
+def validate_inputs(args: argparse.Namespace, manifest: dict[str, Any]) -> None:
+    if manifest.get("schema") != "intervention-response-phase-aware-pilot-manifest-v3":
+        raise RuntimeError("unexpected phase-aware pilot manifest schema")
+    if manifest.get("status") != "PASS":
+        raise RuntimeError("phase-aware pilot manifest did not pass preflight")
+    if abs(
+        float(manifest["budget"]["session_estimate_h20_hours"])
+        - SESSION_ESTIMATE_H20_HOURS
+    ) > 1e-12:
+        raise RuntimeError("controller and manifest session-cost estimates disagree")
+    failed_invariants = [
+        name
+        for name, passed in manifest.get("sanity", {}).get("invariants", {}).items()
+        if not passed
+    ]
+    if failed_invariants:
+        raise RuntimeError(f"phase-aware pilot invariants failed: {failed_invariants}")
+
+    required = {
+        "manifest": args.manifest,
+        "aituner_root": args.aituner_root,
+        "vllm_source": args.vllm_source,
+        "venv_python": args.venv / "bin/python",
+        "venv_vllm": args.venv / "bin/vllm",
+        "model": args.model,
+        "client": args.client,
+        "burnin_study": Path(manifest["burnin"]["study"]),
+    }
+    for replicate, repetition in manifest["repetitions"].items():
+        required[f"rep{replicate}_study"] = Path(repetition["study"])
+        required[f"rep{replicate}_trace"] = Path(
+            repetition["merged_trace"]["path"]
+        )
+    missing = {name: str(path) for name, path in required.items() if not path.exists()}
+    if missing:
+        raise RuntimeError(f"phase-aware pilot input paths missing: {missing}")
+
+
+def warmup_selection(repetition: Mapping[str, Any]) -> Mapping[str, Any]:
+    return max(
+        repetition["selections"].values(),
+        key=lambda selection: float(selection["offered_req_s_per_gpu"]),
+    )
+
+
+def dry_run_plan(args: argparse.Namespace, manifest: dict[str, Any]) -> dict[str, Any]:
+    sessions = []
+    for index, session in enumerate(manifest["sessions"]):
+        cell = f"tp4_mns{int(session['mns'])}"
+        entry = {"cell": cell, "gpus": (0, 1, 2, 3), "port": 8950 + index}
+        repetition = manifest["repetitions"][str(session["replicate"])]
+        session_root = args.run_root / "sessions" / str(session["session"])
+        warmup = warmup_selection(repetition)
+        commands = {
+            "server": base.server_command(cell, entry["gpus"], entry["port"]),
+            "warmup": client_command(
+                entry,
+                study=repetition["study"],
+                anchor=float(warmup["anchor"]),
+                output=session_root / "warmup",
+                warmup=True,
+            ),
+            "burnin": client_command(
+                entry,
+                study=manifest["burnin"]["study"],
+                anchor=float(manifest["burnin"]["anchor"]),
+                output=session_root / "burnin",
+                warmup=False,
+            ),
+        }
+        for level in repetition["load_order"]:
+            selection = repetition["selections"][level]
+            commands[level] = client_command(
+                entry,
+                study=repetition["study"],
+                anchor=float(selection["anchor"]),
+                output=session_root / level,
+                warmup=False,
+            )
+        sessions.append(
+            {
+                "session": session["session"],
+                "replicate": int(session["replicate"]),
+                "mns": int(session["mns"]),
+                "port": entry["port"],
+                "load_order": repetition["load_order"],
+                "remaining_projection_h20_hours": remaining_projection(
+                    len(manifest["sessions"]), index
+                ),
+                "commands": {
+                    role: shlex.join(command) for role, command in commands.items()
+                },
+            }
+        )
+    return {
+        "schema": "intervention-response-phase-aware-pilot-dry-run-v3",
+        "status": "PASS",
+        "manifest": str(args.manifest),
+        "run_root": str(args.run_root),
+        "session_count": len(sessions),
+        "projected_h20_hours": remaining_projection(len(sessions), 0),
+        "hard_cap_h20_hours": float(manifest["budget"]["hard_cap_h20_hours"]),
+        "sessions": sessions,
+    }
+
+
+def load_state(path: Path, hard_cap: float) -> dict[str, Any]:
+    if path.exists():
+        return json.loads(path.read_text(encoding="utf-8"))
+    return {
+        "schema": SCHEMA,
+        "status": "initialized",
+        "hard_cap_h20_hours": hard_cap,
+        "gpu_hours_total": 0.0,
+        "completed_sessions": 0,
+        "sessions": {},
+        "failures": [],
+        "started_at": time.time(),
+    }
+
+
+def save_state(path: Path, state: dict[str, Any]) -> None:
+    atomic_json(path, state)
+
+
+def append_echo(run_root: Path, line: str) -> None:
+    run_root.mkdir(parents=True, exist_ok=True)
+    with (run_root / "launch-echo.log").open("a", encoding="utf-8") as target:
+        target.write(line + "\n")
+    print(line, flush=True)
+
+
+def remaining_projection(session_count: int, index: int) -> float:
+    return (session_count - index) * SESSION_ESTIMATE_H20_HOURS + SAFETY_H20_HOURS
+
+
+def start_server(
+    *, session: dict[str, Any], index: int, run_root: Path
+) -> dict[str, Any]:
+    cell = f"tp4_mns{int(session['mns'])}"
+    gpus = (0, 1, 2, 3)
+    session_root = run_root / "sessions" / str(session["session"])
+    session_root.mkdir(parents=True, exist_ok=True)
+    port = 8950 + index
+    command = base.server_command(cell, gpus, port)
+    with (session_root / "commands.log").open("a", encoding="utf-8") as log:
+        log.write(f"SERVER {shlex.join(command)}\n")
+    server_log = (session_root / "server.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update(
+        {
+            "CUDA_VISIBLE_DEVICES": "0,1,2,3",
+            "VLLM_OPPROF_DIR": str(session_root / "opprof"),
+            "OPPROF_PHASE6_MARKER": base.MARKER,
+            "AITUNER_ROOT": str(base.AITUNER),
+            "HF_HUB_OFFLINE": "1",
+            "TRANSFORMERS_OFFLINE": "1",
+            "PYTHONUNBUFFERED": "1",
+        }
+    )
+    server = subprocess.Popen(
+        command,
+        cwd=base.SOURCE,
+        env=environment,
+        stdout=server_log,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    base.OWNED_PGIDS.add(server.pid)
+    return {
+        "cell": cell,
+        "gpus": gpus,
+        "port": port,
+        "dir": session_root,
+        "server": server,
+        "server_handle": server_log,
+        "spawned_at": time.time(),
+        "results": [],
+    }
+
+
+def client_command(
+    entry: dict[str, Any],
+    *,
+    study: str,
+    anchor: float,
+    output: Path,
+    warmup: bool,
+) -> list[str]:
+    config = base.CELLS[entry["cell"]]
+    command = [
+        "taskset",
+        "-c",
+        base.cpu_mask(entry["gpus"]),
+        str(base.VENV / "bin/python"),
+        str(base.CLIENT),
+        "warmup" if warmup else "run-anchor",
+        "--study",
+        study,
+        "--cell",
+        entry["cell"],
+        "--anchor",
+        str(anchor),
+        "--tp",
+        str(config["tp"]),
+        "--mns",
+        str(config["mns"]),
+        "--base-url",
+        f"http://127.0.0.1:{entry['port']}",
+        "--result-dir",
+        str(output),
+        "--disable-slo-early-stop",
+    ]
+    return command
+
+
+def run_client(
+    *,
+    entry: dict[str, Any],
+    role: str,
+    study: str,
+    selection: dict[str, Any],
+    output: Path,
+    state: dict[str, Any],
+    warmup: bool = False,
+) -> dict[str, Any]:
+    command = client_command(
+        entry,
+        study=study,
+        anchor=float(selection["anchor"]),
+        output=output,
+        warmup=warmup,
+    )
+    with (entry["dir"] / "commands.log").open("a", encoding="utf-8") as log:
+        log.write(f"CLIENT role={role} {shlex.join(command)}\n")
+    handle = (output.parent / f"{output.name}.log").open("ab", buffering=0)
+    environment = os.environ.copy()
+    environment.update({"AITUNER_ROOT": str(base.AITUNER), "PYTHONUNBUFFERED": "1"})
+    process = subprocess.Popen(
+        command,
+        cwd=base.WORKDIR,
+        env=environment,
+        stdout=handle,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+    )
+    deadline = time.monotonic() + (180.0 if warmup else CLIENT_TIMEOUT_S)
+    try:
+        while process.poll() is None:
+            if time.monotonic() > deadline:
+                raise TimeoutError(f"client timeout: {entry['cell']} {role}")
+            if entry["server"].poll() is not None:
+                raise RuntimeError(f"server exited during {entry['cell']} {role}")
+            base.assert_no_other_compute()
+            if state["gpu_hours_total"] + base.live_gpu_hours([entry]) >= base.GPU_LIMIT:
+                raise RuntimeError("phase-aware pilot H20-hour hard cap reached")
+            time.sleep(1.0)
+    except Exception:
+        try:
+            os.killpg(process.pid, signal.SIGTERM)
+        except ProcessLookupError:
+            pass
+        try:
+            process.wait(timeout=10.0)
+        except subprocess.TimeoutExpired:
+            try:
+                os.killpg(process.pid, signal.SIGKILL)
+            except ProcessLookupError:
+                pass
+            process.wait(timeout=10.0)
+        raise
+    finally:
+        handle.close()
+    if process.returncode:
+        raise RuntimeError(
+            f"client failed: cell={entry['cell']} role={role} rc={process.returncode}"
+        )
+    result = json.loads((output / "result.json").read_text(encoding="utf-8"))
+    validate_result(
+        result=result,
+        selection=selection,
+        role=role,
+        warmup=warmup,
+    )
+    entry["results"].append(
+        {
+            "anchor": float(selection["anchor"]),
+            "dir": str(output),
+            "kind": result["kind"],
+        }
+    )
+    return result
+
+
+def validate_result(
+    *, result: dict[str, Any], selection: dict[str, Any], role: str, warmup: bool
+) -> None:
+    if result.get("slo_early_stop_disabled") is not True:
+        raise RuntimeError(f"SLO early stop was not disabled: {role}")
+    if warmup:
+        if result["kind"] != "warmup" or int(result["selection"]["count"]) != 16:
+            raise RuntimeError(f"invalid warmup result: {role}")
+        if not all(
+            result["invariants"].get(key, False)
+            for key in ("warmup_16", "warmup_exact_16", "warmup_long")
+        ):
+            raise RuntimeError(f"warmup invariant failed: {role}")
+        return
+    if bool(result["early_stopped"]):
+        raise RuntimeError(f"uncensored run early-stopped: {role}")
+    if int(result["selection"]["count"]) != int(selection["selected_count"]):
+        raise RuntimeError(f"selection count mismatch: {role}")
+    for key, manifest_key in (
+        ("request_id_order_sha256", "request_id_order_sha256"),
+        ("arrival_order_sha256", "arrival_order_sha256"),
+        ("raw_length_order_sha256", "input_length_order_sha256"),
+    ):
+        if result["selection"][key] != selection[manifest_key]:
+            raise RuntimeError(f"selection hash mismatch {key}: {role}")
+    if int(result["observed_count"]) != int(selection["selected_count"]):
+        raise RuntimeError(f"request accounting mismatch: {role}")
+
+
+def execute_session(
+    *,
+    index: int,
+    session: dict[str, Any],
+    manifest: dict[str, Any],
+    run_root: Path,
+    state_path: Path,
+    state: dict[str, Any],
+) -> None:
+    name = str(session["session"])
+    if state["sessions"].get(name, {}).get("status") == "complete":
+        return
+    projection = remaining_projection(len(manifest["sessions"]), index)
+    if state["gpu_hours_total"] + projection > base.GPU_LIMIT:
+        state["status"] = "budget_projection_stop"
+        state["budget_stop"] = {
+            "before_session": name,
+            "spent_h20_hours": state["gpu_hours_total"],
+            "remaining_projection_h20_hours": projection,
+            "hard_cap_h20_hours": base.GPU_LIMIT,
+        }
+        save_state(state_path, state)
+        raise RuntimeError(f"projected pilot cost exceeds hard cap before {name}")
+
+    replicate = str(session["replicate"])
+    repetition = manifest["repetitions"][replicate]
+    echo = (
+        f"PHASE_PILOT_SESSION_ECHO session={name} tp=4 mns={session['mns']} "
+        f"gpus=0-3 workload={manifest['source']['window_id']} duration_s=300 "
+        f"loads={','.join(repetition['load_order'])} disable_slo_early_stop=true "
+        f"spent_h20h={state['gpu_hours_total']:.6f} "
+        f"remaining_projection_h20h={projection:.3f} cap_h20h={base.GPU_LIMIT:.1f} "
+        f"manifest={run_root / 'pilot-manifest.json'}"
+    )
+    append_echo(run_root, echo)
+    wait_all_idle()
+    session_state = {
+        "status": "starting",
+        "replicate": int(replicate),
+        "mns": int(session["mns"]),
+        "started_at": time.time(),
+        "runs": [],
+    }
+    state["status"] = "running"
+    state["sessions"][name] = session_state
+    save_state(state_path, state)
+    entry = start_server(session=session, index=index, run_root=run_root)
+    failure: Exception | None = None
+    try:
+        base.wait_ready(entry)
+        warmup = warmup_selection(repetition)
+        session_state["status"] = "warmup"
+        save_state(state_path, state)
+        run_client(
+            entry=entry,
+            role="warmup",
+            study=repetition["study"],
+            selection=warmup,
+            output=entry["dir"] / "warmup",
+            state=state,
+            warmup=True,
+        )
+        session_state["status"] = "burnin"
+        save_state(state_path, state)
+        burnin = manifest["burnin"]
+        burnin_result = run_client(
+            entry=entry,
+            role="burnin",
+            study=burnin["study"],
+            selection=burnin,
+            output=entry["dir"] / "burnin",
+            state=state,
+        )
+        session_state["burnin"] = {
+            "pass_rate": burnin_result["pass_rate"],
+            "feasible": burnin_result["feasible"],
+            "elapsed_s": burnin_result["interval"]["elapsed_s"],
+        }
+        session_state["status"] = "measured"
+        save_state(state_path, state)
+        for level in repetition["load_order"]:
+            selection = repetition["selections"][level]
+            result = run_client(
+                entry=entry,
+                role=level,
+                study=repetition["study"],
+                selection=selection,
+                output=entry["dir"] / level,
+                state=state,
+            )
+            session_state["runs"].append(
+                {
+                    "level": level,
+                    "selected_count": selection["selected_count"],
+                    "offered_req_s_per_gpu": selection["offered_req_s_per_gpu"],
+                    "pass_rate": result["pass_rate"],
+                    "feasible": result["feasible"],
+                    "elapsed_s": result["interval"]["elapsed_s"],
+                    "early_stopped": result["early_stopped"],
+                }
+            )
+            save_state(state_path, state)
+        session_state["status"] = "stopping"
+        save_state(state_path, state)
+    except Exception as error:  # noqa: BLE001
+        failure = error
+    finally:
+        try:
+            base.stop_entry(entry)
+        except Exception as error:  # noqa: BLE001
+            failure = failure or error
+        time.sleep(2.0)
+        try:
+            wait_all_idle()
+        except Exception as error:  # noqa: BLE001
+            failure = failure or error
+
+    session_hours = base.live_gpu_hours([entry])
+    state["gpu_hours_total"] += session_hours
+    session_state["gpu_hours"] = session_hours
+    if failure is not None:
+        session_state["status"] = "failed"
+        session_state["failure"] = repr(failure)
+        state["status"] = "failed"
+        state["failures"].append({"session": name, "failure": repr(failure)})
+        save_state(state_path, state)
+        raise failure
+    validation = base.validate_cell(entry)
+    session_state["validation"] = validation
+    session_state["status"] = "complete"
+    session_state["completed_at"] = time.time()
+    state["completed_sessions"] += 1
+    save_state(state_path, state)
+
+
+def parser() -> argparse.ArgumentParser:
+    result = argparse.ArgumentParser()
+    result.add_argument("--manifest", type=Path, required=True)
+    result.add_argument("--run-root", type=Path, required=True)
+    result.add_argument("--aituner-root", type=Path, required=True)
+    result.add_argument("--vllm-source", type=Path, required=True)
+    result.add_argument("--venv", type=Path, required=True)
+    result.add_argument("--model", type=Path, required=True)
+    result.add_argument("--client", type=Path, required=True)
+    result.add_argument("--dry-run", action="store_true")
+    return result
+
+
+def main() -> None:
+    args = parser().parse_args()
+    manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
+    validate_inputs(args, manifest)
+    configure(args, manifest)
+    if args.dry_run:
+        print(json.dumps(dry_run_plan(args, manifest), indent=2, sort_keys=True))
+        return
+    args.run_root.mkdir(parents=True, exist_ok=True)
+    copied_manifest = args.run_root / "pilot-manifest.json"
+    if not copied_manifest.exists():
+        atomic_json(copied_manifest, manifest)
+    state_path = args.run_root / "controller-state.json"
+    state = load_state(state_path, base.GPU_LIMIT)
+    state["status"] = "running"
+    save_state(state_path, state)
+    for index, session in enumerate(manifest["sessions"]):
+        execute_session(
+            index=index,
+            session=session,
+            manifest=manifest,
+            run_root=args.run_root,
+            state_path=state_path,
+            state=state,
+        )
+    state["status"] = "complete"
+    state["completed_at"] = time.time()
+    save_state(state_path, state)
+    wait_all_idle()
+    print(
+        json.dumps(
+            {
+                "status": state["status"],
+                "completed_sessions": state["completed_sessions"],
+                "gpu_hours_total": state["gpu_hours_total"],
+            },
+            sort_keys=True,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()